for the practice change fellows program september 24, 2009 washington, dc dennis a. ehrich, md, facc...
TRANSCRIPT
For the Practice Change Fellows ProgramSeptember 24, 2009
Washington, DC
Dennis A. Ehrich, MD, FACCVice President for Medical Affairs
St. Joseph’s Hospital Health CenterSyracuse, New York
The Importance of Measurement in Health Care
Agenda for the Afternoon
1-When do we measure in health care2-Project alignment and design3-Understanding and selecting measures5-Operational definitions, change concepts, and data
collection6-Data analysis, comparative data, benchmarks and
data reporting7-Time ordered statistics and understanding variation:
the value and the pitfalls
Measurement in Health CareMeasuring for
ResearchMeasuring for
JudgmentMeasuring for Improvement
Purpose To discover new knowledge
To compare to others, to rank
To bring new knowledge into daily practice
Tests One large trial to answer a few
questions
Submission of quarterly aggregated
data for public reporting.
12 month running averages
Many small, sequential, observable tests
Bias Control for as much as possible
Severity or risk adjustment where
available
Stabilize the biases from test to test
Data Gather as much data as possible, just in
case
Measures structure, process or outcomes
Usually applied to process
Duration Requires large numbers of patients/ long periods of time
to obtain results
Ongoing data collection and periodic
public reporting
Short iterative cycles in a limited number of subjects,
followed by spread
Project Design: Selection and Alignment
• Does your project align with– The mission and values of your organization?– Your organization’s vision and strategic plan?
• Do you have support of your leadership?• Are all of the stakeholders involved in the
design process?
Selecting Your Measures
Donabedian’s Quality Triangle-It’s Relevance to Process Improvement
-Avedis Donabedian, MD, MPH (1919-2000)
Donabedian’s TriadStructure
OrganizationPeopleEquipment/Technology
ProcessThe steps taken in accomplishing the change and achieving the
outcomeResults must be client-focusedMust deliver results reliably
OutcomesClinical (mortality, complications)Client perception or satisfactionFinancial
The Three Domains of Measurement
• Structural Measures• Process measures• Outcomes Measures– Balancing measures
Donabedian
The Three Domains of Measurement
• Structural Measures– Describe the environment. How many?– Square footage of a clinical unit– Number of staff– Staff qualifications and competencies– Presence or absence of technology and its
characteristics• Process Measures
• Process step cycle time• Number of defects at key steps of the process
Donabedian
The Three Domains of Measurement
• Outcome Measures• The impact of the change initiative on mortality,
readmissions to the hospital, ED visits, etc.• The satisfaction scores of clients and staff • The cost per case, average LOS, revenue per case
• Balancing Measures – Unintended outcomes that are consequences of the
new program– Unanticipated mortality, morbidity or cost – Has the shifting of resources in an organization
compromised other client or patient populations?Donabedian
The Quality Measurement Roadmap
Modified from Lloyd, Robert: “Quality Health Care A Guide to Using Indicators”
Defining the Project (Process)
Selecting Measures
Operational Definitions/Change Concepts
Data Collection Plan
Data Collection
Data Analysis
Defining targets and Benchmarks
Data Reporting
Establish Operational Definitions Agreed Upon By All Stakeholders• Are clear and unambiguous• Specify the measurement method, procedures and
equipment when appropriate– Clinical data (chart reviews) vs. administrative data– Client logs vs. a computer database
• Define specific criteria for the data to be collected– Define all inclusions and exclusions– For percentages or rates, or ratios, define the criteria
for inclusion in the numerator and denominator• Always ask “How might somebody be confused by this
definition?”
Lloyd, R. Quality Health Care (2004) Jones and Bartlett
Examples of Unclear Definitions
• Timely completion of the screening process• A complete medication list• The readmission rate• Medication error• Cost impact• From the acute care hospital– A patient fall– Surgical start time
Lloyd, R. Quality Health Care (2004) Jones and Bartlett
What Dimension of Performance Will You Affect?
• Appropriateness • Availability• Continuity• Effectiveness• Efficiency• Respect and caring• Financial/Viability• Safety• Time lines
Joint Commission (1996)
Which IOM Domain of Quality Will be Affected?
• Safety• Effectiveness• Patient-centeredness• Timeliness• Efficiency• Equity
IOM: Crossing the Quality Chasm (2001)
What is the “Change Concept”?• Eliminate waste• Improve work flow• Shorten a waiting list• Change the work environment• Improve the Provider/Client interface• Manage time• Focus on variation• Error proofing a process• Focusing on product or service
The Improvement Guide by Langley, Nolan, Nolan, Norman and Provost. Jossey-Bass
The Data Collection Plan and Data Collection
Do You Have Baseline Data?
• Does your project, introduce a new process to your organization?
• Does it improve an existing process?– There is probably some existing data in your
organization?– May not be useful, if its collection was flawed or it
was collected for a purpose unrelated to your work
Collecting Data
• What data will be collected and how often?– Outcome data (monthly or quarterly)?– Process data (daily or weekly)?
• On what patients and how often will data be collected?– Frequent small samples are preferable– Beware of strata
• weekends vs. weekdays, shifts, or different teams of people
• How will the data be collected?– From an existing computer database– Use of a tool such as a “traveler” check-sheet?
Comparative Data and Targets
• Sources of comparative data– Internal targets-trended data– External comparisons– National or regional population averages– Best practice (top decile performance)
• Benchmarking– Outcome benchmarking– Process benchmarking
• Don’t set arbitrary or unattainable goals
Data Reporting
• A data reporting plan– Who will receive the results?– How often will they receive the results?– How will it be formatted?
• Tabular data• Graphics• Spider diagram
– How will the data be disseminated?• Snail mail/Fax/E-mail• Internet• Intranet Online Dashboard
Data Analysis
• Is the outcome (output) of the process improving?• To improve the output, we will need to reduce the variation in
the process inputs– Evaluate the output relative to the inputs
• The inputs – People-Are particular individuals or teams performing tasks in a non-
standardized fashion?– Machines-If body weight is being measured, are all scales accurate?– Methods-Is there variation in the way a procedure is being done?– Measurements-If a cycle time is being measured, is there a standard clock
being used or are data collectors using their individual watches?– Materials-Are all patients receiving the full compliment of brochures?– Environment-Is there a resulting from different time of day, season?
Defining your project in process terms
Cycles of Learning: The IHI Model for Improvement
Establishing Measures
Defining the change concept
Testing and perfecting changes on a small scale using repetitive cycles of learning
T. Nolan et al. www.ihi.org
The Use of Iterative PDSA Cycles
Learning and ImplementingChanges
“Rapid-cycle CQI”
T. Nolan et al. www.ihi.org Multiple Simultaneous Tests of Change
Spreading the Change
1-Executive sponsorship2-Planning and set-up 3-Spread within the target population-social network theory 4-Continuous monitoring and feedback during the spread process5-Capturing and sharing organizational learning
T. Nolan et al. www.ihi.org
Data Reporting Tools
• Would recommend tracking the data in a time-ordered fashion– Run chart– Control chart
• In preference to using– Tabular data– Bar charts– Histograms– Pie charts
Tools for Displaying Time-ordered Data
• Run charts– Plot of data over time– The centerline of the chart is the median of the
data
• Control charts– The centerline of the chart is the mean– Also has upper and lower control limits calculated
from the data– Difficult, but not impossible to construct by hand
The Pitfalls of Aggregated Data
Annual Aggregated Data Before and After Implementation of Process Change: The Change Appears to have improved outcomes
The Pitfalls of Using Aggregated Data
Time-Ordered Data: Do you still feel the same way?
Pitfalls of Aggregate Data
A trend is a series of seven consecutive data points rising or falling. Data points on the centerline are not counted for the trend, but don’t interrupt the trend.
Sample Run Chart
Time Period
Valu
e
Examples of a Trend
Understanding Variation• All data, collected over time has variation• There are two kinds of variation
• Random variation (common cause)– The changes occurring are intrinsic to the process being measured and
is caused by variation in the process inputs• Non-random variation (special cause)
– The changes are being imposed on the system by some external factor– May be unintended and un anticipated or may be by design and a
reflection of your intervention• There are sets of rules for formally diagnosing special-cause
variation on run charts and control charts
A Run Chart With Common Cause (Random) Variation
A Control Chart with Common Cause Variation
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
A Control Chart With Special Cause Variation
A Control Chart with (Deliberate) Special Cause Variation
This change is known as a “process shift”
Let’s Put Some of These Ideas to Work