methodological issues in physician-level measurement of clinical quality

Methodological Issues in Physician-Level Measurement

of Clinical Quality

Elizabeth A. McGlynn, Ph.D.June 26, 2006

McGlynnAcadHealth-2 06/26/06

Information About Individual Physicians’ Performance is Increasingly Sought

• Health plans believe they can save money through differential payments to physicians (pay for performance)

• Employers believe they can save money through increasing consumer cost-sharing (consumer directed health plans)

• Medical groups believe they can negotiate higher rates or market share by demonstrating better performance (tiered networks, rate increases)

• Consumers are likely to demand information on performance as the share they pay for health care increases (public release)

McGlynnAcadHealth-3 06/26/06

What Is Being Measured?

McGlynnAcadHealth-4 06/26/06

Data Sources for Measuring Quality

• Available sources include:– Administrative (claims) data– Manual abstraction of medical records– Surveys of patients– Inspection of office practice– Extraction of data from electronic medical

records– Board certification/Maintenance of certification

• Each of these sources has strengths and weaknesses• No single source is adequate to address all

questions

McGlynnAcadHealth-5 06/26/06

Most Existing Approaches to Measuring Physician Performance Use Claims Data

• Data are readily available and impose less burden on providers

• But they have some significant problems– Generally available one payer at a time– Information availability driven by the benefit

package and the ways coding systems are used– Some confounding of physician practice

patterns with patient behavior

• Pressure to deliver answers driving widespread use of these methods

McGlynnAcadHealth-6 06/26/06

Current Approaches to Quality Measurement

• “Leading indicators”– One measure at a time

• Condition-specific aggregates/composites– Multiple measures on the same population with

the same health problem

• Comprehensive cross-condition measures– Patient as the unit of analysis

McGlynnAcadHealth-7 06/26/06

Examples of Where These Approaches Are Currently Used

Approach Use

Leading indicators Pay for performance

Public reporting

Tiered networks

Disease composites Recognition programs

Maintenance of certification

Comprehensive aggregates Not in widespread use

McGlynnAcadHealth-8 06/26/06

What You Measure May Affect the Conclusions You Draw

0 20 40 60 80 100

Cardiology

Family Practice

Endocrinology

InternalMedicine

% of recommended care delivered

HbA1c DM Overall QA Tools Overall

McGlynnAcadHealth-9 06/26/06

Some Challenges in Measuring Physician Performance

McGlynnAcadHealth-10 06/26/06

Physicians See Multiple [Different] Patients

MD1

PT3

PT2

PT1

So, representing the variety of practice matters:Case Mix Adjustment

MD2

PT5

PT4

PT3

McGlynnAcadHealth-11 06/26/06

A Market Basket of Indicators May Be Necessary to Reflect the Variety of Practice

0%

20%

40%

60%

80%

100%

Cardio

logy

Endocrin

ology

Family

Pra

ctic

e

Inte

rnal

Med

icin

e

OB-GYN

% o

f el

igib

le e

ven

ts

Afib

CAD

CHF

Diabetes

Headache

Hypertension

Hyperlipidemia

Pneumonia

Prenatal

Preventive

UTI

Other

McGlynnAcadHealth-12 06/26/06

Patients See Multiple Providers

PT1

MD3

MD2

MD1PT2

PT3

PT4

PT5

PT6

PT9

PT8

PT7

So, determining who is “responsible” mattersAttribution

Hosp A

Hosp B

McGlynnAcadHealth-13 06/26/06

Information Rarely Available to Link Patients to Physicians a Priori

• As gatekeeper models decline, no clear assignment of patients to a physician exists

• Algorithms are used to “assign” patients to physicians– Done most frequently in economic profiling– Basis is majority of dollars or visits

• We are experimenting with other rules:– First eligible provider seen in study period– Provider “triggering” eligibility for indicator

• Critical to reality test assignments

McGlynnAcadHealth-14 06/26/06

Physicians Have Multiple Contracts

Medicare

MD3

MD2

MD1PacifiCare

Humana

Wellpoint

United

Aetna

Medicare

Medicare

Anthem

So, putting the pieces together matters:Aggregation

McGlynnAcadHealth-15 06/26/06

Few Physicians Can Be Evaluated Using Single Indicators from One Payer

0 10 20 30 40 50

Cardiology

Family Practice

Endocrinology

Internal Medicine

% of MDs with >10 eligibilities

HbA1c DM Overall QA Tools Overall

McGlynnAcadHealth-16 06/26/06

Physicians Practice in Different Systems

So, understanding the organizational context matters:Fair comparisons

McGlynnAcadHealth-17 06/26/06

Little Routine Information Available on Physician Practice Setting

• Taking organizational context into account is challenging because of data limitations

• Using location may be misleading– Shared space vs. shared practice

• Rationale for constructing scores at group level:– Increase sample size– Demonstrate value of integrated medical groups– Avoid scores at the physician level

• Relatively little known about within vs. between group variation

McGlynnAcadHealth-18 06/26/06

Categorizing Physician-Level Results• Many applications of physician-level scoring

require using results to categorize physicians– In/out of network– In/out of performance bonus– Tiering

• We prefer statistical testing to straight cut-points• Applied this to the three different approaches to

MD-level scoring– Test performance compared to the mean– Use 95% confidence interval around each

provider’s score– Those with scores significantly below average

were assigned to the low performance category

McGlynnAcadHealth-19 06/26/06

Different Methods Will Result in Different Category Assignments

0% 20% 40% 60% 80% 100%

HbA1c

DMComposite

QA ToolsOverall

% of internists in category

Not rated

1 star

2 stars

3 stars

McGlynnAcadHealth-20 06/26/06

Different Results Under Different Systems Likely To Produce Challenges from MDs

0% 20% 40% 60% 80% 100%

% of internists

Agree

DM>QAT

DM<QAT

McGlynnAcadHealth-21 06/26/06

Summary

• A number of methodological issues arise in creating quality scores at the physician level

• We need to better understand the implications of these methodological choices

• Because the data on which the scores are based were not intended for this purpose, feedback loops and data quality improvement are essential

• But, the world isn’t going to wait for us to get the methods perfect…

McGlynnAcadHealth-22 06/26/06

This Train Is Headed Your Way!