Performance Information, Transparency, and Accountability in the Health Sector:

Experience in the United States (and a bit beyond)

Joseph White, Ph.D. Case Western Reserve University

Prepared for 5th DELSA GOV meeting on Sustainability of Health Systems

5 February 2016

OECD Conference Center

1

“Paying for Performance” (P4P) in Health Sector and “Performance Budgeting”

• Common Aspirations – • Efficiency or Value for Money, Making Economy More Acceptable

• Common Problems – • Difficulties in Measuring Performance, and Agreeing on Measures

• Both technique and source must be credible.

• Strategic Responses (e.g. “Gaming,” “Teaching to the Test”) • Using the Results

• One Possible Advantage in Health if Not Budgeting • Do Not Have to Compare the Incomparable - Mostly

• Huge Problems if “Performance Budgeting for Health” • Voters Want Rescue and Care, Analysts Look at Health Statistics • In U.S. (and other countries) Institutional Separation of Programs

2

Logical Questions (or choices) • Information About What?

• Activity vs. Outcomes or “Value”

• Provided to Whom? • Citizens? Consumers? Patients? • Payers? Governments? Managers?

• So “Transparent” to Whom? And Why? • “Legitimacy” or Control?

• Accountability for What? • Quality? Spending? Compliance?

• Applied to What Parts of the System? • Hospitals, physicians, medical groups, “health plans”

3

Measuring Activity – Usually to Increase Activity

• To Reduce Waiting Times • NHS “Targets and Terror”

• To Induce Greater Productivity Through Payment • DRGs outside the United States: “Activity-Based Payment”

• To Improve Management • “Knowing what you are buying.” DRGs again

• To Encourage Specific Services – Often “Prevention” • “Pay for Performance” for NHS GPs, U.S. Primary Care Groups

• To Punish Specific Activities • Readmission rates in United States

4

Measuring Quality, Outcomes or “Value” • Results of Treatment

• New York Cardiac Surgery Reporting System • Medicare ACOs patient-reported experience of care and health status

• Overall Rankings or Ratings by “Experts” • Intended to stand in for quality, but usually not based on strong measures.

Many versions in U.S.; NHS Star Ratings

• Various Theories of How These Would Work • Change: Providers act to improve when see comparisons, even if not public • Selection: Providers respond to threat of patients selecting other providers • Reputation: “Naming and Shaming.” e.g. media coverage • Individual self-interest, e.g. managers could be fired, provider income

affected, budgets could be raised or cut, autonomy “earned.” • Voltaire on British Navy: “Ici on tue de temps en temps un amiral pour encourager les

autres”

5

Measurement Focuses Attention on Some Things, Not Others

6

• Source: larrycuban.wordpress.com

“Core Assumptions” in Using Measurement to Inform

“One is that measurement problems are unimportant, that the part on which performance is measured can adequately represent performance of the whole, and that distribution of performance does not matter. The other is that this method of governance is not vulnerable to gaming by agents.” Danger of “hitting the target and missing the point” (Bevan and Hood, 2006)

7

Common Measurement Problems

• Statistical Validity • Too few cases per provider for specific measures. Bigger problem in U.S.

where providers face multiple payers. U.S. IHA vs. English QOF • If providers collect the data, may be gamed; if outsiders do, may not be

trusted by providers (and in U.S., often rightly so – IHA).

• Only Measure What Can Observe (see previous slide)

• Black Boxes – What is Being Measured isn’t Transparent • Aggregate ratings of the “same thing” may be very different

• Documentation May Not Equal Performance • Either because measures provoke documenting activity that already

occurred, or documentation isn’t accurate (gaming)

• Risk Adjustment is Very Difficult and Contentious • Measuring “Improvement” Requires Accurate Baselines

8

Common Implementation Issues • Cost in Time and Money

• Cost in Diverted Attention from Other Activities

• Cost in Inverted Incentives if Linked to Sanctions or Rewards • Ariely et al on theory of motivation. Bevan and Hood on behavior changing

as people are punished for not achieving something they could not achieve. • But: providers happier to act to “improve quality” if sweetened with $ € £

• Public Rarely Uses “Value” Information • Does not have same doubts about quality as the experts do. • Not likely to notice information; if do will ignore highly detailed

presentations; if presentation is simple, may be misleading.

• Governments and other payers may not want bad news on quality • Payers tend to care much more about cost, do not want evidence they

should spend more.

9

“Accountability” Based on Measures Requires Conditions That May be Hard to Meet

• The Message is Clear and Targeted Properly • Measures are Perceived as Accurate • Measures are Perceived as Fair and Legitimate

• (or incentives really strong anyway, but then cheating becomes more likely) • This result may be related as much to organizational context as to the

measures themselves. U.S. physicians in IPAs were in very different situations than English GP practices

• Targets of Incentive Are Able to Respond Effectively • The desired result is potentially subject to significant influence by the

recipient of the incentive (technical capacity) • Organizational setting provides institutional capacity to respond

• Targets Don’t Have Political Capacity to Change the Rules

10

“Accountability” and “Pay for Performance” • English QOF certainly led to higher recorded “performance,” but at

very high monetary cost. • English “Targets and Terror” likely did improve target performance

beyond gaming effects. But at some costs on unmeasured performance dimensions

• U.S. “P4P” experience shows generally modest or null effects. • “largely disappointing results” (Markowitz and Ryan 2016)

• For physicians, “among the strongest studies, there were no or relatively small improvements in performance.” In the hospital setting, “all of the studies… found modest but often statistically insignificant effects” (Damberg et al. 2014)

• Some acceleration of improvement in early years of Premier demonstration. • Performance on some Hospital Inpatient Quality Reporting Program

measures is now so high they are being eliminated. But they are very basic measures.

• Risk adjustment concerns, especially for hospitals with low SES patients 11

Some Other Tentative or Ironic Results • When processes are measured, improved performance is rarely

associated with better final outcomes • Long or weak causal chains between measured feature and outcome

• Bad (or good) outcomes may be rare. Or outcomes may emerge only after many years

• Comparisons over time are difficult because governments and other payers keep changing the measures

• U.S. Policy-Makers Want to Use “Transparency” to Reduce Prices. • The new-new thing. Not so necessary anywhere else. A whole other topic…

• “Transparency” is sometimes associated with avoiding sicker patients • Percutaneous revascularization is less common, and that is associated with

higher mortality, in states that report “quality” of AMI care at the physician level and those that do not.

12

Final Thoughts (for the moment) • Some use of measures for management “is a form of indirect control

necessary for the governance of any complex system” (Bevan and Hood 2006)

• Measurement results from lack of trust in providers, is likely to decrease trust, but will work best in the presence of trust.

• If measures are inaccurate they cannot increase “transparency”

• Hardly anybody likes the current U.S. measurement regime: • “The quality measurement enterprise in U.S. health care is troubled. Physicians, hospitals,

and health plans view measurement as burdensome, expensive, inaccurate, and indifferent to the complexity of care delivery. Patients and their caregivers believe that performance reporting misses what matters most to them and fails to deliver the information they need to make good decisions. In an attempt to overcome these troubles, measure developers are creating ever more measures, and payers are requiring their use in more settings and tying larger financial rewards or penalties to performance. We believe that doing more of the same is misguided: the time has come to reimagine quality measurement.” – (McGlynn et al, NEJM, 2014.)

• But reform ideas require even more organizational capacity…

13

Improvement Through Performance Management Would Be a Beautiful Thing

• “Unicorn in Captivity,” Property of The Cloisters, Metropolitan Museum of Art. One of the great treasures of New York City

Some Sources Reviewed for This Presentation (1) • Austin, J. M., A. K. Jha, P. S. Romano, S. J. Singer, T. J. Vogue, R. M. Wachter, P. Pronovost. 2015. National Hospital Rating Systems Share Few Common Scores and

May Generate Confusion Instead of Clarity. Health Affairs 34(3): 423-430.

• Bevan, G. and C. Hood. 2006. What’s Measured is What Matters: Targets and Gaming in the English Public Health Care System. Public Administration 84(3): 517-538.

• Bevan, G. and R. Hamblin. 2009. Hitting and missing targets for ambulance services for emergency calls: effects of different systems of performance measurement within the UK. Journal of the Royal Statistical Society 172(1): 161-190

• Hood, C. and R. Dixon. 2015. A Government that Worked Better and Cost Less? Evaluating Three Decades of Reform and Change in UK Central Government. Oxford, UK: Oxford University Press.

• Markowitz, A. A. and A. M. Ryan. 2016. Pay-for-Performance: Disappointing Results or Masked Heterogeneity? Medical Care Research and Review 71 (online first) 1-76.

• Damberg, C. L., M. E. Sorbero, S. L. Lovejoy, G. Martsolf, L. Raaen, D. Mandel. 2014. Measuring Success in Health Care Value-Based Purchasing Programs. RAND Corporation Research Report sponsored by the Office of the Assistant Secretary for Planning and Evaluation, Department of Health and Human Services

• Friedberg, M. W. and C. L. Damberg. 2012. A Five-Point Checklist To Help Performance Reports Incentivize Improvement and Effectively Guide Patients. Health Affairs 31(3): 612-618.

• Hibbard, J. H. 2008. Editorial: What Can We Say About the Impact of Public Reporting? Inconsistent Execution Yields Variable Results. Annals of Internal Medicine 148(2): 160-161.

• Hibbard, J.H., J. Greene, S. Sofaer, K. Firminger, J. Hirsh. 2012. An Experiment Shows That A Well-Designed Report On Costs And Quality Can Help Consumers Choose High-Value Health Care. Health Affairs 31(3): 560-67.

• Kahn, C. N., T. Ault, L. Potetz, T. Walke, J. H. Chambers, S. Burch. 2015. Assessing Medicare’s Hospital Pay-For-Performance Programs And Whether They Are Achieving Their Goals. Health Affairs 34(8): 1281-1288.

• Laverty, A. A., P. C. Smith, U. J. Pape, A. Mears, R. M. Wachter, C. Millett. 2012. High-Profile Investigations Into Hospital Safety Problems in England Did Not Prompt Patients To Switch Providers. Health Affairs 31(3): 593-601.

• Lawson, E. H., D. S. Zingmond, B. L. Hall, R. Louie, R. H. Brook, C. Y. Ko. 2015. Comparison Between Clinical Registry and Medicare Claims Data on the Classification of Hospital Quality of Surgical Care. Annals of Surgery 261(2): 290 – 296

• McDonald, R., J. White, T. R. Marmor. 2009. Paying for Performance in Primary Medical Care: Learning about and Learning from “Success” and “Failure” in England and California. Journal of Health Politics, Policy and Law 34(5): 747-776.

• McGlynn, E. A., E. C. Schneider and E. A. Kerr. 2014. Reimagining Quality Measurement. New England Journal of Medicine 371(23): 2150-2153

15

Some Sources Reviewed for This Presentation (2) • Parast, L., B. Doyle, C. L. Damberg, K. Shetty, D. A. Ganz, N. S. Wenger, P. G. Shekelle. 2015. Perspective: Challenges in Assessing the Process-Outcome Link in

Practice. Journal of General Internal Medicine 30(3): 359-64.

• Rosenbaum, L. 2015. Scoring No Goal: Further Adventures in Transparency. New England Journal of Medicine 373(15): 1385-1388.

• Ryan, A. M., B. K. Nallamothu, J. B. Dimick. 2012. Medicare’s Public Reporting Initiative on Hospital Quality Had Modest Or No Impact on Mortality From Three Key Conditions. Health Affairs 31(3): 585-592.

• Smith, M.A., A. Wright, C. Queram, G. C. Lamb. 2012. Public Reporting Helped Drive Quality Improvement In Outpatient Diabetes Care Among Wisconsin Physician Groups. Health Affairs 31(3): 570-577.

• Tanenbaum, S. J. 2009. Pay for Performance in Medicare: Evidentiary Irony and the Politics of Value. Journal of Health Politics, Policy and Law 34(5): 717-746.

• Waldo, S. W., J. M. McCabe, C. O’Brien, K. Kennedy, K. E. Joynt, R. W. Yeh. 2015. Association Between Public Reporting of Outcomes With Procedural Management and Mortality for Patients With Acute Myocardial Infarction. Journal of the American College of Cardiology 65(11): 1119-1126.

• Werner, R. M., J. T. Kolstad, E. A. Stuart, D. Polsky. 2011. The Effect Of Pay-For-Performance In Hospitals: Lessons For Quality Improvement. Health Affairs 30(4): 690-698.

• Woolhandler, S., D. Ariely, D. Himmelstein. 2012. Will Pay for Performance Backfire? Insights From Behavioral Economics. Health Affairs blog, http://healthaffairs.org/blog/2012/10/11/will-pay-for-performance-backfire-insights-from-behavioral-economics/

16