the revised oecd health systems performance framework: methodological issues and implications for...

The revised OECD Health Systems Performance Framework: methodological issues and implications for comparative public reporting at a global levelFabrizio CarinciProfessor of Health Systems and PolicySchool of Health Sciences, University of SurreyMember of the OECD Expert Group on Health Care Quality Indicators

ForumAgency for Clinical InnovationSydney, AustraliaWednesday, 24th February 2016

[email protected]

Fabrizio Carinci

OECD Health Care Quality Indicators Project Under the umbrella of the Organisation for Economic Cooperation

and Development (OECD), the ‘Health Care Quality Indicators’ (HCQI) Project was initiated in 2001

The general objective was to help Member States (MS) identify priority areas for quality improvement to provide achievable standards by examining results among best performing countries

In 2006, the OECD released a common conceptual framework for health system performance. Nested “quality matrix” with vertical dimensions of ‘effectiveness’, ‘patient safety’ and ‘responsiveness/patient-centeredness', horizontally subdivided according to levels of health care needs over the life cycle: ‘staying healthy’ for healthy subjects, ‘getting better’ for people affected by a disease, ‘living with illness or disability’ for those with a chronic condition and ‘coping with end of life’ for terminal patients.

Fabrizio Carinci

Matrix Dimensions: Quality

Effectiveness

Achieving desirable outcomes, given the correct provision of evidence-based health care services to all who could benefit

Achieving desirable outcomes, given the correct provision of evidence-based health care services to all who could benefit

Safety

System has the right structures, renders services and attains results in ways that prevent harm to the user, provider, or environment

System has the right structures, renders services and attains results in ways that prevent harm to the user, provider, or environment

Responsiveness/

Patient centeredness

System actually functions by placing the patient/user at the center of its delivery of health care

System actually functions by placing the patient/user at the center of its delivery of health care

Fabrizio Carinci

HCQI Outputs

Since 2007, results of the HCQI project have routinely contributed to international comparisons through the publication of the OECD series ‘Health at a Glance’ and the release of OECD Health Statistics alongside other international health data on expenditure, resources, utilization and outcomes.

In 2013, the OECD HCQI data collection process included a total of 70 indicators covering the following ‘themes’: Primary Care (PC); Acute Care (AC); Mental Health (MH); Cancer Care (CC); Patient Safety (PS) and Patient Experiences (PEs). The collection reports data from 34 countries, including non-OECD member countries eg Singapore and Latvia.

Fabrizio Carinci

NEW DEFINITIONS

Revised OECD-HCQIHealth Systems Performance Framework 2015

Fabrizio Carinci

Criteria used to select HCQIs

Fabrizio Carinci

Carinci F, Van Gool K, Mainz J, Veillard JH, Januel JM, Kim SM, Arispe I, and Klazinga N, Towards actionable international comparisons of health system performance: expert revision of the OECD framework and quality indicators, Int J Qual Health Care. 2015 Apr;27(2):137-46.

Fabrizio Carinci

Effectiveness – Getting better


Fabrizio Carinci

Effectiveness – Chronic care


Fabrizio Carinci

Safety – Primary and secondary prevention


Fabrizio Carinci

Safety – Getting better


Fabrizio Carinci

Responsiveness / Patient centrednessPrimary and Secondary Prevention


Fabrizio Carinci

HCQI 2015

Fabrizio Carinci

Mortality after AMISource: OECD Health at a Glance 2015http://www.oecd.org/health/health-systems/health-at-a-glance-19991312.htm

Fabrizio Carinci

Mortality after StrokeSource: OECD Health at a Glance 2015http://www.oecd.org/health/health-systems/health-at-a-glance-19991312.htm

Fabrizio Carinci

OECD Patient Safety Indicators 2015

Fabrizio Carinci

Performance Dashboards: AustraliaSource: OECD Health at a Glance 2015http://www.oecd.org/health/health-systems/health-at-a-glance-19991312.htm

Fabrizio Carinci

Emerging international issues in public reporting

The ability to report on health systems performance is directly related to the capacity of the information infrastructure (eg possibility to perform data linkage at varying degrees of complexity)

The level of implementation may also depend from specific cultural, organizational and political conditions that can vary both within and across countries

A range of OECD studies show that the applicability of standardized definitions is still limited: more efforts are required

Public reporting as a cost effective solution can hardly be demonstrated in general at regional/state/national level

Fabrizio Carinci

Ketelaar NA, Faber MJ, Flottorp S, Rygh LH, Deane KH, Eccles MP. Public release of performance data in changing the behaviour of healthcare consumers, professionals or organisations. Cochrane Database Syst Rev. 2011 Nov 9

Fung CH, Lim YW, Mattke S, Damberg C, Shekelle PG. Systematic review: the evidence that publishing patient care performance data improves quality of care. Ann Intern Med. 2008 Jan 15;148(2):111-23.

Berwick DM, James B, Coye MJ. Connections between quality measurement and improvement, Med Care. 2003 Jan;41(1 Suppl):I30-8.

Selection pathway Limited evidence

Change pathway Good evidence for hospitals No evidence for individual

providers Scant evidence on improved

clinical outcomes

Is Transparency evidence-based?

Fabrizio Carinci

Why public reporting?

So, why doing public reporting? To implement legislation passed by Parliaments around the world

to achieve higher levels of transparency “no matter what” to respect the citizens' rights to be informed

To achieve greater efficiency by making all public servants directly accountable for their performance

To support pay for performance schemes that can routinely use open benchmarking for continuous quality improvement

To provide the basis (eg hospital reporting system) for specific solutions that can be proved to be cost effective in selected clinical areas or for targeted population subgroups (RCTs)

To improve data quality through a continuous relation with a prurality of users

Fabrizio Carinci

OECD HCQI Bureau Meetings

Rome, ITALY - Friday 26th September 2014

OPPORTUNITIES AND CHALLENGES OF HOSPITAL PERFORMANCE PUBLIC REPORTING AT THE NATIONAL LEVEL: INTERNATIONAL EXPERIENCES AND FUTURE PERSPECTIVES

International workshop

organized by: Italian National Agency for Regional Health Services (AGENAS) , Italian Ministry of HealthProgetto Mattone Internazionale

Seoul, KOREA - Thursday 17th September 2015INTERNATIONAL SYMPOSIUM ON EXPERIENCES OF HOSPITAL PERFORMANCE MEASUREMENT OF OECD MEMBER COUNTRIESOrganized by: Health Insurance Review and Assessment Service (HIRA)

Fabrizio Carinci

Canada: Health Systems Performance Framework

Fabrizio Carinci

Canada: Interactive OECD tool

Fabrizio Carinci

Canada: benchmarking Provinces vs Canada OECD variation of HCQI

Fabrizio Carinci

UK: NHS Choices

Fabrizio Carinci

UK: NHS Consultant choice

Fabrizio Carinci

UK: Funnel Plots Risk-adjusted in-hospital mortality after elective abdominal aortic aneurysm repair: surgeon figures in comparison to national average

Source: National Vascular Registry. 2013 Report on Surgical Outcomes. Consultant level Statistics. June 2013 c

Fabrizio Carinci

Italian National Outcome Programhttp://95.110.213.190/PNEed14_EN/index.php

Outcome measures by Hospital/Local Health Unit

Audit tools

Report Card by Hospital/Local Health Unit

ER Information System

Pilot studies

Fabrizio Carinci

Italy: National Outcomes Program (PNE)

Fabrizio Carinci

Denmark: sundhed.dk

Fabrizio Carinci

Denmark: Danish Quality Registries

Data entryData entry

Clinical observations

Nationalaudit

Regionalaudit

Dataanalysis

Database

Quality improvement

Detailed feedback:Once per year

Standard feedback:Monthly

Results made publically available

Fabrizio Carinci

Korea: decreased variation of AMI quality score

Fabrizio Carinci

Dartboard diagram

To visually represent the results of the six areas, each organization is presented with a specific “target” diagram, divided into five assessment bands.

The closer the organization is to the target level of each performance indicator, the nearer the relative circle will be to the centre.

This approach is nice, but it has known methodological limitations and interpretative drawbacks. There is no perfect graphical representation!

Fabrizio Carinci

Exchanging best practices for fair, transparent and effective communication to the public

PERSONAL JUDGMENT

LEAGUE TABLES SCIENCE

Fabrizio Carinci

Why international collaboration To share best practices that would improve:

understanding of the methods interpretation of national results and interregional variability through

common benchmarks understanding of data limitations and targeted strategies to

strengthen the information infrastructure and overall data quality uptake of effective strategies for quality and outcomes improvement

To gain efficiency through fostered international cooperation, allowing to share costs and gain access to high level skills/tools that are difficult to obtain at national level

To help defining the “Essential Levels of Health Information” for the definition of international standards of performance evaluation

Fabrizio Carinci

Essential levels of health information in EuropeCarinci F. Health Policy. 2015 Apr;119(4):530-8.

Recently, the European Commission coordinated a discussion between MS and Expert Networks on how (and whether) to realize a EU health information system.

The conceptual framework for the „Essential Levels of Health Information” originated from this experience

Fabrizio Carinci

Essential levels of health information in EuropeCarinci F. Health Policy. 2015 Apr;119(4):530-8.

Fabrizio Carinci

Open questions

Can system-level indicators be used at provider level? How can a system-oriented performance framework be adapted to hospitals or specific providers? Which OECD quality indicators can be adopted for hospital reporting?

How can we identify international benchmarks for continuous hospital performance reporting? Which indicators shall be routinely communicated to the public? How to share relevant data, and to which level of detail for reliable analyses?

Which countries shall be routinely compared (EU, OECD, universal/insurance, or any rule to define “peer countries”)?

How to communicate results to the public (best practices, new approaches)? Is public reporting effective, sustainable and practically convenient? Which forms of new communication and how can citizens contribute to the process?

Lessons from the world......your turn!

Fabrizio Carinci

ForumAgency for Clinical InnovationSydney, AustraliaWednesday, 24th February 2016

Thanks for your attention!

the revised oecd health systems performance framework: methodological issues and implications for...

Health & Medicine