The performance evaluation system of a network of Italian Regions: challenges and results
Prof. Sabina Nuti, Laboratory of Management and Healthcare, Institute of Management, Scuola Superiore Sant’Anna, Pisa (Italy)
Scuola Superiore Sant’Anna is a public University of excellence,specialized in applied sciences: Economics, Law and Political Science inthe Social Sciences Department; Agrarian Sciences, Medicine, andIndustrial and Computer Engineering in the Experimental SciencesDepartment
Scuola Superiore Sant’Anna
http://www.santannapisa.it/en
Undergraduate students are admitted to theSant'Anna School of Advanced Studies by anational public competition.
Winners are enrolled in the appropriate degreecourse of the University of Pisa and they haveto attend the integrative courses of the School:all these services are free of charge.
Scuola Superiore Sant’Anna offers and oversees quality undergraduate, graduate and continuing education
The Management and Health Laboratory was formally established in December 2004 and now is part of the Management Institute
MeS - Management and Health Laboratory
Research EvaluationTraining
A dynamic learning community……to develop knowledge, share information, build innovation for the
public health system
The Premises:The Italian Healthcare System
It is a Beveridge-like model: Universal, Comprehensive (almost), Free, Financed by General Taxation
It is organized in three levels:
– The national level is responsible for general aims and annual financial resourcesand for ensuring a uniform level of services, care and assistance (LEA)
– The regional level has the responsibility for planning, organizing andmanaging its health care system through LHA’s activities in order to meet theneeds of their population
– The local level (Local Health Authorities): provides care through public and/orprivate hospitals, primary care and prevention services
with a Public University guarantying
the benchmarking process…
Measuring and benchmarking
performance among Regions…
on a voluntary basis …
With data public disclosure…
Engaging health professionals in the process
setting targets and priorities…
The Starting point: 2004 Tuscany, 2008 the Network … 2016… 13 Regions
270 indicators in total
6 areas
40 index indicators
100 evaluation indicators
Structure of the evaluation system
7
Efficiency and financial
performance
Employees Satisfaction
Patients Satisfaction
Clinical performance
Capacity to pursue regional strategies
Population health status
The multidimensional reporting system shared by the network of the Italian regions
In order to describe the performance evaluation system, six areas have beenidentified to highlight the core results of the regional healthcare system.
Performance reporting
areas
Outcome
The Reference Criteria for the Assessment Bands
1. International standards, if existing based on EBM
2. Regional standards set out by the Regional Governments
3. The inter-regional mean, standardized by several factors to allow
comparisons across Health Authorities and Regions
To visually represent the results of the six areas, each Region & Health Authority has a personal “target” diagram, divided in five assessment bands.
The more the Region/Health Authority is able to reach objectives and obtain good results in each of the six areas, the nearer the performance indicator is to the centre.
The DARTBOARD!
Y Axis:
Capacity to improve. Itis the difference betweenthe performance of twoconsecutive years. It iscalculated giving a rangebetween -2 and +2:
-2: highest worsening
+2 highest improvement.
The performance map
X Axis: Performance as the actualevaluation score between 0 and 5
Excellent
Warning
On the right
path
Keep an
eye onIm
pro
vem
ent
Wors
enin
g
Poor Good
The Performance evaluation system is able to drive improvement...
But is it also able to achieve equity?
Vertical: “not equal parts among unequals” (don Lorenzo Milani)
Horizontal: citizens with same needs require same level of answer
Intergenerational: young people should have the right to the samelife and health opportunity of the old persons
Is there more room to reduce waste, and have more savings and
value for money?
The complementary representation of performance
Performance
Trend
Reducing
Avoidable
Variation
The
performance
map
The
variability
pie
29
The variability pie
Excellent: Improvement
and reduction of
variability
On the right way:
Improvement and
increase of variability
Stability
Keep an eye on:
Worsening and
increase of variability
Warning: Worsening
and reduction of
variability
Variability within the Region. The
units are the Healthcare organizations
(Local Health Authorities and
Teaching Hospitals)
Nuti S., Seghieri C, Vainieri M. Assessing the effectiveness of a performance evaluation system in the public health care sector: some novel
evidence from the Tuscany Region experience. Journal of Management and Governance online first 2012. DOI 10.1007/s10997-012-9218-5
QUALITY OF CARE
SUSTAINABILITY
EQUITY
Pinnarelli L., Nuti S,Sorge C, Davoli M.Fusco D,Agabiti N, Vainieri M, Perucci CA, 2012
What drives hospital performance? The impact of comparative outcome evaluation of
patients admitted for hip fracture in two Italian regions.BMJ Quality and Safety Vol.2
Strategies and results…
PES integrated
with other
management tools
drives
improvement
Plot Per Capita Cost vs % Overall Performance (Tuscany, 2007-2013)
Correlation coeff. = - 0.38
P < 0.001
140
01
60
01
80
02
00
02
20
0
co
sto
pro
-ca
pite
0 20 40 60 80performance
MORE QUALITY …
MORE FINANCIAL
SUSTAINABILITY
1.5
22
.53
3.5
perf
orm
ance
med
ia
1400 1600 1800 2000 2200 2400costo sanitario pro - capite
SAME RESULT FOR
ALL THE NETWORK
When Benchmarking, Public disclosure and Evalution process are in place
38
You can implement the following strategies
Evaluation Governance
3) Coping with avoidable variation: reallocation strategies
Reducing variability amongproviders…
Tuscany health system couldre-allocate about 7% of its
financial budget
All the regions in the network were able to improve?
Nuti S., Seghieri C, Vainieri M. Assessing the effectiveness of a performance evaluation system in the public health care sector: some novel
evidence from the Tuscany Region experience. Journal of Management and Governance online first 2012. DOI 10.1007/s10997-012-9218-5
QUALITY OF CARE
SUSTAINABILITY
EQUITY
Policy
makers
Citizens
patients managers
Health
professionals
SOCIAL
PACT
There are only three common elements to healthcare systems that have improved on cost and quality over time
1. A public, specific statement of goals for improvement with a plan for reaching these goals
2. Public reporting of results with a clear link to improvement plans that become part of the strategy
3. Strong physician & clinical leadership of improvement efforts aligned to improvement goals, again supported by useful data
Strategies that fail to include these elements will fail an appeal to intrinsic incentives
Brown AD, Baker GR, Closson T, Sullivan TS. The journey towards high performance and excellent quality. Healthcare Quartely, 2012;15.
Regions taking care for population health should use this approach to reduce unwarranted variation, improve
and achieve equity, achieve sustainability
Nuti, S., Seghieri, C., & Vainieri, M. (2013). Assessing the effectiveness of a performance
evaluation system in the public health care sector: some novel evidence from the Tuscany
region experience. Journal of Management & Governance, 17(1), 59-69
Nuti, S., Vola, F., Bonini, A., & Vainieri, M. (2015). Making governance work in the health care
sector: evidence from a ‘natural experiment’in Italy. Health Economics, Policy and Law, 11(01),
17-38.
Vainieri M., Vola F., Gomez Soriano G., Nuti S. (2016), “How to set challenging goals and
conduct fair evaluation in regional public health systems. Insights from Valencia and Tuscany
Regions”, Health Policy in press
Nuti S; Seghieri C (2014) Is variation management included in regional healthcare governance
systems? Some proposal from Italy. Health Policy vo.114
Nuti S. Vainieri M, Bonini A (2010) Disinvestment for reallocation: a process to identify priorities
in healthcare. Health Policy. Vol95
Nuti S. Vainieri M (2014) Strategies and tools to manage variations in regional governance
systems. Handbook on health services research Vol 1 Springer
Nuti S. Vola F. Vainieri M. (2017) Priorities and targets: a methodology to support the policy-
making process in healthcare. Public money and management, forthcoming
Nuti S., Grillo-Ruggieri T., Podetti S. (2016), Do university hospitals perform better than general
hospitals? A comparative analysis among Italian regions, BMJ Open 2016;6:e011426.
doi:10.1136/bmjopen-2016- 011426
2014 Report Evaluating the network healthcare system performance, accessible online
www.meslab.sssup.it/en/
46
Selected bibliography