cihi’s performance measurement framework journey: the ... › wp-content › uploads › 2017 ›...
TRANSCRIPT
@NHLC2018 #NHLC2018
CIHI’s Performance Measurement Framework Journey: the challenges and rewards
Presenters: Sandra Mitchell, Manger, Governance and Strategy, CIHI Karen Weir, Program Lead, Governance and Strategy, CIHI
2 2
CIHI’s PMF Journey
• PMF Overview
• PMF Indicator Development
• Reporting
• Challenges and Rewards
• Next steps in the Journey
Achieve Results: Learning Objective: Learn how an effective performance measurement framework can facilitate accountability and lead to change.
3 3
CIHI
• Created in 1994
• Independent, not-for-profit organization that provides essential information on Canada's health systems
• Led by a 16-person Board of Directors, with representation from across the country
4
CIHI’s PMF Journey
Strategic Plan Performance
Measurement Framework
Logic Model Indicators Targets Reporting
5 5
PMF Overview
6
CIHI’s Strategic Plan 2016 to 2021
7 7
Purpose of the Performance Measurement Framework
• Guide the measurement of CIHI’s progress in achieving its 2016-2021 Strategic Plan
• Tool provides senior leadership with concrete data and information to:
‒ make sound decisions
‒ continuously improve performance
8 8
Key Features
• Aligned with Strategic Plan
• Demonstrating results – impact and outcomes
• Measuring actionable elements
• Based on accurate and reliable performance data
• Consistency with Treasury Board / Health Canada requirements
9 9
Process to develop PMF and Logic Model • Steering Committee (January – March 2016)
‒ Executive Team, Directors, Governance and Strategy
• PMF was developed through a series of 8 working sessions
‒ CIHI profile
‒ Logic model
‒ Identification of the associated PMF indicators
‒ measuring achievement against the Annual Plan
10 10
CIHI’s Logic Model
11
Overview
12 12
Activity Stream 1: Standards and Quality Data Activity/Box 1: Develop data standards and perform
quality assurance • Obtain consensus on health information standards
• Develop standards in priority areas and support their adoption
• Increase timeliness of data receipt and exchange
Activity/Box 2: Acquire/collect data and develop/implement data access • Acquire/collect data in priority areas
• Implement data access strategy
• Develop flexible data collection tools streamlining the data collection process
Output/Box 7: Data and Standards • Enhanced and optimized library of data standards and enhanced data
quality tools and assessments
• Accessible data sources in priority areas, enhanced coverage of existing data holdings
13 13
PMF Indicators
• Developed 19 PMF indicators
‒ 11 PMF indicators reported to the Board • Specific indicators developed for each output and outcome
‒ Output indicators: largely “counts” of completed products
‒ Outcome indicators: measure stakeholder use, access, and behavioural changes as a result of CIHI data and products
14 14
A Few PMF Indicators
• Output
‒ % of analyses released that align with priority population themes
• Immediate Outcomes
‒ Increase in access to CIHI’s public data
• Long-Term Outcomes
‒ Extent to which CIHI has contributed to health system improvements
15 15
Developing PMF Indicators
16 16
Process to develop PMF Indicator Measures • Working Group established November 2016
• Managers and Directors
‒ Chaired by Governance and Strategy
• For each indicator
‒ Determine scope (products and services)
‒ Define appropriate measures
‒ Identify data sources and frequency of reporting
‒ Identify targets
17 17
Targets
• Target setting occurs during Annual planning process
• Set by Executives
• May cross divisions
• Targets shared with the Board’s Governance and Privacy Sub-Committee
18 18
Reporting • Semi-annual data collection for select indicators
• Annual data collection for all indicators
• Data sources
‒ CIHI administrative data
‒ Analytical plan
‒ External stakeholder satisfaction survey
‒ to support 4 indicators based on stakeholder feedback
‒ Vignettes
19 19
Long-term Outcomes • Indicators:
‒ Extent to which CIHI has contributed to pan-Canadian population health
improvements
‒ Extent to which CIHI has contributed to health system improvements
• Track long-term outcomes on how CIHI’s products and services are contributing to population health and health system improvements
• CIHI follows up with stakeholders to learn how its products and services are used to identify an opportunity for improvement
• Triggers an implementation plan to improve an outcome
• Subsequent product and service reporting demonstrates improvements
20 20
Your Health System (YHS): Helping a Hospital focus on key quality-
of-care initiatives
• YHS: In Depth, In-Hospital Sepsis indicator 2014-2015
• Results for In-Hospital Sepsis indicator different from their expectations, from peer, regional and provincial comparators
• Urinary tract infections (UTIs) were the greatest contributors to the In-Hospital Sepsis results
• Goal: To decrease surgical catheter–associated UTIs from 2.5% to 1.7% by April 2016.
• Action: Implemented an initiative to reduce the number of catheter insertions and the number of catheter days
• Outcomes: UTI rate dropped from 2.5% to 1.5%
21 21
Challenges and Rewards
22 22
Challenges of Developing the PMF and Logic Model
• Not the natural ‘day job’ of executives or managers
• Discussions with Health Canada resulted in changes along the way
• Took more time than anticipated
23 23
Developing the Indicator Definitions and Indicator Measures
• People who developed the indicators were not at the table to develop the measures
• Managers often had more insight on what could or could not be measured
‒ In some cases this led to the rephrasing and re-scoping of the indicators
• Indicators created based on what needs to be measured
‒ i.e. not based on what data is readily available
24 24
Collecting and Capturing Indicator Data
• Staff responsible for providing the data were not involved in the development of the indicators or the measures
‒ therefore did not understand data requirements or rationale
‒ not the natural ‘day job’ for staff,
‒ concept of Performance Measurement is not intuitive
• Request for data to numerous departments and numerous staff
• Conflicting data between departments for the same measure
• Provision of data particularly arduous as departments had not previously been asked to track data in the format required for the indicators
25 25
Reporting on Indicator Results
• Choosing what to highlight
• Ensuring that the results have been explained appropriately
• Challenge to explain why there are inconsistencies in targets and actuals achieved
26 26
Rewards
• Year one (2016-2017) reported:
‒ 14 out of 19 indicators
‒ 1 long term outcome
‒ 6 out of 13 indicators were meeting or exceeding targets
• Year two (2017-2018) reported:
‒ 18 out of 19 indicators
‒ 3 long term outcomes and tracking many more
‒ 9 out 13 indicators were meeting or exceeding targets
27 27
Take Away
• Different way of thinking and new skill set for the organization
• Developed new indicator measures where there are no pre existing data collection methods in place
• Set up as a project – can’t be done in a short timeframe off the side of people’s desk
• Identification and capture of performance measurement indicator measures is an evolving process
28
Next Steps in the CIHI PMF Journey
Strategic Plan Performance
Measurement Framework
Logic Model Indicators Targets Reporting
cihi.ca cihi.ca @cihi_icis [email protected] @personal twitter handle
31
Performance Indicators - Outputs Area Indicator Frequency
Data and standards # of planned new and major standard enhancements Annual
# of planned data quality assessments completed by CIHI Annual
Increase (%) in coverage of data collection in priority areas * Annual
Analytical tools and methods
% of planned new and enhanced analytical tools completed by CIHI
Annual
% of planned health information infrastructure enhancements completed within the year
Annual
Health information infrastructure
% of planned health information infrastructure enhancements completed within the year
Annual
% of data sources included in integrated e-reporting (IeR) against the 2021 target
Annual
Analytical products % of analyses released that align with priority population themes*
Annual
Knowledge exchange / capacity development
# of capacity development events or activities Annual / semi-annual
32
Performance Indicators – Immediate Outcomes Area Indicator Frequency
Increased access to quality, integrated data
Increase in access to CIHI’s public data* Semi-annual
% improvement in the quality of the data Annual
# of linked data files available through third parties*
Annual
Increased access to analytical tools and products
Level of stakeholder satisfaction with access to and usefulness of tools and products*
Periodically
Increase in total number of users of CIHI’s private online tools/products*
Semi-annual
Increased stakeholder knowledge and capability to use products and services
% of stakeholders (target group) reporting increased knowledge (awareness) of CIHI products and services in their setting*
Periodically
% of stakeholders (target group) reporting increased capability to use CIHI products and services in their setting*
Periodically
33
Performance Indicators – Intermediate and Long-Term Outcomes
Area Indicator Frequency
Intermediate: Increased use of CIHI products and services to support decision making
% of stakeholders who report using evidence from a CIHI knowledge product or service to support decision-making in their setting*
Periodically
Long-term: Contribution to pan-Canadian population health and health system improvements
Extent to which CIHI has contributed to pan-Canadian population health improvements*
Annual
Extent to which CIHI has contributed to health system improvements*
Annual
*Indicates PMF indicators that are reported to the CIH Board of Directors