implementation of quality indicators: barriers and facilitators

Implementation of quality indicators: Implementation of quality indicators: barriers and facilitatorsbarriers and facilitators

Peter HJ van der Voort, MD, PhD, MSc

Dept of intensive care

Onze Lieve Vrouwe Gasthuis

Amsterdam, The Netherlands

ContentContent

• Indicators and quality improvement

• The Dutch Intensive Care Registry

• Barriers to implementation of QI

• Facilitators

• Implementation strategies

• From indicator to improvement of care– InFoQi study with the Dutch indicator set

Dutch National Intensive Care Dutch National Intensive Care Registry (NICE)Registry (NICE)

Benchmark of outcome

• SMR

• length of stay

Benchmark of outcome

• SMR

• length of stay

• How to improve?

Dutch Society of Intensive CareDutch Society of Intensive Care

• 2003-2004 development of an indicator set to analyse quality of care

• Based on quality domains of IOM

National Guideline On Intensive National Guideline On Intensive Care OrganisationCare Organisation

• 1993, revised 2005, implemented 2006

• Section on quality improvement:

• Quality Indicator set NVIC incorporated in guideline

• No indicators to follow implementation of the guideline

* Indicators

How to use Quality IndicatorsHow to use Quality Indicators

• E.g. 2006 Guideline on organisation advises to have regional partners for collaboration (volume – outcome relation). To discuss individual patients.

• Indicator: % patients discussed from total admitted

How to use Quality IndicatorsHow to use Quality Indicators

The Dutch Quality Indicator SetThe Dutch Quality Indicator Set

• Cooperation between NICE and NVIC

• Set developed by NVIC

• Benchmark by NICE

• Pilot: registration workload, definitions

• Active promotion to Dutch ICU’s

Implementation of the indicator set appeared to be a QI plan by itself

Implementation of the indicator set appeared to be a QI plan by itself

Problems– Create the sense of urgency to use indicators– How to implement the registration of

indicators– How to feed-back– How to make conclusions– How to implement changes

Definition of quality

indicators

AIRE

Quality Domains

AdoptionDiffusion

Dissemination

• Purpose, relevance and organisation where the indicator appoints to

• Involvement of professionals• Scientific evidence• Additional reasoning and use

Definition of quality

indicators

AIRE

Quality Domains

PDSA

Avedis Donabedian 1919-2000

o Structure:o Organisation, resources and equipment

o Process:o Process of care between caregiver and patient

o Outcomeo Results (at patient level)

Dutch indicators - internalDutch indicators - internal•Availability of intensivists

•Nurse to patient ratio

•Policy to prevent medication errors

•Registration of patient- and family satisfaction

Structure

Process •Length of stay in the ICU

•Duration of ventilation

•Number of days with 100% beds occupied

•Glucose regulationOutcome•Mortality•Incidence of pressure sores•Unplanned extubation

Definition of quality

indicators

AIRE

Quality Domains

PDSA

Decision to use a set of indicators

Organize and implement registration of data

Data validity

Data export

Analysis and benchmark (NICE)

Feedback

Interpretation and conclusion

Plan for change

Implementation of changes / new methods

Barriers on all levels

Decision to use a set of indicators– Sense of urgency– Intrinsic motivation to improve– Legislation– Hospital directors– Society of Intensive Care– Convince that using indicators

improve care

What we did:

Inform, offer tools

Decision to use a set of indicators

Organize and implement registration of data

Data validity

Data export

Analysis and benchmark (NICE)

Feedback

Interpretation and conclusion

Plan for change

Implementation of changes / new methods

• Intervention study: InFoQi to improve QI using indicators

• 3 interventions:– Extensive feedback– QI team– Educational Outreach

• To develop an optimal intervention– Literature search on optimal feedback– Search for barriers in literature, expert

groups, questionnaire

Definition of quality

indicators

AIRE

Quality Domains

PDSA

Implementation strategiesImplementation strategiesusing indicator datausing indicator data

Educational meetingEducational outreachAudit and feedbackDevelopment of a quality improvement planFinancial incentives

Supporting activities:– Distribution of educational material– Use of a local opinion leader– Development of a quality improvement team

Educational meetingEducational meeting

Participation in conferences, seminars, lectures, workshops or training sessions.

During these meetings, feedback of quality indicators is presented, and study participants discuss how to improve performance.

Educational outreachEducational outreach

A trained independent person or investigator meets with health professionals or managers in their practice setting to provide information (e.g. feedback of quality indicators).

Development of a quality Development of a quality improvement planimprovement plan

A plan based on indicator data to be used to improve the quality of care.

Financial incentivesFinancial incentives

Rewarding individual health professionals or institutions with higher payments when they improve performance.

Audit and feedbackAudit and feedback

Giving a report, including a summary of clinical performance over a specified period of time.

“any summary of clinical performance over a specified period of time”

“It is striking how little can be discerned about the effects of audit and feedback based on the 118 trials included in this review.”

“Low baseline compliance with recommended practice and higher intensity of audit and feedback were associated with larger adjusted risk ratios (greater effectiveness) across studies.”

Feedback of analysed data

• Jamtvedt et al. reviewed information feedback based on any health care data source, while we focused on feedback based on data from medical registries

• we aimed to include not only RCTs, but any peer-reviewed paper on information feedback within the context of a medical registry. Furthermore, where Jamtvedt et al. only reported on the effectiveness of information feedback, we also aimed to identify the barriers and success factors to this effectiveness as reported in the literature

• 53 papers; 50 feedback initiatives• 24 analytic papers for 22 studies evaluating the

effect of a feedback method on one (n=8) or more (n = 14) primary, clinical outcome measures

• Positive effect on all outcome measures: 4 • Mix of positive and no effects: 8• 10 not any effect. • None of the 22 studies reported a negative

effect.

MFA = multifaceted approach

“To review the literature concerning strategies for implementing quality indicators in hospital care, and their effectiveness in improving the quality of care”

• 21 studies (9 RCT, 2 CCT and 10 B-A)• 17 US; 14 cardiovascular care• 1-379 participating hospitals

• Indicators and hospital care• 20 on process care• 6 on patient outcomes• Follow up 6 months

de Vos et al. Int J Qual Health Care 2008;1-11

de Vos et al. Int J Qual Health Care 2008;1-11

• Study design unrelated to effectiveness

• Results on outcomes: – 4 studies indicators ineffective– 1 partially effective– 1 effective

Effective: > 50% sign improvement; partially effective: +/- 50% improvement; ineffective: <50% improvement

de Vos et al. Int J Qual Health Care 2008;1-11

• Results on process:– 20 studies

– 3 no significant improvement at all– 8 improvement in some– 7 partially effective– 2 significant improvement in all process

indicators

Most are effective on process of carede Vos et al. Int J Qual Health Care 2008;1-11

de Vos et al. Int J Qual Health Care 2008;1-11

Successful implementation of Successful implementation of indicatorsindicators

Feedback reports combined with

Educational implementation strategy

and/or Quality Improvement plan

de Vos et al. Int J Qual Health Care 2008;1-11

BarriersBarriers

Barriers should be identified before an implementation strategy is launched

Facilitating factorsFacilitating factors

• Supportive / collaborative management

• Administration support

• Detailed and credible data feedback

• Ability of persons receiving feedback to act on it

ICU

Barriers “knowledge”:

Factors limiting adherence through a cognitive component

Barriers “attitude”:

Factors limiting adherence through an affective component

Barriers “behavior”

Factors limiting adherence through a restriction of physician ability

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

11) Monitoring of quality indicators does not take too much time (n=139)

10) Monitoring of quality indicators can be done without huge investments (n=140)

9) Monitoring of quality indicators fits into the daily routines in the hospital setting (n=140)

8) Monitoring of quality indicators leads to reliable benchmark data for ICUs (n=140)

7) Monitoring of quality indicators stimulates quality improvement (n=140)

6) Feedback on quality indicators stimulates me to adjust my practice (n=142)

5) I am willing to implement quality indicators in daily practice (n=141)

4) In general, I do not offer resistance towards working with quality indicators (n=142)

3) I understand the importance of using quality indicators (n=142)

2) I am familiar with the Dutch set of ICU quality indicators (n=142)

1) I am familiar with the use of quality indicators as a tool to improve quality of care (n= 142)

Proportion of responding respondents

Strongly disagree

Disagree

Neutral

Agree

Strongly agree

Barrier 1 of 15Barrier 1 of 15

Barrier 1 How the barrier is targeted by the feedback intervention

Lack of knowledge on how to interpret the data

“people are not being taught how to handle the results, how to interpret them.”

Barrier 1 of 15Barrier 1 of 15

Barrier 1 How the barrier is targeted by the feedback intervention

Lack of knowledge on how to interpret the data

During educational outreach visits the facilitators support the QI team in interpreting their performance data in the reports and in formulating a QI action plan

“people are not being taught how to handle the results, how to interpret them.”

Barrier 2 of 15Barrier 2 of 15

Barrier 2 How the barrier is targeted by the feedback intervention

Lack of information to initiate QI actions

“You want to improve the quality, but you don’t know where to start or where the real problems lie”

Barrier 2 of 15Barrier 2 of 15

Barrier 2 How the barrier is targeted by the feedback intervention

Lack of information to initiate QI actions

The feedback reports contain extended information on six of the indicators; During educational outreach visits the facilitators support the QI team in further exploration of data in the NICE registry

“You want to improve the quality, but you don’t know where to start or where the real problems lie”

Barrier 3 of 15Barrier 3 of 15

Barrier 3 How the barrier is targeted by the feedback intervention

Lack of trust in data

“The data are often regarded as unreliable. If you put rubbish in, you will only get rubbish out. Trust in

the data is essential.”

Barrier 3 of 15Barrier 3 of 15

Barrier 3 How the barrier is targeted by the feedback intervention

Lack of trust in data During educational outreach visits the facilitators discuss with the QI team completeness and correctness of the data sent to the NICE registry and -if necessary- support them in formulating actions to improve their data quality.

“The data are often regarded as unreliable. If you put rubbish in, you will only get rubbish out. Trust in

the data is essential.”

Barrier 4 of 15Barrier 4 of 15

Barrier 4 How the barrier is targeted by the feedback intervention

Lack of statistical power for small ICUs

Not targeted by the intervention

“small number of patients lead to broad confidence intervals”

Barrier 5 of 15Barrier 5 of 15

Barrier 5 How the barrier is targeted by the feedback intervention

Lack of case-mix correction

“the ‘my patients are sicker’ syndrome.”

Barrier 5 of 15Barrier 5 of 15

Barrier 5 How the barrier is targeted by the feedback intervention

Lack of case-mix correction Besides already available case-mix corrected hospital mortality data, data are stratified based on admission type or on APACHE IV diagnosis. During educational outreach visits the facilitators support the QI team in formulating additional case-mix related analyses on data in the NICE registry

“the ‘my patients are sicker’ syndrome.”

Barrier 6 of 15Barrier 6 of 15

Barrier 6 How the barrier is targeted by the feedback intervention

Level of aggregation too high

“the care providers need data at practice level, not only at the organisation level.”

Barrier 6 of 15Barrier 6 of 15

Barrier 6 How the barrier is targeted by the feedback intervention

Level of aggregation too high Besides data aggregated on ICU level, the feedback reports contain data on patient or shift level for six of the indicators.

“the care providers need data at practice level, not only at the organisation level.”

Barrier 7 of 15Barrier 7 of 15

Barrier 7 How the barrier is targeted by the feedback intervention

Insufficient timeliness

“…the information might not have been presented close enough to the time of decision making.”

Barrier 7 of 15Barrier 7 of 15

Barrier 7 How the barrier is targeted by the feedback intervention

Insufficient timeliness As the monthly reports do not contain comparisons with other ICUs, it is possible to decrease the time between the end of a period and reporting data on this period from ten (for quarterly reports ) to six weeks (for monthly reports).

“…the information might not have been presented close enough to the time of decision making.”

Barrier 8 of 15Barrier 8 of 15

Barrier 8 How the barrier is targeted by the feedback intervention

Lack of intensity

“…the care providers received prescriber feedback letters only once.”

Barrier 8 of 15Barrier 8 of 15

Barrier 8 How the barrier is targeted by the feedback intervention

Lack of intensity In addition to the quarterly reports, the QI team receives monthly feedback reports containing their performance data presented in a different way.

“…the care providers received prescriber feedback letters only once.”

Barrier 9 of 15Barrier 9 of 15

Barrier 9 How the barrier is targeted by the feedback intervention

Lack of outcome expectancy

“…the current rates were not considered a problem.”

Barrier 9 of 15Barrier 9 of 15

Barrier 9 How the barrier is targeted by the feedback intervention

Lack of outcome expectancy During educational outreach visits the facilitators discuss with the QI team the opportunities for improvement

“…the current rates were not considered a problem.”

Barrier 10 of 15Barrier 10 of 15

Barrier 10 How the barrier is targeted by the feedback intervention

Lack of trust in QI principles

“It is difficult to convince staff to use continuous quality improvement principles.”

Barrier 10 of 15Barrier 10 of 15

Barrier 10 How the barrier is targeted by the feedback intervention

Lack of trust in QI principles The facilitators discuss with the QI team members the principles of systematic QI during the educational outreach visits.

“It is difficult to convince staff to use continuous quality improvement principles.”

Barrier 11 of 15Barrier 11 of 15

Barrier 11 How the barrier is targeted by the feedback intervention

Lack of dissemination of information

“…inadequate dissemination within the hospitals.”

Barrier 11 of 15Barrier 11 of 15

Barrier 11 How the barrier is targeted by the feedback intervention

Lack of dissemination of information Each QI team member receives the feedback reports by e-mail. During educational outreach visits and in monthly reminders they are encouraged to share their findings with the rest of the staff

“…inadequate dissemination within the hospitals.”

Barrier 12 of 15Barrier 12 of 15

Barrier 12 How the barrier is targeted by the feedback intervention

Lack of motivation

“As the intervention was unsolicited, the participants had not agreed to review their practice.”

Barrier 12 of 15Barrier 12 of 15

Barrier 12 How the barrier is targeted by the feedback intervention

Lack of motivation The members of the QI team should be selected based on their affinity and experience with measuring and improving quality of care and their capability to convince staff to be involved in QI activities

“As the intervention was unsolicited, the participants had not agreed to review their practice.”

Barrier 13 of 15Barrier 13 of 15Barrier 13 How the barrier is targeted by the

feedback intervention

Organizational constraints

“Patient care is the main task and [QI activities are] just an extra”

“You will need a change of organizational culture…That will take some time to achieve.”

Barrier 13 of 15Barrier 13 of 15Barrier 13 How the barrier is targeted by the

feedback intervention

Organizational constraints The QI team forms the organizational basis for monitoring performance and initiating QI activities. One of their tasks is formulating a QI action plan corresponding with the opportunities for improvement within their own organization. They are also asked to discuss their performance during monthly QI team meetings, using the available reports and their QI plan as a basis. They are encouraged to report their findings during regular existing staff meetings.

“Patient care is the main task and [QI activities are] just an extra”

Barrier 14 of 15Barrier 14 of 15

Barrier 14 How the barrier is targeted by the feedback intervention

Lack of resources Not targeted by the intervention

“Monitoring of quality indicators takes too much time and money”

Barrier 15 of 15Barrier 15 of 15

Barrier 15 How the barrier is targeted by the feedback intervention

External barriers Not targeted by the intervention

“…there is [a lack of] public awareness now of the need to [improve the quality of care]”

0% 20% 40% 60% 80% 100%

Opinion leader (n=140)

Quality improvement team (n=140)

Education (n=140)

Reminders for registration (n=140)

Receiving feedback (n=140)

Administrative support (n=141)

Possibilities to improve care (n=141)

Social pressure from hospital management (n=140)

Social demand for transparency (n=140)

Encouragement from scientific society (n=137)

Intrinsic motivation (n=140)

Pay-for-performance (n=138)

Rules and policy (n=140)

Proportion of responding respondents

Strongly disagree

Disagree

Neutral

Agree

Strongly agree

Determinants of facilitating Determinants of facilitating factorsfactors

• Administrative support p=0.02 (physicians)

• Education p=0.01 (nurses)

• Feedback p=0.001 (managers)

• Opportunities to improve care p=0.003 (physicians)

Additional facilitatorsAdditional facilitators

• Patient Data Management System / user friendly software

• Appointment of a Quality Manager

• Appointment of one person responsible for coordination

DeterminantsDeterminants• Of knowledge:

– Being manager > health care prof (p=0.004)

– Intensivists > nurses (p=0.01)

– 40-49 yr

– Academic/teaching hosp > non-teaching

• Of attitude

– No significant differences

• Of behaviour

– >49 yr pos related to overall behaviour

– Non teaching neg related

Definition of quality

indicators

AIRE

Quality Domains

PDSA

Decision to use a set of indicators

Organize and implement registration of data

Data validity

Data export

Analysis (NICE)

Feedback

Interpretation and conclusion

Plan for change

Implementation of changes / new methods

• Intervention study: InFoQi to improve QI using indicators

• 3 interventions on identified barriers:– Extensive feedback– QI team– Educational Outreach

• Multicenter cluster randomised trial

• From October 2008 – October 2010

• 30 ICU’s

Definition of quality

indicators

AIRE

Quality Domains

PDSA

Conclusions (1)Conclusions (1)Quality Indicators should meet strict criteria

The implementation of Quality Indicators should follow the PDSA cycle

The effect of implementation of QI has not been studied in ICU’s

Conclusions (2)Conclusions (2)

To achieve effect a multifaceted

approach is needed

based on pre-defined barriers

Nicolette de Keizer Gert Westert Nice participants

Sabine vd VeerMaartje de Vos

• Implementation policy– National– Hospital– ICU

• Support• QI implementation team

– multidisciplinary

Implementation of indicators

Decision to use a set of indicators

Organize and implement registration of data

Data validity

Data export

Analysis and benchmark (NICE)

Feedback

Interpretation and conclusion

Plan for change

Implementation of changes / new methods

• Who– distribute

• When– Part of the care process– Daily workflow– Responsibility

• How– Paper / PDMS

Registration of raw data

• Complete

• Reliable

• Validation

Registration pitfallsRegistration pitfalls

• Definitions– NICE datadictionary

• Erroneous measurements– Room temp

• Inappropriate data collected– Wrong time period

• Extractions to export file– Default values

• Decision to use a set of indicators• Organize and implement registration of data

• Responsible persons for parts of the registration on daily basis

• Overall responsibility for one person

• Data validity– Data quality checks every year locally– Data quality reports monthly

• Data export– We do not offer a ICT tool for export– Close collaboration with providers

• Decision to use a set of indicators

• Organize and implement registration of data

• Data validity

• Data export

• Analysis (NICE)– Clinical Information Department in an

academic hospital

• Decision to use a set of indicators

• Organize and implement registration of data

• Data validity

• Data export

• Analysis (NICE)

• Feedback

Studies on barriers?Studies on barriers?

Problems in: • gathering support

• personnel, management• disseminating information• registration / technical / political problems

“% of hospital staff and % of senior managers participating in formally organized QI teams are associated with better values on quality indicators. Percentage of physicians participating in QI teams is not associated with better values”

“41% felt that monitoring programs did not assist them in improving care. Providers cited numerous barriers to improving care processes.”

Am J Psychiatry 2004; 161:146–153