A methodology to develop quality indicators for Health IT

Nicolette de Keizer, Elske Ammenwerth, Hannele Hyppönen

on behalf of the EFMI and IMIA WG on Evaluation

Program

Welcome and introduction Nicolette de Keizer

eHealth indicator development

Hannele Hyppönen

Method for developing quality indicators

Nicolette de Keizer

Case study on CPOE systems

Elske Ammenwerth

Group work

Summary and next steps Michael Rigby

Background

EU and WHO: ICT solutions in health care are prerequisites for modern, patient-centered and efficient health care services

Potential benefit of health ICT is widely endorsed, but software used in health care is not necessarily inherently safe

ICT interventions should therefore be thoroughly evaluated before wide scale implementation

Thorough evaluation

Dissemination of successful ICT interventions is only possible if quality and success of the ICT intervention can be measured

-> Quality Indicator (QI) a measurable element of a system for which there is

evidence or consensus that it can be used to assess a defined aspect of the system in question

Problem

There are so many health IT systems National infrastructures for EMR Local EMRs Decision Support Systems CPOE systems Telemedicine applications PACS …….

Some QIs can be shared, others are specific

Aims of the workshop

to inform on and discuss a methodology for developing QIs for health IT systems

to draw up the interest within the community to develop QI for health IT systems

to recruit groups that will develop QIs for a specific type of health IT systems

Program

Welcome and introduction Nicolette de Keizer

eHealth indicator development

Hannele Hyppönen

Method for developing quality indicators

Nicolette de Keizer

Case study on CPOE systems

Elske Ammenwerth

Group work

Summary and next steps Michael Rigby

eHealth indicator development

Definitions eHealth: A review in 2005 included 36 different

definitions for the term e-health. - Using, processing, sharing and controlling health related information in electronic format who, for what purpose, how?

Indicator: a measurable element of practice or system for which there is evidence or consensus that it can be used to assess a defined aspect of the practice or system in question

Why are eHealth indicators needed? to learn from initiatives, to see if they are worth the money, to

make informed decisions needed in management of eHealth on local and national (international) level

State of the art:eHealth indicator-work is lagging behind, indicators ambiguous or

missing. Methodologies not transparent, gaps in existing classifications

used to group eHealth indicators, gaps in indicators proposed by the experts - need for a more formal generic methodology to define indicators

Development of eHealth indicators Literature on indicator methodology

Articles discussing suitability of specific indicators as measures in a specific field (e.g. drug treatment data as an epidemiological indicator)

Articles describing a methodology for defining indicators for a specific assessment topics (e.g. care process quality indicator)

Articles describing a framework for defining indicators for specific policy goals (e.g. eHealth or sustainable development)

Category 3 was most suited as a starting point for creation of a generic methodology (common for different topics)

Ge

ne

ric

-

s

pec

ific

Two main approaches for indicator definition Expert-led Top-down methodology: predominant in fields

focusing on monitoring implementation of policies and their impact on society level (e.g. economic growth, main aim also in European level eHealth indicator work). .

Bottom-up methodology: predominant in the fields aiming to monitor or assess policy or strategy implementation and impacts on micro level – e.g. on local environment. indicators tailored to the needs and resources of the indicator

users, still rooted in the policy in question (e.g. sustainable development in environmental policy)

Common phases to proceed Defining the context (human and environmental) for

measurement: key stakeholders and relevant systems. For whom, about what, what for? Defining systems OR key functionalities needed for core tasks?

Defining the goals. Top–down approaches rarely include this, the goals are pre-determined by government offices or policies. Whose goals, which goals for which stakeholders?

Defining methods for indicator selection and categorization by reviewing existing indicator work, expert knowledge, peer- reviewed literature, selection of key indicators per purpose

Defining the data. This step tests the indicators by applying them. Initiative data are collected, analyzed, reported and feedback is acquired from different user groups.

Questions

system vs functionality-based indicator work (OECD eHealth indicators are functionality-based, much of the existing indicator data is collected system-based)

Need for transparency of goals, stakeholders and use purposes.

Program

Welcome and introduction Nicolette de Keizer

eHealth indicator development

Hannele Hyppöne

Method for developing quality indicators

Nicolette de Keizer

Case study on CPOE systems

Elske Ammenwerth

Group work

Summary and next steps Michael Rigby

A methodology to develop quality indicators for Health IT

Background

RAND cooperation – methodology for clinical QI developmentweak reliability of the rating and consensus

procedures modified by Van Engen et al [MIE2011] We adjusted this ‘modified RAND method’

to the context of quality of health IT

Methodology

1. Expert and stakeholders panel

2. Literature research

3. Review of guidelines

4. Inductive content analysis: Draft set of QI

5. Individual rating

6. Group discussion and anonymous voting

7. Define target standards (repeat step 1-6)

Expert and stakeholders panel

Search for representatives of stakeholders of the system: developers, researchers, users

Clear invitation goal of the questionnaire and a response date their involvement in the rest of the project etc.

Web-based survey (LimeSurvey) on QI subtasks or dimensions of the system UMIT support development survey

Literature research

Search terms concerning the field of interest (e.g. CPOE) combined with MeSH terms and keywords referring to ‘effectiveness’, ‘assessment’, ‘outcome’, ‘quality assurance’ or ‘quality indicators’, ‘evaluation’ or ‘monitoring’.

Use the evaluation inventory available from http.//evaldb.umit.at to

Review of guidelines

Clinical guidelines important for clinical QIdesirable level of outcomes of careminimum procedures, standards and facilities

that services should include (ISO/CEN) standards in the field of interest

Inductive content analysis Combine results of the three sources Indentify themes/concepts

“I like to know how much time it cost to enter a patient consult”

1. Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care: Trends in evaluation research 1982 - 2002. Methods of Information in Medicine. 2005;44:44-56

Inductive content analysis Combine results of the three sources Indentify themes/concepts Classify QIs:

http.//evaldb.umit.at 1

structural quality information logistics quality effects on quality of processes effects on outcome quality of care

Delone and Maclean Information quality System quality Use User satisfaction

Further fine-grained classifications UMIT support inductive content analysis

1. Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care: Trends in evaluation research 1982 - 2002. Methods of Information in Medicine. 2005;44:44-56

Individual rating

Likert scale from 1 (total disagreement) to 5 (total agreement): Importance Actionability Easy to record or obtain

Rank on mean score

Group discussion and anonymous voting Consensus in expert panel through rounds

of debate, discussion and an anonymous voting process Face to face meetings web-based chat meetings

Define target standards

Repeat step 1-6 to obtain target values for each of the review criteriaQI: time to enter one patientValue: < 1 minute

Program

Welcome and introduction Nicolette de Keizer

eHealth indicator development

Hannele Hyppöne

Method for developing quality indicators

Nicolette de Keizer

Case study on CPOE systems

Elske Ammenwerth

Group work

Summary and next steps Michael Rigby

CPOE

Computerized Physian Order Entry Systems

Typical functionality: Review recently given drugsModify drugs/order new drugsDecision support: Drugs overdosage, drug-drug-

interaction, contraindications, allergies etc.

CPOE: Benefit

CPOE systems have been found to be effective to reduce medication errors and - partly - ADEs

Risk ratio

.1 1 10

Study Risk ratio (95% CI)

Potts (2004) 0.01 ( 0.00, 0.01)

Pestotnik (1996) 0.02 ( 0.01, 0.02)

Cordero (2004) 0.04 ( 0.00, 0.67)

Colpaert (2006) 0.13 ( 0.09, 0.17)

Bates (1999) 0.14 ( 0.11, 0.20)

Teich (2000) 0.27 ( 0.24, 0.30)

Kim (2006) 0.27 ( 0.11, 0.64)

Bizovi (2002) 0.30 ( 0.16, 0.57)

King (2003) 0.33 ( 0.27, 0.41)

Evans (1998) 0.34 ( 0.28, 0.40)

Oliven (2005) 0.37 ( 0.32, 0.43)

Gandhi (2005) 0.39 ( 0.27, 0.55)

Kirk (2005) 0.45 ( 0.39, 0.51)

Koide (2000) 0.46 ( 0.40, 0.52)

Galanter (2005) 0.53 ( 0.46, 0.61)

Mullett (2001) 0.59 ( 0.52, 0.67)

Igboechi (2003) 0.61 ( 0.58, 0.64)

Chertow (2001) 0.65 ( 0.63, 0.68)

Peterson (2005) 0.66 ( 0.58, 0.74)

Shulman (2005) 0.72 ( 0.54, 0.97)

Tamblyn (2003) 0.80 ( 0.73, 0.88)

Feldstein (2006) 0.85 ( 0.77, 0.94)

Maurer (2003) 0.87 ( 0.81, 0.94)

Mitchell (2004) 1.02 ( 0.88, 1.19)

Spencer (2005) 1.26 ( 1.03, 1.55)

Ammenwerth et al, JAMIA, 2008

CPOE: Challenges

Increasing reports of problems when introducing CPOE systems

Unexpected increased mortality after CPOE implementation, YY. Han, 2005

Unintended consequences of CPOE, Joan Ash, 2007

CPOE: Needs for evaluation

Systematic monitoring and evaluation of CPOE seems needed

What are good quality indicators to monitor CPOE quality?

Workshop: CPOE QI indicators

1. Expert and stakeholders panel

2. Literature research

3. Review of guidelines

4. Inductive content analysis: Draft set of QI

5. Individual rating

6. Group discussion and anonymous voting

7. Define target standards (repeat step 1-6)

Group 1: The Expert Panel

Task: Collect characteristics of excellent CPOE

performance What do you need to know about a CPOE

implementation in order to assess its quality?

Expected results:List of proposed quality indicators

Group 2: The Literature Team

Task: Read abstracts of CPOE studies from

http://evaldb.umit.at or from PubMed Identify CPOE quality indicators used in these

studies

Expected results:List of identified quality indicators

Classifying the Quality Indicators

1. QI concerning structural CPOE quality E.g. system quality, information quality, usage

2. QI concerning impact on process quality E.g. impact on clinical workflow, communication,

medication errors

3. QI concerning impact on outcome quality E.g. mortality, length of stay, ADE

Classifying the Quality Indicators

You can modify these three main categories by adding further sub-categories when needed

Setting up the groups

Define time frame for groups

Working in groups

Result reporting

Next steps

Are you interested to lead a group on QI for a special type of system (e.g. CPOE, LIS, RIS, EHR)? Please contact us!

Evaluation WG provides support for organizing group, conducting survey, analysing intermediate results

Contact

Elske Ammenwerth, UMIT, elske.ammenwerth@umit.at

Nicolette de Keizer, Amsterdam Medical Center, n.f.keizer@amc.uva.nl

Further information: http://iig.umit.at/efmi