Evidence Report/Technology Assessment Number 201

Enabling Medication Management Through Health Information Technology

EvidenceBased Practice

Health Information Technology

Agency for Healthcare Research and Quality

Advancing Excellence in Health Care • www.ahrq.gov

Evidence Report/Technology Assessment Number 201

Enabling Medication Management Through Health Information Technology

Prepared for: Agency for Healthcare Research and Quality U.S. Department of Health and Human Services 540 Gaither Road Rockville, MD 20850 www.ahrq.gov

Contract No. 290-2007-10060-I

Prepared by: McMaster Evidence-based Practice Center Hamilton, ON Canada

Investigators: K. Ann McKibbon, Ph.D. Cynthia Lokker, Ph.D. Steve M. Handler, M.D., Ph.D Lisa R. Dolovich, Pharm.D., M.Sc. Anne M. Holbrook, Pharm.D., M.D. Daria O’Reilly, Ph.D. Robyn Tamblyn, Ph.D. Brian J. Hemens, B.Sc.Phm., M.Sc. Runki Basu, B.Sc. (Hons.), M.A. Sue Troyan, B.A., R.T. Pavel S. Roshanov, B.Sc. Norman P. Archer, Ph.D. Parminder Raina, Ph.D.

AHRQ Publication No. 11-E008-EF April 2011

This report is based on research conducted by the McMaster Evidence-based Practice Center under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2007-10060-I). The findings and conclusions in this document are those of the author(s) who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. No statement in this article should be construed as an official position of the Agency for Healthcare Research and Quality or of the U.S. Department of Health and Human Services.

The information in this report is intended to help health care decision-makers; patients and clinicians, health system leaders, and policymakers, make well-informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.

This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.

This document is in the public domain and may be used and reprinted without permission except those copyrighted materials noted for which further reproduction is prohibited without the specific permission of copyright holders.

No investigators have any affiliations or financial involvement (e.g., employment, consultancies, honoraria, stock options, expert testimony, grants or patents received or pending, or royalties) that conflict with material presented in this report.

Suggested citation: McKibbon KA, Lokker C, Handler SM, Dolovich LR, Holbrook AM, O’Reilly D, Tamblyn R, Hemens BJ, Basu R, Troyan S, Roshanov PS, Archer NP, Raina P. Enabling Medication Management Through Health Information Technology. Evidence Report/Technology Assessment No. 201. (Prepared by the McMaster University Evidence-based Practice Center under Contract HHSA 290-2007-10060-I). AHRQ Publication No. 11-E008-EF. Rockville MD: Agency for Healthcare Research and Quality. April 2011.

ii

Acknowledgments The research team would like to thank those who helped with screening, abstracting, and

article retrievals: Danika Walden, Nathan McKibbon, Catherine Salmon, Jan Burke-Gaffney, Connie Freeborn, Jamie O’Donnell, Rizwan Bhaloo, Bryan Cheeseman, Hafsa Jan Qureshi, and Pouyan Ahangar. We would like to offer special thanks to Mary Gauld, Maureen Rice, and Roxanne Cheeseman for assistance and guidance with project management and editorial help. The project would not be complete without their wisdom, experience, good will, and sense of humor. Further thanks to Dr. Chris Gibbons and Dr. Paul Gorman who provided insights into their work on previous AHRQ evidence reports at the start of this project. We acknowledge the hard work of Nicholas Hobson, our computer programmer, for creating our evolving systematic review management software. The report is stronger and more focused because of the continued expert guidance of Rebecca Roper of the Agency for Healthcare Research and Quality as our Task Order Officer (TOO).

Our Technical Expert Panel provided valuable insights and challenges as well as ways to meet them. Our technical experts were David Bates, Doug Bell, Ken Boockvar, Chris Gibbons, Joy Grossman, Jerry Gurwitz, Joe Hanlon, Kevin Johnson, John Poikonen, Gordon Schiff, Bimla Schwarz, and Dennis Tribble. Their contact information is included in Appendix D. They represent a broad range of expertise and experience and the report is stronger because of them. Another group of experts who have been extremely helpful at improving the analyses of our data were our technical reviewers. Some of those who reviewed our first draft were members of the Technical Expert Panel: Joy Grossman, Joe Hanlon, Kevin Johnson, Bimla Schwartz, John Poikonen, Dennis Tribble, and our TOO Rebecca Roper. Other expert reviewers were Anne Bobb, Elizabeth Chrischilles, Alan Flynn, and Kevin Marvin. Their contact information is also in Appendix D.

iii

Enabling Medication Management Through Health Information Technology Structured Abstract Objective. The objective of the report was to review the evidence on the impact of health information technology (IT) on all phases of the medication management process (prescribing and ordering, order communication, dispensing, administration and monitoring as well as education and reconciliation), to identify the gaps in the literature and to make recommendations for future research.

Data sources. We searched peer-reviewed electronic databases, grey literature, and performed hand-searches. Databases searched included MEDLINE,® EMBASE,® CINAHL® (Cumulated Index to Nursing and Allied Health Literature), Cochrane Database of Systematic Reviews, International Pharmaceutical Abstracts,© Compendex,© INSPEC© (which includes IEEE®), Library and Information Science Abstracts,® E-Prints in Library and Information Science,® PsycINFO,® Sociological Abstracts,© and Business Source® Complete. Grey literature searching involved Internet searching, reviewing relevant Web sites, and searching electronic databases of grey literatures. AHRQ also provided all references in their e-Prescribing, bar coding, and CPOE knowledge libraries.

Methods. Paired reviewers looked at citations to identify studies on a range of health IT used to assist in the medication management process (MMIT) during multiple levels of screening (titles and abstracts, full text and final review for assignment of questions and data abstrction). Randomized controlled trials and cohort, case-control, and case series studies were independently assessed for quality. All data were abstracted by one reviewer and examined by one of two different reviewers with content and methods expertise.

Results. 40,582 articles were retrieved. After duplicates were removed, 32,785 articles were screened at the title and abstract phase. 4,578 full text articles were assessed and 789 articles were included in the final report. Of these, 361 met only content criteria and were listed without further abstraction. The final report included data from 428 articles across the seven key questions. Study quality varied according to phase of medication management. Substantially more studies, and studies with stronger comparative methods, evaluated prescribing and monitoring. Clinical decision support systems (CDSS) and computerized provider order entry (CPOE) systems were studied more than any other application of MMIT. Physicians were more often the subject of evaluation than other participants. Other health care professionals, patients, and families are important but not studied as thoroughly as physicians. These nonphysicians groups often value different aspects of MMIT, have diverse needs, and use systems differently. Hospitals and ambulatory clinics were well-represented in the literature with less emphasis placed on long-term care facilities, communities, homes, and nonhospital pharmacies. Most studies evaluated changes in process and outcomes of use, usability, and knowledge, skills, and attitudes. Most showed moderate to substantial improvement with implementation of MMIT. Economics studies and those with clinical outcomes were less frequently studied. Those articles that did address economics and clinical outcomes often showed equivocal findings on the effectiveness and cost-effectiveness of MMIT systems. Qualitative studies provided evidence of

iv

strong perceptions, both positive and negative, of the effects of MMIT and unintended consequences. We found little data on the effects of forms of medications, conformity, standards, and open source status. Much descriptive literature discusses implementation issues but little strong evidence exists. Interest is strong in MMIT and more groups and institutions will implement systems in the next decades, especially with the Federal Government’s push toward more health IT to support better and more cost-effective health care.

Conclusions. MMIT is well-studied, although on closer examination of the literature the evidence is not uniform across phases of medication management, groups of people involved, or types of MMIT. MMIT holds the promise of improved processes; clinical and economics studies and the understanding of sustainability issues are lacking.

v

Contents Executive Summary.................................................................................................................. ES-1 Introduction..................................................................................................................................... 1

Scope and Purpose of the Systematic Review .......................................................................... 1 Key Questions (KQs).......................................................................................................... 2

Background ............................................................................................................................... 4 Methods........................................................................................................................................... 6

Recruitment of Technical Experts and Peer Reviewers............................................................ 6 Key Questions........................................................................................................................... 6 Analytic Framework ................................................................................................................. 7 Literature Search Methods........................................................................................................ 8

Sources................................................................................................................................ 9 Search Terms and Strategies ............................................................................................... 9 Organization and Tracking of the Literature Search........................................................... 9 Title and Abstract Review .................................................................................................. 9 Defining Medication Management Health IT................................................................... 10

Data Abstraction ..................................................................................................................... 11 Assessment of Study Quality ............................................................................................ 11 Data Synthesis................................................................................................................... 13 Data Entry and Quality Control ........................................................................................ 13 Grading the Evidence........................................................................................................ 13 Peer Review ...................................................................................................................... 13

Results........................................................................................................................................... 14 Results of the Literature Search.............................................................................................. 14 KQ1. Within and Across the Phases of Medication Management Continuum, What Evidence Exists That Health IT Applications Are Effective?....................................... 16 Effectiveness Studies Overall ................................................................................................. 16

Strengths and Limitations of the Evidence ....................................................................... 17 Process Changes—Prescribing ............................................................................................... 21

Summary of the Findings for Process Changes ................................................................ 21 General Study Characteristics........................................................................................... 21 Outcomes .......................................................................................................................... 23 Summary........................................................................................................................... 25

Order Communication ............................................................................................................ 26 Summary of the Findings for Process Changes ................................................................ 26 Strengths and Limitations of the Evidence ....................................................................... 26 General Study Characteristics........................................................................................... 26 Outcomes .......................................................................................................................... 26 Summary........................................................................................................................... 27

Dispensing............................................................................................................................... 28 Summary of the Findings for Process Changes ................................................................ 28 Strengths and Limitations of the Evidence ....................................................................... 28 General Study Characteristics........................................................................................... 29 Outcomes .......................................................................................................................... 29 Summary........................................................................................................................... 30

Administering ......................................................................................................................... 30

vi

Summary of the Findings for Process Changes ................................................................ 30 Strengths and Limitations of the Evidence ....................................................................... 30 General Study Characteristics........................................................................................... 31 Outcomes .......................................................................................................................... 31

Monitoring .............................................................................................................................. 32 Summary of the Findings for Process Changes ................................................................ 32 General Study Characteristics........................................................................................... 33 Outcomes .......................................................................................................................... 34

Reconciliation, Discharge Summaries, and Education ........................................................... 35 Summary of the Findings for Process Changes ................................................................ 35

Combined Phases of Medication Management....................................................................... 36 Summary of the Findings for Process Changes ................................................................ 36

PDAs....................................................................................................................................... 37 Summary of the Findings for PDAs ................................................................................. 37

Intermediate Outcomes ........................................................................................................... 37 Summary of the Findings.................................................................................................. 37 Strengths and Limitations of the Evidence ....................................................................... 38 General Study Characteristics........................................................................................... 38 Outcomes .......................................................................................................................... 40

Economic Outcomes ............................................................................................................... 44 Full Economic Evaluations ............................................................................................... 44 Partial Economic Evaluations ........................................................................................... 45 Economics Summary ........................................................................................................ 50

Clinical Outcomes................................................................................................................... 51 Summary of the Findings.................................................................................................. 51 Outcomes .......................................................................................................................... 54 Prescribing—Strengths and Limitations of the Evidence................................................. 54 Prescribing—General Study Characteristics..................................................................... 55 Prescribing—Clinical Outcomes ...................................................................................... 55

Qualitative Studies .................................................................................................................. 57 Summary of Findings........................................................................................................ 57 Strengths and Limitations of Evidence ............................................................................. 57

Population Level Outcomes.................................................................................................... 65 Composite Outcomes .............................................................................................................. 65 Variation in Impact Depending on Medication Type or Form ............................................... 66

Summary of the Findings.................................................................................................. 66 Strengths and Limitations of the Evidence ....................................................................... 67 General Study Characteristics........................................................................................... 67 Outcomes .......................................................................................................................... 68

Unintended Consequences of MMIT Applications ................................................................ 69 Summary of the Findings.................................................................................................. 69 Strengths and Limitations of the Evidence ....................................................................... 69 General Study Characteristics........................................................................................... 70 Outcomes .......................................................................................................................... 70

KQ2. What Knowledge or Evidence Deficits Exist Regarding Needed Information To

Support Estimates of Cost, Benefit, Impact, and Net Value With Regard To Enabling Health

vii

IT Applications in Terms of Prescribing, Order Transmission, Dispensing, Administering, and Monitoring, as Well as Reconciliation, Education, and Adherence? Discuss Gaps in

Research, Including Specific Areas That Should Be Addressed and Suggest Possible Public and Private Organizational Types To Perform the Research and/or Analysis........................ 71

Introduction....................................................................................................................... 71 General Gaps..................................................................................................................... 73 Summary........................................................................................................................... 79

KQ3. What Critical Information Regarding the Impact of Health IT Applications Implemented To Support the Phases of Medication Management Is Needed To Give

Clinicians (Physicians, Nurses, Psychologists, Dentists), Pharmacists, Health Care

Administrators, Patients, and Their Families a Clear Understanding of the Value Proposition

Particular to Them?................................................................................................................. 79

Summary of the Findings.................................................................................................. 81 Financial Benefits ............................................................................................................. 81 Clinical Benefits................................................................................................................ 81 Organizational Benefits .................................................................................................... 82 Conclusions....................................................................................................................... 83

KQ4. What Evidence Exists Regarding the Impact of the Characteristics of Medication Management Health IT Applications, Such as Open Source, Proprietary, Conformity With Federal and Other Interoperability Standards, and Being Certification Commission for Healthcare Information Technology (CCHIT) Certified, Impact, Likelihood for Purchase, Implementation, and Use of Such IT Applications................................................................. 83

Summary of the Findings.................................................................................................. 83 Strengths and Limitations of the Evidence ....................................................................... 85 General Study Characteristics........................................................................................... 85 Outcomes .......................................................................................................................... 85

KQ5. What Factors Influence Sustainability (Use and Periodic Updates) of Health IT Applications That Support a Phase of Medication Management Continuum (Prescribing, Dispensing, Administering, and Patients’ Taking of Medications)? ...................................... 88

Sustainability of Health IT and Medication Management Systems.................................. 88 Future Sustainability of Health IT and Medication Management Systems ...................... 89 Conclusions....................................................................................................................... 89

5a. To What Extent Does the Evidence Demonstrate That Health Care Settings (Inpatient,

Ambulatory, Long-Term Care, etc.) Influence Implementation, Uuse, and Effectiveness of Such Health IT Applications?................................................................................................. 92

Implementation ................................................................................................................. 92 Effectiveness ..................................................................................................................... 93 Use .................................................................................................................................... 93

5b. What is the Impact (Challenges, Merits, Costs, and Benefits) of Having Electronic Access to Patients’ Computerized Medication Records, Formulary Information, Billing Information, Laboratory Records in the Quality and Safety of Care Provided by Health IT Applications That Support at Least One Phase of the Continuum of Medication Management (Prescribing, Dispensing, Administering, and Patients Taking of Medications)?.................. 93 KQ6. Two-Way Prescriber and Pharmacy Electronic Data Interchange (e-Prescribing) (a) What Evidences Exists Demonstrating the Barriers and Drivers of Implementation of Complete EDI That Can Support the Prescription, Transmittal and Receipt, and Perfection

viii

Process of e-Prescriptions? (b) How Do Barriers, Facilitators, and Economic Incentives Vary Across Pharmacists, Physicians, and Other Relevant Stakeholders With Respect to Adoption and Use of Complete EDI (e-Prescribing/Ordering With e-Transmission)?............................................................. 94

Summary of the Findings.................................................................................................. 94 Summary of Evidence....................................................................................................... 96

KQ7. What Evidence Exists Regarding the Extent of Integration of Electronic Clinical

Decision Support (CDS) in a Health IT System for Prescribing and Dispensing of Medications? To What Extent Does the Use of CDSS in a Health IT System for Prescribing and Dispensing of Medications Impact the Various Outcomes of Interest Including Health Care Process, Intermediate and Clinical? ...................................................................................................... 96

Summary of the Findings: All Phases of Medication Management ................................. 96 Discussion ................................................................................................................................... 100

Summary of Key Findings .................................................................................................... 100 KQ1. Effectiveness ......................................................................................................... 100 KQ2. Gaps in Evidence and Knowledge ........................................................................ 100 KQ3. Value Proposition for Implementers and Users .................................................... 101 KQ4. System Characteristics .......................................................................................... 101 KQ5. Sustainability......................................................................................................... 101 KQ6. Two-Way EDI....................................................................................................... 102 KQ7. RCTs in CDSS ...................................................................................................... 102 CDSS............................................................................................................................... 105

Future Research .......................................................................................................................... 109 Need for High Quality Evidence..................................................................................... 111 Need for Well-Designed Research.................................................................................. 111

Conclusions................................................................................................................................. 116 References................................................................................................................................... 118 Acronyms and Abbreviations ..................................................................................................... 160

Tables Table A. Research Design for studies across the Phases of Medication Management and

Education and Reconciliation ................................................................................................... ES-6 Table B. Settings for the Phases of Medication Management and Reconciliation

Table C. Clinicians Evaluated in Outcomes Studies of Medication Management Phases,

Table D. Main Health IT Studied by Medication Management Phase and Education and

Table 1. Research Design for Studies Across the Phases of Medication Management and

Table 2. Settings for the Phases of Medication Management and Reconciliation

Table 3. Clinicians Evaluated in Outcomes Studies of Medication Management Phases,

Table 4. Patients and Caregivers Studied by Phase of Medication Management and Education

and Education............................................................................................................................ ES-6

Education, and Reconciliation .................................................................................................. ES-7

Reconciliation ........................................................................................................................... ES-7

Education and Reconciliation ....................................................................................................... 16

and Education................................................................................................................................ 18

Education, and Reconciliation ...................................................................................................... 18

and Reconciliation ........................................................................................................................ 19

ix

Table 5. Main Health IT Studied by Medication Management Phase and Education and

Reconciliation ............................................................................................................................... 20

Table 7. Research Methods of Studies That Evaluated Process Changes Associated With the

Table 8. Summary of the Number of Studies Reporting Statistically Significant Process Changes

Table 9. Summary of the Number of Statistically Significant Process Changes in Studies of

Table 11. Study Designs Used in Studies Measuring Intermediate Outcomes Across the Phases

Table 12. Clinician Study Participants in Studies Assessing Intermediate Outcomes Across the

Table 13. Patient Study Participants in Studies Assessing Intermediate Outcomes Across the

Table 14. Research Design for Studies Across the Phases of Medication Management and

Table 15. Summary of the Number of Studies Reporting Statistically Significant Differences in Clinical Primary Endpoints Between Study Groups for Hospital and Ambulatory Based

Table 16. Number of Studies Across the Medication Management Phases Using MMIT To Assist

Table 18. Frequency of Medication Management Phases Studies Plus Reconciliation

Table 20. Number of Studies That Evaluated the Effects of MMIT on Clinicians (the Clinicians

Table 21. Frequency With Which Patients or Caregivers Across Age Groups Were Studied as the

Table 22. Study Settings in Which the MMIT Application Was Studied (Studies Could Take

Table 25. Number of Articles Addressing System Type in Relation to Likelihood To Purchase,

Table 26. List of Articles Addressing Various Features That Were Instrumental in the Decision

Table 27. Frequency of Core Informatics Journal Articles That Mention Sustainability to the End

Table 29. Study Design of Included Studies Across the Medication Management Phases (Plus

Table 6. Health IT Integrated With the Health IT Being Studied................................................. 20

Prescribing Phase of MMIT.......................................................................................................... 21

in Studies of Prescribing by Process for Hospital and Ambulatory Based Studies ...................... 24

Order Communication by Process for Hospital and Ambulatory Based Studies ......................... 28 Table 10. Intermediate Outcomes Across the Phases for Medication Management .................... 38

for Medication Management......................................................................................................... 38

Phases of Medication Management .............................................................................................. 39

Phases of Medication Management .............................................................................................. 39

Education and Reconciliation That Address Clinical Outcomes as Their Main Outcomes ......... 52

Studies........................................................................................................................................... 52

in the Management of Specific Drugs or Drug Classes................................................................ 68 Table 17. Summary of Gaps and Needs Across Key Questions................................................... 72

and Education................................................................................................................................ 74 Table 19. Frequency of Research Designs for Included Studies .................................................. 74

Were the Major Focus of the Outcomes of the Articles) .............................................................. 75

Main Focus of an Article (How MMIT Affects Patients)............................................................. 75

Place in More Than One Setting).................................................................................................. 76 Table 23. Technologies That Were the Main Focus of the Studies of MMIT.............................. 76 Table 24. Summary of the Evidence in Relation to the CITL Value Framework ........................ 80

Implement, or Use an MMIT System ........................................................................................... 84

To Purchase, Implement, and Use ................................................................................................ 86

of 2009 .......................................................................................................................................... 89 Table 28. Issues of Consideration and/or Further Exploration in Future Research.................... 109

Education and Reconciliation).................................................................................................... 111

x

Figures Figure 1. A Functional Model of the Medication Management Continuum Created by Bell et al. .................................................................................................................................... 2 Figure 2. Conceptual Model Addressing the Seven Key Questions: Enabling Medication Management Through Health IT. ................................................................................................... 8 Figure 3. Literature Flow of Medication Management Studies.................................................... 15 Figure 4. Trends in Publication of Articles Relating to the Phases of Medication Management Across Years Until Searching Was Completed in June 2010....................................................... 17 Figure 5. Summary Overview of Meaningful Use Objectives ..................................................... 91

Appendixes Appendix A. Exact Search Strings Appendix B. Sample Screening and Data Abstraction Forms Appendix C. Evidence Tables Appendix D. Technical Expert Panel and Peer Reviewers Appendix E. Excluded Studies Appendix F. Glossary of Terms

xi

Executive Summary Background

Medication management is a continuum that covers all aspects of prescription medications. Medication management includes prescribing and ordering, order communication (or order transmission) between prescribers and pharmacists, dispensing, administering, and monitoring, as well as reconciliation, adherence, and education.1 Medication management is complex and costly and enhances the health and well-being of more than half of the population in the developing world. Health information technology (health IT) holds great promise to improve the quality of health care and reduce potential and real errors in medication management while at the same time providing cost-effective care. The Agency for Healthcare Research and Quality (AHRQ) is committed to summarizing and providing the evidence base for health IT. It has produced evidence summaries on health IT related to costs and benefits;2 barriers and drivers of health IT for the elderly, chronically ill, and underserved;3 the impact of consumer informatics applications;4 and telemedicine.5 AHRQ also has contracted for evidence summaries on the use of health IT in decisionmaking,6 patient-centered care,7 and decision support for health care decisionmaking.8 The contracted reports will be available through www.healthit.AHRQ.gov in mid-2011. Although these reports often mention medication management, the body of published evidence on all aspects of the medication management process and how it is affected by multiple health IT systems has not been consolidated. A single document is needed to summarize the evidence evaluating the effects of health IT on the medication management process across providers, settings, patients, and research methods.

The objectives of this report are to: 1. Review the literature on the effects of health IT on medication management. 2. Synthesize available evidence regarding the effectiveness and effects of health IT in all

phases of medication management as well as reconciliation and education. 3. Identify gaps in the literature. 4. Make recommendations for future research.

For the purposes of this review, medication management includes the processes that encompass the five phases of the medication process (i.e., prescribing and ordering, order communication, dispensing, administering, and monitoring) across groups of health professionals, patients, and their informal caregivers, and two aspects of quality with respect to medication management across the five phases of medication management (medication reconciliation and education, both postprofessional education of training and patient education related to medication management). Medication management can also include procurement, storage, and reporting from the first assessment of patients to determine their need for drugs through to optimal care and monitoring after the drugs are prescribed. The organization of the information in this report is based on the Bell framework of the five phases across the continuum of medication management and reconciliation and education.1

To address the goals of this report, we further define medication management health IT (MMIT) applications as electronic systems that (1) collect, process, or exchange health information about patients; (2) are integrated with existing health IT systems such as electronic health records or electronic medical record (EMR) systems; and (3) provide advice or suggestions to either the health care provider or the patients and their families on issues or

ES-1

decisions related to medication management. We recognize that functional elements of the MMIT will vary across particular implementation approaches within a given phase of medication management. Many of the MMIT applications we found were designed to encompass more than one phase of medication management. The sophistication of the systems, degree of integration of the health IT into workflow systems, and the broad range of settings in which a particular health IT is implemented and used are also complex and varied. Many health professionals, support staff, patients, and patients’ families were involved in medication management in the studies assessed.

The evidence assessing MMIT is large, diffuse, and published across many disciplines. People who can benefit from the knowledge in this report include health professionals, researchers, administrators, and other decisionmakers and those who develop and implement health IT applications. This report is timely because of the Federal emphasis on the use of health IT to improve health care while at the same time making health and wellness care more cost effective and safer. Seven questions structure this evidence report. Within reporting related to the questions, sections are based on phases of medication management. Reporting is done to address the multiple settings where medication management is important, the range of health care providers who deliver and support care using medications, and classes of medications, specific drugs, or a broad spectrum of medications.

Key Questions (KQs) KQ1. Effectiveness

Within all phases of the medication management continuum, what evidence exists that health IT applications are effective in improving:

a. Health care processes, b. Other intermediate outcomes (e.g., satisfaction with system, usability, knowledge, skills,

and attitude), c. Costs and economic outcomes, d. Clinical outcomes for patients, e. Population level outcomes, and f. Composite outcomes. g. To what extent does the impact of health IT on improving health care processes, other

outcomes, costs and economics, and clinical outcomes vary depending on the type of medication (controlled or noncontrolled substance) or the form of the medication (e.g., oral, injection, intravenous)?

KQ2. Gaps in Knowledge or Evidence What knowledge or evidence deficits exist to support estimates of cost, benefit, impact, and

net value with regard to health IT applications in all phases of medication management?

KQ3. Value Proposition for Implementers and Users What critical information regarding the impact of health IT applications implemented to

support the phases of medication management is needed to give clinicians, health care facility administrators, patients, and their families a clear understanding of the value proposition particular to them?

ES-2

KQ4. System Characteristics What evidence supports or refutes the impact of any of: open source, homegrown,

proprietary, local configuration ability, system configuration ability, conformity with standards being Certification Commission for Healthcare Information Technology (CCHIT) certified, system architecture, or feature set on the decision to purchase, implement, or use health IT in medication management systems?

KQ5. Sustainability What factors influence sustainability of health IT applications that support a phase of the

medication management continuum? a. What evidence exists to demonstrate that health care settings (ambulatory, long-term

care, etc.) influence implementation, use, and effectiveness of such health IT applications?

b. What is the impact (challenges, merits, costs, and benefits) of having electronic access to patient data on the quality and safety of care provided by health IT applications that support at least one phase of the continuum of medication management?

KQ6. Two-Way Prescription Electronic Data Interchange (EDI) In a two-way electronic data interchange (EDI) between the prescribers and pharmacists: a. What evidence exists demonstrating the barriers and drivers of implementation of

complete EDI that can support the prescription, transmittal and receipt, and perfection process of e-Prescriptions?

b. How do barriers, facilitators, and economic incentives vary across pharmacists, physicians, and other relevant stakeholders with respect to adoption and use of complete EDI (e-Prescribing/ordering with e-Transmission)?

KQ7. Randomized Controlled Trials (RCTs) of Clinical Decision Support Systems (CDSS)

What evidence exists regarding the extent of integration of electronic clinical decision support in a health IT system for the prescribing, dispensing, and administering of medications, and to what extent does the use of clinical decision support systems impact the various outcomes (e.g., health care process, intermediate, cost and economics, and clinical) of interest?

Methods We anticipated finding few RCTs across all phases of medication management and MMIT

applications. Studies that employ other research methods can also provide valuable evidence for understanding MMIT applications. We therefore included studies employing a range of research methodologies. We restricted our analysis to hypothesis-driven studies with group comparisons and appropriate statistical analysis in addition to qualitative studies with explicit methods for KQ1: Effectiveness. The only methodological limit was for assessment of the effect of CDSSs on prescribing, for which sufficient RCTs were available to provide evidence for synthesis.

Through consultation with our internal team and AHRQ, we determined that the answers to KQ2: Gaps in Knowledge or Evidence and KQ3: Value Proposition for Implementers and Users would become evident from our review of the evidence in KQ1: Effectiveness. We supplemented these articles with other studies addressing values propositions by stakeholders. KQ4: System

ES-3

Characteristics addresses the impact of MMIT application features on the likelihood that the systems will be purchased, implemented, and used. The evidence for this question comes from studies of all designs that measure implementation, use, and purchasing decisions. KQ5: Sustainability addresses the factors influencing the sustainability of MMIT applications, specifically the impact of the setting and access to other electronic data within integrated systems on health care quality and safety. To identify articles that addressed this question, the team, in consultation with AHRQ, used the definition of sustainability by Humphreys et al.,9 which restricted our choice of articles to only a few. Their definition of sustainability was the ability of a health service to provide ongoing access to appropriate quality care in a cost- and health-effective manner. KQ6: Two-Way Prescription EDI relates to the barriers and facilitators to complete EDI between prescribers and pharmacies during the time between prescription writing and dispensing and how these vary across stakeholders. The best evidence available for KQ6 is found in articles studying EDI between prescribers and pharmacies that include original data (qualitative or quantitative). Because insufficient evidence was found on two-way EDI, we included one-way EDI as well. KQ7: RCTs of CDSS addresses the extent to which CDSS systems are integrated into health IT systems for medication management and the impact on outcomes as described in KQ1: Effectiveness. As a team we felt that adequate evidence was available to address this issue so that we could limit our scope to RCTs.

Given the broad range of questions and outcomes addressed, we searched peer-reviewed electronic databases by first using textwords relating to the various types of health IT applied to medication management (Appendix A of the full report). These searches were then combined with a search using subject headings related to the five medication management phases plus reconciliation and education as well as specific health IT application terms (e.g., CDSS). We combined these medication management terms with computer and technology terms. When possible, we excluded letters, editorials, commentaries, and animal studies. Because our interest was in all study designs, we did not limit based on methodology. We also put no limits on language or time to capture the global literature and early studies.

Databases searched included MEDLINE, Embase, CINAHL (Cumulated Index to Nursing and Allied Health Literature), Cochrane Database of Systematic Reviews, International Pharmaceutical Abstracts, Compendex, Inspec (which includes IEEE Xplore), Library and Information Science Abstracts, E-Prints in Library and Information Science, PsycINFO, Sociological Abstracts, and Business Source Complete. We also looked for eligible studies by reviewing grey literature sites, performing hand searches of pertinent reviews, querying our experts, and by reviewing the AHRQ National Resource Center for Health IT Knowledge Library resources (available at: http://healthit.ahrq.gov/portal/server.pt/community/knowledge_library/653).

The search results were downloaded into Reference Manager version 10 (ISI ResearchSoft) and uploaded into a customized systematic review management system (Health Information Research Unit, McMaster University).

Studies were eligible for inclusion if they used health IT in any aspect of the medication management process. We included articles on MMIT only if the system was integrated with at least one existing health IT system and if they processed patient-specific information and provided advice or suggestions. A critical inclusion requirement was the integration of information.

Personal digital assistants (PDAs), which integrated patient-specific information provided by either the clinicians or the patients, were analyzed to assist in medication management decisions

ES-4

(by request of AHRQ). This exception is made because PDAs and hand-held devices are considered an important, and perhaps unique, means of improving health care quality in relation to medications. The use of PDAs to manage medications is especially important for clinicians and patients who are in settings that do not have large, sophisticated, and integrated information systems. Other stand-alone devices with no integration of information with another health IT were excluded. Articles on all five phases of the management process plus medication reconciliation and postprofessional education related to MMIT were included. Once we tagged the articles for content, we assessed whether those that passed our inclusion criteria were pertinent to specific key questions. Many articles were analyzed in several phases of medication management and sections of the report.

Studies were classified as being observational, case-control, cohort, or RCTs. The quality of included studies was assessed using the same criteria employed by Jimison et al. in their AHRQ report.3 RCT scoring was based on the Delphi consensus work done by Verhagan and colleagues.10 This scale is referred to in this report as the Verhagen/AHRQ RCT quality scale. Observational studies with before–after, time series, surveys, or qualitative methods were not assessed for quality because few well-validated instruments exist. Bibliographies of systematic and narrative reviews were examined to identify studies, and select reviews were integrated into sections of the report.

Data were abstracted from relevant articles and tagged for applicability to the various key questions. Given the range of questions addressed, data abstraction was performed by a core group of staff and entered into online data abstraction forms. One reviewer did the abstraction, and a second, senior reviewer checked its accuracy. The authors of this report performed a final check on the abstracted data. The reviewers were not blinded to the identity of the article authors, institutions, or journal. Data abstraction was difficult in many instances because of the lack of accepted definitions and absence of important features of the study or MMIT application. For example, we identified problems with the differences between computerized provider order entry (CPOE) for ordering and e-Prescribing systems. Definitions for medication errors and related terms were often inconsistently used. To make data abstraction easier, we established working definitions, which can be found in Appendix F of the full report.

Meta-analysis was not performed on any data because of the heterogeneity of the studies in terms of interventions, populations, technologies used, and outcomes measured, as well as the presence of mostly descriptive and observational studies.

Throughout the project, the core team sought feedback from the internal advisors, our Task Officer from AHRQ, and the Technical Expert Panel.

Results Our literature search retrieved 40,582 articles. After duplicates were removed, 32,785 articles

were screened at title and abstract stage. From a full-text screen of 4,578 articles, we identified 789 articles that were eligible for inclusion in this report. Of these articles, 361 met only our inclusion criteria for content and did not have group comparisons, hypothesis testing, or appropriate analysis. These are listed in the bibliography of the report. Across the seven key questions, we synthesized the information from 428 articles.

KQ1. Effectiveness All outcomes. KQ1: Effectiveness contains 379 studies assessing changes in process, intermediate outcomes, clinical outcomes, and economic and cost outcomes. The majority of

ES-5

studies were observational, with a fair number of RCTs for prescribing and monitoring phases (Table A). Fifty-three qualitative studies are included in this total. Prescribing and monitoring were the most frequently studied phases of medication management (Table A), with hospital and ambulatory care settings well-represented to the near exclusion of long-term care, home, and community (Table B).

Though dealing with prescriptions and medications, pharmacists were poorly represented in studies, most focused on physicians (Table C). CDSS and CPOE systems were the most often studied MMIT technologies (Table D).

Table A. Research design for studies across the phases of medication management and education and reconciliation Design P OC D A M E R

RCT 69 1 2 2 37 1 1 Cohort 13 2 2 1 6 0 1 Observational 144 18 10 26 29 2 4 Qualitative 37 5 3 10 5 0 0 Total 263 26 17 39 77 3 6

Note: some studies cross more than one phase. Column headings: P = Prescribing, OC = Order Communication, D = Dispensing, A = Administering, M = Monitoring, E = Education, R = Reconciliation Abbreviations: RCT = randomized controlled trial

Table B. Settings for the phases of medication management and reconciliation and education Setting P OC D A M E R

Ambulatory care (e.g., clinic, doctors office) 94 6 2 1 40 2 0

Community (e.g., school, community center) 0 0 0 0 1 0 0

Home 2 0 0 2 5 0 0 Hospital 164 12 9 34 36 1 6

Long-term care 4 0 2 3 1 0 0

Pharmacy 11 13 10 2 4 0 1 Note: some studies cross more than one phase or setting. Column headings: P = Prescribing, OC = Order Communication, D = Dispensing, A = Administering, M = Monitoring, E = Education, R = Reconciliation

ES-6

Table C. Clinicians evaluated in outcomes studies of medication management phases, education, and reconciliation

Clinicians P OC D A M E R

Primary care physicians 25 2 1 0 3 1 0 Specialists 11 0 0 0 1 0 0 Hospitalists 18 0 0 0 2 0 0 Other physicians 7 0 0 0 2 0 0 Physicians undifferentiated 26 1 0 1 3 0 1 Nurses 20 1 0 16 2 0 1 Midlevel practitioners (e.g., PA, NP, MW) 6 0 0 0 2 0 0 Pharmacists 13 6 5 2 1 0 1 Other health professionals 10 0 2 4 1 0 0 Hospital administrators 4 1 0 1 0 0 0

Note: some studies cross more than one phase and clinician type. Column headings: P = Prescribing, OC = Order Communication, D = Dispensing, A = Administering, M = Monitoring, E = Education, R = Reconciliation Abbreviations: MW = midwife, NP = nurse practitioner, PA = physician assistant

Table D. Main health IT studied by medication management phase and education and reconciliation Health IT P OC D A M E R

CDSS/reminders 177 8 4 5 63 1 1

CPOE/POE system 90 12 5 9 11 0 0

e-Prescribing 31 10 3 4 2 1 1

Order transmission of the prescription to and from doctor to pharmacy electronically 2 3 0 0 0 0 0

Pharmacy information system 2 3 4 1 0 1 1

Barcoding medication administering 1 0 2 20 0 0 0

Barcoding dispensing 0 0 1 0 0 0 0

eMAR, e-TAR 2 2 2 14 0 0 0

Other 13 2 3 7 14 1 5

Personal digital assistants or hand-helds 7 1 0 4 5 0 1 Note: some studies cross more than one phase and technology. Column headings: P = Prescribing, OC = Order Communication, D = Dispensing, A = Administering, M = Monitoring, E = Education, R = Reconciliation Abbreviations: CDSS = Clinical decision support system, CPOE = Computerized provider order entry, POE = Provider order entry, eMAR = Electronic Medication Administration Record system, eTAR = Electronic Treatment Administration Record system

The results from this section suggest that care processes such as medication errors, time for tasks, workflow and knowledge, skills, and attitudes can be improved with the use of MMIT. The evidence is strongest specifically during the prescribing and monitoring phases. Few studies evaluated clinical outcomes associated with the use of MMIT. Those that did often did not show statistically significant improvements in clinical outcomes. Most of the studies with statistically

ES-7

significant differences in clinical outcomes found small differences. The small number of articles with data on clinical outcomes is probably due, at least in part, to the difficulty in evaluating and establishing a direct association between the use of MMIT and clinical outcomes. This difficulty arises because of the distant nature of the outcome compared with the application of the health IT. Other contributing factors could also be considered.

Much of the relatively new research is addressing the type of research needed to come to a realistic and useful assessment of MMIT: pilot and demonstration projects and quantitative studies. Limited evidence suggests that MMIT can likely be cost effective, although most of the economic data come from cost analyses, which were often incomplete and seldom from head-tohead cost-effectiveness, cost utility or cost-benefit trials.

A substantial body of qualitative literature indicates support for the use of health IT in the various phases of medication management by a number of health care providers and patient groups. Survey studies of satisfaction and use reflect similar findings of acceptance and satisfaction, although most indicated room for improvement. Issues relating to changing care practices and workflow are frequently mentioned. The studies also provide useful summaries of unintended consequences of MMIT applications, which are discussed in detail in the full report.

Process changes. Most of the studies evaluating MMIT applications provided data on changes in process (225 of 378). Distribution in the number of studies across the five phases, plus reconciliation and education, was not equal. Prescribing was studied in 174 studies, order communication in 16 studies, dispensing in 9 studies, administering in 19 studies, and monitoring in 47 studies. Four studies evaluated reconciliation and one studied patient education. Studies often evaluated more than one phase.

Prescribing. The prescribing phase is well studied (174 studies), especially in hospital (61 percent of studies) and ambulatory care settings (39 percent). Long-term care centers (one study) and community and home settings (no studies) are not well studied. Physicians are by far the most studied group of health professionals. More studies are needed that evaluate nonphysician use of MMIT, specifically pharmacists, mental health professionals, nurses, and other nonphysician prescribers, as well as patients and their caregivers. Many of the studies of health care providers who were not physicians were purely descriptive of the people involved with them, and the systems themselves.

Based on the studies of process changes, CDSS and CPOE systems can play an important role in making prescribing and ordering more accurate, improving record keeping, and speeding up and improving communication. Both systems, either alone or, more often, integrated, are well studied (multiple studies with strong methods). Other MMIT applications lack evidence, especially those that involve nurses, pharmacists, and patients and their families.

MMIT in prescribing is associated with improvements in patient safety-related processes of the prescribing process, especially in hospital-based studies (87 percent, 52 of 60 studies), and somewhat less in ambulatory-based studies (68 percent, 28 of 41 studies). Errors related to prescribing and ordering were reduced in hospital-based studies (68 percent, 15 of 22 studies), but prescribing errors were not studied as often in ambulatory settings (two of two studies were positive). Reductions in time were related to the time taken to order or prescribe or the speed of the prescribing-to-administering processes. Most reductions in time were not seen as often in hospital-based studies (four of seven studies positive), but were positive more often in ambulatory settings (four of five studies). Adherence to treatment guidelines, reminders, and

ES-8

recommended practice was improved in hospital studies (83 percent, 19 of 23) and to a lesser but still significant extent in ambulatory studies (64 percent, nine of 14 studies). Workflow was not evaluated in these studies of changes in process, although issues of workflow are addressed in qualitative studies in other sections of this report.

Order communication. Order communication, like dispensing, is one of the two medication management phases with the least number of studies—only 16 were identified. Two-way EDI holds promise to increasing the effectiveness of perfecting the prescription/order interactions between clinician prescribers and pharmacists. Currently, evidence on one-way communication predominates. The changes in process were also varied (two studies of errors, two of prescribing changes, five on time considerations, and three on workflow). Most studies were done using quantitative observational methods and all showed positive results.

Dispensing. Nine studies (three RCTs) assessed process improvements in dispensing. All process changes that were evaluated were found to be positive: four on modifications of the drugs that the pharmacists dispensed, three on errors, two on workflow, and one on adherence to good practice. With these few studies and multiple outcomes, evidence is limited on the role of MMIT in improving dispensing. This supports the findings of a Canadian health technology assessment report on MMIT that evaluated hospital dispensing and administering medications in hospitals.11

Administering. Many articles dealing with administering medications were not included in this report because they were descriptive and did not include comparative data. Nineteen studies, 1 RCT, 1 cohort study, and 17 quantitative observational studies, were included. All studies were set in hospitals and included nurses. The MMIT systems were well integrated into multiple hospital IT systems. Error-reduction goals were common in the studies and almost always found to be improved (8 of 13 studies of errors). Errors were mixed, as some related to transcription and some to timing of administration, while some identified more serious errors. Four studies showed no improvement in errors while one study showed increases in errors, mostly related to timing of administration.12 Four of five studies showed reductions in time from ordering to administering medication. Two studies evaluated the allocation of nursing time: one showed change and one did not in the proportion of time spent on various nursing tasks, including direct patient care, with the introduction of integrated MMIT for medication administering.

Monitoring. In our analysis, 70 percent (33 of 47 studies) of the included studies were associated with a 50 percent improvement in half or more process measures. Of these studies, most targeted physicians exclusively (34 studies), were conducted in academic institutions (33 studies), were developed for use in the ambulatory care setting (28 studies), focused on the adult population (36 studies), and provided CDSS with alerts or reminders to support chronic disease management (12 studies).

Studies that involved laboratory-based medication monitoring were most likely (76 percent of the time) to be associated with a greater than 50 percent improvement in a process outcome(s) than sign- or symptom-based medication monitoring. The most successful types of studies focused on changing prescriber behavior, improving response time to generated alerts, and improving the diagnosis and management of chronic diseases.

ES-9

Reconciliation. Two systematic reviews and four studies provided evidence for improved reconciliation of medications with health IT. Reconciliation is the matching of medication lists over time, from different health care systems or from different prescribers. The evidence on reconciliation of medication lists is sparse, especially for systems that are fully integrated and capable of providing electronic comparisons of historical and current medications for individual patients at hospital discharge or on transfer to other facilities. All four studies showed improvements in agreement among lists of medications and two extended the evaluation to show improved prescribing13 and reduced errors.14

Unintended consequences. Eighteen studies provided data on adverse effects or unintended consequences. Two qualitative studies identified classes or categories of unintended consequences of health IT, many of which apply to MMIT applications. Some unintended consequences are minor, and some are major. In addition, some are seen to be positive and helpful. Some consequences are serious. For example, a small but statistically significant increase in mortality was seen in a children’s hospital that installed a CPOE system that did not match workflow needs.15 A similar study showed another children’s hospital that did not see the same increase in mortality in admitted children after their careful planning and implementation of health IT.16 Several authors contend that all health IT has unintended consequences. Formal evaluations of health IT installations should seek these unintended consequences and report them in their publications related to the evaluation. The importance of unintended consequences of MMIT also depends on the severity of the event, the degree of invasiveness of the MMIT, and the extent to which the use of the MMIT system disrupts existing workflow and processes. Consideration of formal reporting of serious unintended consequences might benefit all involved in development and implementation of MMIT systems. The qualitative studies in this report supplied a richer understanding of the adverse effects of MMIT, and they can form a strong base for more qualitative and quantitative studies of unintended consequences.

Education. Education related to MMIT centers on three aspects: formal informatics training during professional education or after graduation, training to use the MMIT systems, and improved outcomes based on knowledge and skills because of the use of the MMIT systems for health care providers, patients, and their families. This report does not include preprofessional or professional education related to the use and understanding of MMIT systems or certification in informatics or eHealth, all important aspects of MMIT application development and integration. Although we sought articles assessing postprofessional education related to changes in process associated with MMIT systems, we did not identify any articles that met our criteria. Training in the use of systems was often mentioned in articles but was not evaluated. Only one article was related to the educational component of MMIT systems for patient and family use, and it was associated with improved clinical outcomes. More information on health care professional and patient education is included in the sections of this report dealing with intermediate outcomes.

Intermediate outcomes. Intermediate outcomes deal with use, usability, education, knowledge, skills, and attitudes. Most studies with intermediate main endpoints focused on measuring use, correlates of use, perceptions, and satisfaction in the prescribing phase (26 of 42 studies). As for changes in process, clinicians and prescribing were well-studied. Use, perceptions, and satisfaction were reported to be improved. Factors such as ease of use, perceived usefulness, and improved quality of care predominated. Satisfaction and attitudes varied depending on the role of

ES-10

the health care provider. Variation in needs and roles of health professionals with respect to use of health IT are real and should be considered when choosing or implementing any new IT system. Usability studies with comparison groups are sparse but can provide useful suggestions to improve systems. Usability studies are often difficult to generalize or transfer across settings, in part because MMIT effectiveness is linked strongly to the culture, institutional leadership, and other situation specific factors. Therefore, applicability of findings related to usability is problematic in MMIT applications.

Economic outcomes. Five of 31 articles dealing with costs conducted comprehensive economic evaluations (costs and consequences). Two evaluated a CPOE system and three evaluated CDSS. Most of the studies that included monetary data (22 of 31 studies) were partial economic evaluations in the form of cost analyses (assessing costs of alternatives without analysis of effectiveness or efficacy). Most of these partial economic analyses assessed costs of prescribed medications with the MMIT system compared with not having the MMIT system.

Several studies found that health IT interventions may offer cost advantages despite their increased acquisition costs. These studies showed that over time, a net benefit accrued based on cost reductions resulting from the MMIT (such as lower adverse drug events (ADEs), drug costs, and laboratory test usage). However, given the uncertainty that surrounds the cost and outcomes data, and limited study designs available in the literature, it is difficult to reach any definitive conclusion as to whether the additional costs and benefits represent value for money.

Clinical outcomes. A total of 76 studies sought to measure improvement in clinical outcomes or reduction in ADEs, of which 26 (34 percent) reported significant benefits of health IT. One reported harm—a small but clinically important increase in mortality when an inflexible CPOE was implemented in a children’s hospital.15 Because of the seriousness of the implications of this study, many people reviewed this article and its methods.17 A later and similar study showed that with careful planning another children’s hospital did not see the same increase in mortality in admitted children after the implementation of a health IT.16

An additional two studies implemented CDSSs to reduce costs and assessed whether reductions in drug use increased mortality15 and length of stay.18 Both studies lacked sufficient power to conduct a valid assessment.

Studies that used laboratory-, sign- and symptom-based monitoring approaches were mostly clinician based. If theMMIT monitoring was used to identify and intervene with patients with actual problems (e.g., excess blood pressure) or needed care (e.g., hemoglobin A1c monitoring), this appears to be more effective than CDSS approaches that identified theoretical problems (potential for ADEs), particularly if patients are also sent reminders and decision support recommendations.

Highly targeted interventions, which focused on specific problems that provide problem-related specific interventions, appear to be more effective than more diffusely focused systems such as CDSS and CPOE. Some of these highly targeted interventions involved CDSS tools for improving the effectiveness of anticoagulants (proportion of days with blood clotting parameters within the therapeutic range), improving the choice, route, and duration of antibiotics, and reducing ADEs related to antibiotic use, and most were successful.

Studies that have been successful in improving patient outcomes target high risk and vulnerable populations who have poor disease control, lack sufficient access to health care providers to manage their condition or subpopulations with sufficient economic resources to

ES-11

respond to the CDSS intervention. The effect of similar CPOE systems on mortality can vary substantially as a function of the extent to which implementation strategies disrupt or delay critical activities in the clinical setting and demand additional time for order entry from clinical staff. Critically ill patients (i.e., those who are most vulnerable) are most likely to be affected by dysfunctional technology and implementation strategies.

Qualitative studies. Qualitative studies seek to understand phenomena and answer questions of why and how as well as to gain insights into real life situations. They often study the more human or “soft” side of health and health care. The preceding sections concentrated on studies with quantitative outcomes. Fifty-three qualitative studies are included in this section. Patient safety was the main health aspect evaluated in qualitative studies. Before MMIT implementation most studies found that clinicians expected that MMIT would improve patient safety and once implemented most clinicians felt that MMIT had improved safety.

The qualitative studies focused on system design including workflow changes, challenges with the system interface, and new communication processes—all of which can generate new kinds of medical errors, which in some cases were detrimental to patient safety.

Early implementers associated MMIT with a lot of self-reported “hard work” by those who were expected to use the new systems. These people, most often health professionals, struggled, often independently, with limited guidance with respect to planning and implementation tactics during preparation for and implementation of the MMIT applications. During planning and early implementation, the users often experienced unanticipated effects. Frequently, the initial stage was disruptive and, consequently, clinicians found provision of care to be more challenging with the MMIT system than without. However, after the initial stage was over, the attitudes of the care providers changed, and the potential benefits of the system become clearer to most. Of special note is that the implementation of MMIT systems generated emotional responses in a broad range of health professionals, both positive and negative. For example, strong feelings were associated with reminders and alerts19 and CPOE.20

MMIT implementation did not just mean that a clinician needed to learn a new IT system, but the implementation also affected most of the other parts of the delivery of care processes, including how the interdisciplinary care team worked together.

KQ2. Knowledge and Evidence Gaps We identified gaps in the report, some that we expected and some that we did not. We

address the question of knowledge deficits across phases and outcomes, settings and participants, grouping similar gaps together.

Phases of medication management. Because of the preponderance of publications on the prescribing and monitoring phases, they are less in need of more study than the other phases of order communication, dispensing and administering, and medication reconciliation. In addition, the educational or training requirements for effective use of MMIT applications by health professionals need to be studied as well as education related to patients as new MMIT applications are developed for their use.

Research methods. MMIT applications are complex interventions and need to be studied in pragmatic (i.e., does it work in real settings?) evaluation projects and using complex interventions methods. The applications also should ideally be studied by teams of researchers

ES-12

with, or teams that seek consultation from, those who have experience in clinical practice, research methods, statistical analysis, and informatics training and experience. Qualitative studies are also vital to understand the complex nature of how systems are used and valued, especially across groups of health professionals who often have different needs and expectations.

Health care providers. Physicians are well studied. Nurses, midlevel practitioners (nurse practitioners, physician assistants, midwives), pharmacists, other prescribers such as dentists and mental health practitioners, and hospital administrators need studies directed at their needs, practice patterns, and health IT tools.

Patients. Many studies included data related to patients, usually in the measurement and reporting of process changes and other outcomes. Few studies, however, concentrated on how the MMIT systems directly affected patients and clinical outcomes important to them. Traditionally, MMIT systems were developed as clinician and administrator tools. Patient and family use of MMIT systems is becoming more important, and this gap in our understanding needs to be addressed.

Settings. Hospitals and ambulatory care settings are well studied. Gaps exist in our knowledge of the effectiveness of MMIT in long-term care facilities, the community, and homes. Long-term care facilities most need strong qualitative and quantitative studies because they rely heavily on medication. Homes, schools, and other community settings will also become more important with shifting care to more self- reliance in relation to wellness care and chronic disease management.

Health IT. Much research has gone into evaluating CDSS and CPOE systems, either alone or integrated. For example, 77 of 88 RCTs evaluated some aspect of CDSSs. Other MMIT applications, especially those that are used by nonphysicians or outside the prescribing and monitoring phases, lack evidence. Examples with little evidence on effectiveness are bar coding for administering and dispensing, pharmacy information systems, electronic medication administration record systems, and fully integrated comprehensive information systems.

Process changes. Patient safety processes such as error reductions and improvement in prescribing have a strong evidence base. Issues related to workflow, communication changes, and unintended consequences are understudied. More study of laboratory-based monitoring of medications, especially in facilities that have highly integrated information systems, is important. More qualitative and controlled studies are needed as well as multicenter studies and those that use methods developed by groups focusing on health technology assessment (HTA). These HTA methods include integrated reports that bring together research syntheses, modeling of processes and full economic reports, and cost studies. Often these HTA reports do not, but can, involve additional collection of evidence.

Intermediate outcomes. More study is needed on the importance of usability testing in all stages of development and use. This must be done with all users and not just segments of those involved in using MMIT. Usability studies have not traditionally been generalizable or transferrable but more limited to a specific setting. AHRQ might consider a research program in how to make these usability studies more applicable to multiple institutions, training in usability

ES-13

methods, collection of usability tools and completed studies, and research into the need for standards of usability testing for new or modified systems. Usability studies must also include all users of systems. For example, systems that have been optimized only for physician users are usually systems that nurses and other health professionals have difficulty using. Workarounds have often been unofficially implemented by users instead of system modifications and improvements.

Clinical outcomes. Findings associated with improvement in clinical outcomes are still equivocal. These studies are difficult to do well, expensive, and time consuming, but they must be done. Multicentered trials planned by strong teams of experienced people from multiple backgrounds are vital.

Cost and economic outcomes. Although many studies exist that list costs and outcomes, few comprehensive and definitive studies of the economic value of MMIT applications exist. Both the potential for improvement and the costs of implementing and maintaining these systems are huge. Again, well-planned studies with broad input from many stakeholders are necessary for understanding the true worth of MMIT applications. HTA or other studies that integrate costs and consequences of MMIT systems would be ideal.

Qualitative. Qualitative studies have provided much valuable information about MMIT. Gaps in qualitative knowledge center on the lack of qualitative studies that address the effects of MMIT on health outcomes. In addition, very few qualitative studies examined the effects of MMIT from the perspective of the patient.

KQ3. Value Proposition for Implementers and Users Value proposition is determined from a balance of financial, clinical, and organizational

benefits. A clear assessment of each of these from the viewpoint of each stakeholder is needed to make a clear value judgment. For each stakeholder—and many are involved with MMIT implementation—the relative importance of these three elements is different. Values will also vary depending on the setting and the type of technology employed. Multiple stakeholders, some of whom may be distant from the MMIT, need to be considered in any value proposition study. Based on the evidence in KQ1: Effectiveness, knowledge about the three elements needed to make value judgments is slowly accumulating. We cite only 31 papers in this section, although some of our assessments come from sections of this report that have included more studies. Gains in productivity and process of care outcomes have been shown, but good evidence of improvement in patient outcomes with MMIT is weak or lacking. The body of economic literature is still sparse and lacks vigorous study. We found little theoretical work or actual studies that were done to determine what each stakeholder takes into account to reach value proposition judgments related to MMIT.

KQ4. System Characteristics Few studies (n = 21) demonstrated evidence of the impact of the characteristics of MMIT

applications on the likelihood to purchase, implement, and use such IT applications. No studies assessed open-source health IT applications, with only one study each on conformity with standards and CCHIT-certified systems. Twenty of the articles related to the prescribing and ordering phase. Almost all of the articles suggest that feature sets of health IT applications have

ES-14

been instrumental in reaching decisions to adopt MMIT applications. Certain features of systems improve the likelihood of purchase, implementation, and use of MMIT. The literature, however, is sparse and observational in nature. Most often authors described barriers and concerns toward implementation and acceptance rather than characteristics of MMIT that could facilitate implementation, purchase, and use of such systems. Authors seldom provided enough details about the technology to form conclusions about the value of feature sets and system characteristics. Head-to-head comparisons of systems differing in their features were not found.

KQ5. Sustainability Our literature review revealed three important findings: sustainability is frequently

mentioned in the core biomedical informatics literature, it is poorly defined, and none of the articles included in this evidence report explicitly studied sustainability. These findings are not entirely surprising. A previous AHRQ-sponsored evidence report that assessed the costs and benefits of health IT in pediatrics found only one article that explicitly discussed sustainability.21

Future research would be beneficial for many if a study or group would develop an operational definition of sustainability that could be used to study its determinants. Moreover, it is likely that the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 will lead to improvements and sustainability of health IT applications that specifically support the medication management continuum through meaningful use.

We have summarized a body of literature that uses surveys to detect patterns in the characteristics of people and organizations that are more likely to implement various technologies. These surveys are often the basis for further study into barriers and facilitators to increasing uptake and adoption.

Integration of MMIT with other systems was an inclusion criterion for our report (except for PDAs that analyzed patient-specific data). Some technologies were integrated with a greater number of components than others. Frequently, the descriptions of the systems were inadequate to fully determine how the systems were connected. Access to various other information sources, most notably laboratory reports, enhanced the performance and acceptance of the MMIT applications.

KQ6. Complete Two-Way Electronic Data Interchange No reports documenting the use of complete two-way EDI systems were found. Evidence

from the limited set of one-way, e-Prescribing studies was extrapolated to identify possible key facilitators and barriers to completely electronic, two-way, e-Prescribing systems. Possible facilitators include monetary or other incentives to providers, a permissive regulatory environment, and the existence of an established standard for prescription EDI. Barriers included the low rate of EMR adoption in the United States, regulatory and legal uncertainties, and inadequate consideration of the effects of e-Prescriptions on pharmacists and pharmacies and their processes. While answering this question, we found that the Bell model does not represent the two-way communication between pharmacists and prescribers—it shows only a one-way linear movement of information.

KQ7. Effectiveness of CDSS Seventy-seven RCTs were designated as primarily studying CDSS related to medication

management and integrated with other health IT. These studies involved 4,709 providers and 828,441 patients in total. All studies assisted with at least the prescribing or monitoring phases of

ES-15

medication management. Overall, we found a lack of RCTs addressing electronic decision support integrated with other types of health IT. Statistically significant process changes were often shown in these RCTs. Only a small