health information systems in clinical settings
DESCRIPTION
HISTRANSCRIPT
ORIGINAL PAPER
Methods to Evaluate Health information Systemsin Healthcare Settings: A Literature Review
Bahlol Rahimi & Vivian Vimarlund
Received: 1 February 2007 /Accepted: 6 July 2007 / Published online: 27 July 2007# Springer Science + Business Media, LLC 2007
Abstract Although information technology (IT)-basedapplications in healthcare have existed for more than threedecades, methods to evaluate outputs and outcomes of theuse of IT-based systems in medical informatics is still achallenge for decision makers, as well as to those who wantto measure the effects of ICT in healthcare settings. Theaim of this paper is to review published articles in the areaevaluations of IT-based systems in order to gain knowledgeabout methodologies used and findings obtained from theevaluation of IT-based systems applied in healthcaresettings. The literature review includes studies of IT-basedsystems between 2003 and 2005. The findings show thateconomic and organizational aspects dominate evaluationstudies in this area. However, the results focus mostly onpositive outputs such as user satisfaction, financial benefitsand improved organizational work. This review shows thatthere is no standard framework for evaluation effects andoutputs of implementation and use of IT in the healthcaresetting and that until today no studies explore the impact ofIT on the healthcare system’ productivity and effectiveness.
Keywords Evaluation studies .Medical informatics .
Literature review
Introduction
Information technology (IT)-based applications in health-care have existed for more than three decades and havegained widespread use [1, 2]. Nowadays, it is hard toimagine healthcare without IT-based applications for boththe accumulation and interchange of clinical information[3]. This is in part because IT has been recognized as an“enabler,” that is, as a tool that offers solutions to theproblem of the increasing accumulation of patient data [4, 5].Because of their central role enabling the diverse use of in-formation, IT systems ensure the timely and accurate collec-tion and exchange of information, and thus a more efficientuse of the scarce resources of healthcare organizations.
With an increased need for the implementation of IT inall healthcare domains—such as primary healthcare andclinical settings or home healthcare environments—for thepurpose of providing an optimal use of resource invest-ment, its use is expected to rise. Evaluating such ICTapplications to help decision makers acquire knowledgeabout the impact of IT-based systems therefore becomes akey issue to all organizations that aim to implement anynew application [6].
However, despite the fact that evaluation studies haveincreased in importance, they usually only provide answersto questions such as why the system should be studied, whya specific IT application should be chosen among manyother systems, or why they only present a general pictureabout costs and benefits to both users and the organization[7]. Case studies that examine the costs and benefits ofspecific IT applications are the most common examples ofempirical analysis, but the published work is neitherextensive nor comprehensive. Studies discussing evaluationmethodologies in the area of medical informatics haveusually been performed descriptively, and often use
J Med Syst (2007) 31:397–432DOI 10.1007/s10916-007-9082-z
B. Rahimi (*) :V. VimarlundDepartment of Computer and Information Science,Linköping University,Campus valla,Linköping 58183, Swedene-mail: [email protected]
B. RahimiDepartment of Social Medicine,Urmia University of Medical Sciences,Urmia, Iran
approaches from domains such as computer science,cognitive science, economics, and organization, and usuallydo not use in a multi-actor focus that includes the conse-quences of the implementation and use of IT-based systemsfor all participants, including care recipients and stake-holders, involved in the healthcare process.
The aim of this review is to outline the methodologicalapproaches and results of studies that measure the impact ofIT on healthcare organizations during 3 years (between2003 and 2005). We limit the study to review evaluations ofhealth information systems, including computer-basedpatient records, electronic health records, electronic medicalrecords, telemedicine, and decision support systems.
Previous literature reviews
There are several previous literature reviews in the field ofevaluation in medical informatics before 2003. They have,however, been conducted with several different aims, andtheir results are therefore difficult to compare. In thissection we present three reviews relevant to this currentpaper; all related to the evaluation of effect of IT-basedapplications for either clinical decision support systems orcomputer-based patient records systems.
Kaplan [72] reviewed studies focusing on the evaluationof clinical decision support systems (CDSS), with the mainemphasis upon changes in clinical performance andsystems that could improve patient care. Kaplan’s studyincludes many evaluations of CDSSs using designs basedon laboratory experiments or Randomized ControlledClinical Trials (RCTs). The author used Medline for aliterature search from 1997 to 1998. In addition to this, amanual search identified 27 papers referenced frequently byother experts in medical informatics, and these are includedin Kaplan’s review. Kaplan concluded that conductingRCTs is the standard method of evaluation approach forCDSS. She argued that RCT-type studies could not beuseful to investigate what influences whether systems areused. She also concluded that RCT-type studies do notanswer questions such as why some systems tend to beused while others are not, why the same system may beuseful in one setting but not in another, why a CDSS mayor may not be effective, or why different results areobtained in different studies. For this reason, more researchin this area is needed.
Ammenwerth and Keizer’s [71] review identified thetrend of evaluation research in the area of medical infor-matics from 1982 to 2002. The aim of the review was toanswer questions such as which countries or journalsdominate the evaluation research publications, which typesof information systems have been evaluated, which aimshave been studied in selected papers, which methods wereapplied for evaluation, how many papers have been
published, in which locations and settings did the studiestake place, and which evaluation strategy has been applied.Ammenwerth and Keizer’s review was based on a sys-tematic literature search in PubMed from 1982 to 2002,during which time 1,035 articles were selected. The authorsconcluded that the number of evaluation studies in the areaof medical informatics is rising significantly. However,evaluation studies cover only around 1% of all medicalinformatics publications indexed in PubMed. Ammenwerthand Keizer noted that explanatory research and quantitativemethods dominated evaluation studies during the periodchosen for analysis. The most common, recurring aspectsof the evaluation in the studies reviewed were appropriate-ness of patient care, efficiency of patient care, usersatisfaction, and software quality. They showed thatcomparable development in medical informatics since1982 was that the number of lab studies focusing ontechnical aspects was found to have declined. They alsoshowed that focus upon the quality of care processes andpatient outcomes was found to have increased. Theyinterpreted this shift as a sign of the maturation of evaluationresearch in medical informatics.
Delpierre et al. [8] reviewed studies of computer-basedpatient record systems (CBPRS). The objective of theirreview was to carry out a systematic survey of studiesanalyzing the impact of CBPRS on medical practice,quality of care, and user and patient satisfaction. Theyreviewed selected papers published from January 2000 toMarch 2003, identified by their search through Medline,Cochrane, and Embase databases. They selected a total of26 articles using this method. The authors concluded thatthey could show a better understanding of the relationshipbetween the use of CBPRS and medical practice. Anincreased satisfaction of users and patients was noted,which could lead to significant changes in medical practice.They also noted that most of the studies did not includequalitative factors such as characteristics of the disease andthe tool, the ward in which it was developed, and therelationship between various healthcare professionals,which could have an impact upon the use of CBPRS.Delpierre et al. [8] suggested in his study, a broad reviewincluding all the factors that may influence the success orfailure of the use of CBPRS in medical practice.
Some notable differences between the included reviews
The following table categorizes the differences and simi-larities among the three reviews discussed above.
As can be seen in Table 1, evaluation studies have beenperformed with different aims and objectives, and are con-cerned with different domains and areas of focus. However,all of them are focused on identifying trends in evaluationstudies, analyzing the impact of CPR, or to identify how
398 J Med Syst (2007) 31:397–432
CDSS changes clinical performance. Despite the largenumber of studies included in the reviews, there are nostudies that look specifically at the impact of IT onhealthcare efficiency or productivity. Almost all evaluationsare therefore partials, and the results are difficult to gen-eralize from each particular study to a broader context.Most of the studies that explored the impact of IT con-sidered decision support systems or electronic healthrecords. The main benefits identified to quality areincreased adherence to guidelines, better surveillance andmonitoring, and decreased medical error. However none ofthe reviews found studies that have a clear impact on laborinput or on the efficacy of the provision of care.
Materials and methods
A literature review was performed for published evaluationstudies of IT-based systems in healthcare, including CPR(including EMR and EHR), telemedicine, and differentkinds of DSS related to information systems like CPOE,between January 2003 and March 2006.
The search for the related material started in January2006 and finished in March 2006. Linköping University’sdatabase was used to gain access to papers on this subject.The keywords used to search for the articles were: patientrecords, medical Records, health records, informationtechnology, medical informatics, healthcare information,health informatics, hospital information system, patient careinformation system, CPOE, evaluation methods/theory,assessment, appraisal, information system/technology, eco-nomic evaluation, and evaluation study. In addition, themost important database, MEDLINE, was used to searchfor related papers. We conducted our search with the help
of search rules, for example MeSH (major topic), language,publication type, and publication date.
The following strategy was used to select the articlesfrom MEDLINE:
(1) We searched using the MeSH database, using theterms computerized patient records (CPR), decisionsupport systems, clinical (DSS), and telemedicine.
(2) In the next step we searched MeSH for evaluationstudies and found two categories called “evaluationstudies” and “Evaluation Studies [Publication Type].”
(3) We combined step 1 and 2 according to the followingdetails: (“Evaluation Studies”[MeSH] OR “EvaluationStudies”[Publication Type]) AND (“Medical RecordsSystems, Computerized”[MeSH] OR “Decision Sup-port Systems, Clinical”[MeSH] OR “Telemedicine”[MeSH])
(4) We limited these reviews to articles published between2003 and 2005 in English and with the aim of toevaluate some IT applications. We hit 674 articles.
We excluded papers concerning software and technicalevaluation, as well as ethical and education-related evalua-tions. We also excluded some other evaluation articles thathad no direct relationship with information systems, such asdocumentation and ordering through computers. Sixty-onearticles were left at the end of this review process. Thirty-eight studies had been performed in hospital settings, 15studies in primary care, and nine studies in homecare, withsome of the studies performed in two domains at the sametime. Two studies were performed in all domains (hospital,primary care and home care). Some projects, like IDEATelor Kaiser Permanente telehomecare, had a larger samplesize and publication. However, we limited our review tothis range, and excluded studies published before 2003 orafter March 2006.
Table 1 Differences and similarities among Kaplan, Ammenwerth, and Delpierre’s reviews
Review category Kaplan review Ammenwerth et al. review Delpierre et al. review
Aim of study How CDSS changes clinicalperformance
To identify trends of evaluation studies Analyse the impact of CPRon medical performance
Domains in whichsystems are evaluated
Clinical area National, Hospital, primary care, homecare
Hospital, primary care, homecare
Type of systemsevaluated
Published articles which included CDSSfrom 1997 to 1998
All kinds of systems—included publishedarticles from 1982 to 2002—onlyabstracts were reviewed
Published articles from 2000to 2002 based on CPR,CDSS, and CPOE
Study design Mostly RCT All kinds of study designs Mostly RCTThe approaches ofpapers which havebeen reviewed
Mostly quantitative approach Both quantitative and qualitative, butquantitative methods dominated
Mostly quantitative approach
Focus of evaluationstudies
Usefulness of system improved carewith RCT and show RCT as a goodmethod for evaluation
Appropriateness of patient care, usersatisfaction, and software quality
Clinical practice, patientoutcome, user and patientsatisfaction
J Med Syst (2007) 31:397–432 399
Results
In this section we will present a summary of the 61published articles in the medical informatics area from 2003to 2006 previously classified as evaluation studies. Thefollowing three tables present a brief description of thenames of the authors, the domains the study was performedin, the design of each study, the time of evaluation, thesources of evidence, the aim of the study, and the findings.The studies are divided into the following main categories:CPR evaluation studies, telemedicine evaluation studies,and DSS evaluation studies. In Tables 2, 3 and 4 wesummarize the evaluation studies classified as evaluation ofCPR, telemedicine, and DSS:
Comments
Evaluations of CPR tend to be concerned with systemusefulness regarding the quality of care, and user-relatedissues such as user acceptance and satisfaction and attitudestowards new systems, as well as the financial effects,usually limited to the identification of the costs of systemimplementation [9–11, 12–22].The inclusion of indirectcosts is not a common procedure, at least, in the studiesincluded in this literature review Usually the evaluationsclassified as “economic” include only the direct costs of theimplementation of the new system [11, 14, 23].
Another important remark is that some of the studiesincluded in this review tended to evaluate the effects of thenew system’s implementation on the quality of workperformance, such as user job performance and computerknowledge, and investigation of skill among other users[24–28].
From Table 2 it can be seen that introducing a newcomputerized patient record system had found to havepositive effects such as economic benefits [9, 11, 14], highacceptance score and satisfaction among the users in theimplemented sites [10, 13, 16, 21, 24, 29] and alsoImprovements in management and work process [11, 16,18, 22, 24, 28].
From Table 3 we can see that evaluations of tele-medicine tend to be concerned with the economic effects ofsystem implementation [30–34]. Usually the outputs areexpressed as costs and benefits, or effectiveness in thetelemedicine area. There are also, however, studies regard-ing user attitudes and perspectives, user satisfaction [35–38],and the usefulness of the system [30, 39–43], such as timeof service delivery, usability, feasibility, and changes in thenumber of visits after implementation of the new system.
From Table 3 it can be seen that introducing atelemedicine system had positive effects such as reducingspent time per patient during the visiting by clinical staff
[41, 42], economic benefits [30, 31, 39], and also quality ofcare [30, 39, 42, 44].
Table 4 provides insight into some issues regardingevaluations of DSS. It can be seen from Table 4 that mostof the systems that have been evaluated are CPOE, with theaim being to evaluate the usability of systems [45–49] orthe effectiveness of the system for patients [32, 50–53].Some of the studies reviewed tend to also evaluate thefinancial impacts of introducing the new system [54, 55]thus without systematic identification of all costs. There areeven some in this area that included measures of usersatisfaction and attitudes towards the system [31, 56].
From Table 4 it can be seen that introducing a clinicaldecision support systems in healthcare organizations hadpositive effects such as improved quality of care [46–49],satisfaction among users in the implemented sites [56], andalso improvements in management and work process [45,50, 52, 57].
However, most of the studies did not discuss a specifictheory to be applied when evaluating IT-based applicationsin healthcare. Few studies presented discussion of someeconomic theories such as cost-benefit/effectiveness analy-sis [9, 31–33], and none generated new theories or extendedolds ones.
It is also interesting that many studies used formativetests as an evaluation method, and a high proportion ofstudies used summative tests. Moreover, more than half ofthe papers used summative tests simultaneously. Focusingon most of the articles included in our study, we find thatthey have used objective and subjective perspectivessimultaneously.
Most of the studies based on the financial model, likecost-benefit/effectiveness, showed that there were improve-ments with the introduction of the new systems, especiallyin the telemedicine area. In contrast, some studies showedthat the implementation of new DSS or Telemedicine hadno economic benefits [32, 33, 54], and few showed that theintroduction of the new CPR or DSS were problematic[58–61].
Discussion
The purpose of this paper was to conduct a literature reviewfocusing upon the evaluation of IT-based systems inhealthcare. The results show that several similar studieshave been performed that focus upon clinical supportsystems, CPR for medical practice, telemedicine, decisionsupport systems, etc. Previous review studies concludedthat experimental designs are excellent for studying systemperformance or changes in clinical practice behaviour, butnoticed that evaluation approaches such as usability testing,cognitive studies, ethnography studies or socio-technical
400 J Med Syst (2007) 31:397–432
Tab
le2
CPRevaluatio
nstud
ies
Autho
rsDom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
ies
Finding
Sam
uelJ.
Wanget
al.
[200
3][9]
Primary
care
Survey
After
implem
entatio
nEcono
micdataon
costsand
benefitscamefrom
patient
medical
records
–Exp
ertop
inion
Toestim
atethenetfinancialbenefitor
costof
implem
entin
gelectron
icmedical
record
system
sin
prim
arycare
during
aperiod
of5years.
Thisstud
yshow
sthat
apo
sitiv
ebenefitof
abou
t$86,400was
obtained
bytheorganisatio
nover
the
period
of5years.The
implem
entatio
ncost
invo
lved
inthewho
leprocesswas
$46
,400
and
catego
rizedinto
twotypesof
costs:system
cost
(system
implem
entatio
n,supp
ortandmaintenance,
andhardware);andindu
cedcost(tho
seinvo
lved
intransm
ission
from
paperto
electron
icsystem
).The
benefitsob
tained,werecatego
rizedas;A)payer-
independ
entbenefits(obtainedfrom
thefactors
such
aschartpu
llandtranscription);B)capitated
patient
benefits(e.g.from
theadvancedrug
events;
from
theutilizatio
nof
drug
labo
ratories
and
radiology);andC)fee-for-servicepatients(w
hich
includ
edbettercharge
captureandredu
cedbilling
costsdu
eto
less
errors.)Significant
benefitfactors
werefoun
dto
be:saving
sin
drug
expend
iture
by33
%;theredu
ctionin
theutilizatio
nof
radiolog
yby
17%;im
prov
ementin
thecaptureof
charge
and
decrease
inbilling
costwhich
equalledto
15%
each
ofthetotalbenefits.
Hallvard
Laerum
etal.[200
3][10]
Hospital
Survey
After
implem
entatio
n–Q
uestionn
aire
(openended
questio
ns)completed
by70
physicians
–Interview
with
8ph
ysicians
Evaluatingtheeffectsof
scanning
and
elim
inationof
paperbasedrecordsby
stud
ying
physicians
andtheirattitud
etowardthesystem
Thisstud
yshow
sthat
scanning
andelim
inating
paperbasedmedical
recordswerefoun
dto
beacceptable,asitprov
ided
access
totheoldmedical
recordsin
theform
ofan
imageforthedo
cument.
The
stud
yalso
show
sthat
scanning
papers
are
beingused
with
ahigh
frequency,
butitis
considered
only
aninterm
ediate
stagetowards
having
fullaccess
totherecords.Thisstud
yshow
spo
sitiv
eresults
regardingtheperformance
ofthe
medical
tasksanduser
satisfaction.
The
useof
the
system
asawho
lewas
foun
dto
offersatisfactory
functio
nality,
i.e.forinform
ationretrieval,for
exam
ple,
9ou
tof
11tasksbecomeeasier
toperform.Moreover,ph
ysicians
also
foun
diteasier
toworkwith
thesystem
whereby
partof
the
system
hand
lingwas
combinedwith
theregu
lar
electron
icdata.How
ever,no
tallph
ysicians
were
foun
dto
besatisfied
with
theuseof
electron
icmedical
recordsdu
eto
basicdraw
backssuch
as,
J Med Syst (2007) 31:397–432 401
morenavigatio
ntim
eisrequ
ired
whenaccessing
thedata
ascomparedto
thetim
eused
indirect
interaction,
andthereislim
itedfunctio
nalityof
the
scannedim
ages
dueto
therigidstructureof
the
system
.Cornelia
M.
Ruland
etal.[200
3][11]
Hospital
Coh
ort
After
implem
entatio
n–E
cono
mic
data
extracted
from
nursingrecords
–Sem
istructured
Interview
(7nu
rsemanagers)
–Focus
grou
pwith
nurses
–Questionn
aire
to7nu
rse
managers
Todescribe
theeffectsof
thenew
system
for
evaluatio
non
:costredu
ctions;financial
managem
ent,anddecision
making
Thisstud
yshow
sthat
during
theevaluatio
nperiod
of4mon
ths,theim
plem
entatio
nof
CLASSICA
hadan
overallpo
sitiv
eim
pact
oneffectivenu
rsing
resource
managem
ent.Duringtheevaluatio
nperiod
test,un
itshada41
%redu
ctionin
expend
iture
comparedwith
theaverage
expend
iture
before
theintrod
uctio
nof
thenew
system
.Thismeans
that
theim
plem
entatio
nof
the
system
resultedin
better-costmanagem
ent.A
sign
ificantim
prov
ementin
inform
ation
managem
entw
asalso
noticed.Itw
asalso
observed
that
itbecameeasier
toanalysetherelatio
nship
betweenpatient
activ
ities,staffing
,as
wellas
the
costof
care.Highuser
satisfactionwith
thesystem
andtheinform
ationanddecision
supp
ortit
prov
ided
was
observed.The
system
was
foun
dto
beeasy
touse.
The
interviewsregardingtheease
ofuseanduser
satisfactionrevealed,accordingto
theauthors,po
sitiv
eresults.Itwas
also
observed
that
ithadbecomeeasier
torecogn
izethecauses
andexplanations
forbadandgo
odresource
managem
ent.
AdamsW.G.
etal.[200
3][64]
Hospital
and
Primary
care
Pre–p
ost
interventio
nanalysis
Beforeandafter
implem
entatio
nReview
ofmedical
records
(235
paperbasedvisits
and98
6compu
terbased
visitsdo
cuments)
Toevaluate
thequ
ality
ofdo
cumentatio
nand
deliv
eryof
paediatric
prim
arycare
before
and
aftertheim
plem
entatio
nof
anEMR
The
find
ings
ofstud
yshow
that
userswere
prom
pted
toaskabou
triskfactorsandcouldseeby
thetemplatedesign
that,theiransw
erswou
ldbe
availableat
thenext
visit,theseitemsare
frequently
lost.The
quality
ofanticipatory
guidance
also
improv
edconsiderably.The
outputs
ofthesystem
werewellaccepted
byfamilies.
Nearlyallusersalso
agreed
that
thesystem
improv
edthequ
ality
ofgu
idance
givento
families.
Clin
icians
areoftenov
erwhelm
edby
time
limitatio
nsin
theprim
arycare
setting
.The
electron
icmedical
records(EMR)im
prov
edthe
completenessof
deliv
eryof
ageapprop
riate
Tab
le2
(con
tinued)
Autho
rsDom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
ies
Finding
402 J Med Syst (2007) 31:397–432
guidance.Also6of
the7clinicians,who
use
compu
ter-basedrecords,repo
rted
usingthesystem
for95
%to
100%
ofno
nurgentcare
visits.Nearly
allusersagreed
that
thesystem
was
easy
touse,
increasedcompleteness,andremindedthem
todo
things
that
they
might
otherw
iseforget.4of
7usersfeltthat
thesystem
extend
edthedu
ratio
nof
visits.Mostusersalso
agreed
with
thestatem
ent
that
useof
thesystem
redu
cedeye-to-eye
contact.
Allusersagreed
thatthesystem
shou
ldcontinue
tobe
used.
Hou
ston
T.K
etal.[200
3][12]
Hospital
Survey
After
implem
entatio
nInterview
with
93patients
and82
physicians
Tosurvey
patients’
perceptio
nsof
hand
held
compu
teruseby
physicians,and
compare
those
with
theirprov
iders’
perceptio
ns
The
find
ings
ofthisstud
yshow
that
thegeneral
attitud
etowards
compu
ters
intheexam
ination
room
was
positiv
eandgenerally
patientsat
that
clinic
hadpo
sitiv
eattitud
estowardhand
held
compu
ters.33
patientswho
neverused
compu
ters
less
frequently
repo
rted
that
they
liked
theidea
ofresidentsusinghand
held
compu
terscomparedwith
thosewho
didusecompu
ters.Patientswho
seph
ysicianhadused
ahand
held
compu
terhada
sign
ificantly
morepo
sitiv
eattitud
etoward
hand
held
compu
ters.The
patientsexpressedthat
theuseof
hand
held
compu
tercouldbe
very
useful
inhelpingthem
toim
prov
ethecare
that
they
received.66
%ph
ysicians
agreed
that
therewas
enou
ghtim
eto
usethehand
held
compu
terin
the
clinic
practices.Som
eph
ysicians
(23%
)repo
rted
reservations
abou
tusingthehand
held
with
patients.
Elske
Ammenwerth
etal.[200
3][13]
Hospital
Coh
ort
Beforeand
during
and
after
implem
entatio
n
–Questionn
aire
completed
by39
nurses
–Interview
with
12nu
rses
Toevaluate
theprecon
ditio
nsandconsequences
ofcompu
ter-basednu
rsingprocess
documentatio
nwith
aspecialem
phasison
acceptance
issues
amon
gnu
rses.
Thisstud
yshow
sthat
sign
ificantly
high
initial
acceptance
scores
wereob
served
inallwards
before
theintrod
uctio
nof
thecompu
terbased
documentatio
nsystem
.A
difference
betweenthe
psychiatricwards
andthesomatic
wards
was
noticed
with
ahigh
erinitial
acceptance
scorefor
thesomatic
wards,as
thepsychiatricwards
were
nearly
completelyelectron
ically
documented.
No
overallchangesin
theacceptance
scores
were
noticed
during
thestud
y,ho
wever
rather
high
ermeanscores
wereob
served
onallwards.The
introd
uctio
nof
thecompu
ter-baseddo
cumentatio
n,ho
wever,didno
tseem
tohave
anymeasurable
influenceon
thegeneralacceptance
ofcompu
ters.
Acceptancescores
from
medium-to-high
were
J Med Syst (2007) 31:397–432 403
observed
aftertheim
plem
entatio
n.In
thecase
ofwardC(one
ofthewards)theinitial
acceptance
scaledeclined
andthen
rose
again.
Itwas
noted
that
meanscores
typically
decreasedafterthe
introd
uctio
nof
thesystem
,asseen
inotherstud
ies,
thus
show
ingadecrease
intheattitud
etowards
itsuseov
ertim
eat
theinterm
ediate
stage.
The
level
ofacceptance
bytheuserswas
observed
tobe
rang
ingfrom
medium
tohigh
.The
majority
ofthe
interviewsshow
edtheirlik
enesstowards
the
continuatio
nof
workwith
thePIK
inthefuture.
Thu
s,theov
erallacceptancescores
werequ
itehigh
andwerefoun
dto
rise,except
inthecase
ofward
C.The
high
eracceptance
scores
werefoun
dto
becorrelated
with
thenu
rse’syearsof
compu
ter
experience,aswellaswith
thehigh
erinitialscores.
Noinitial
know
ledg
eof
compu
ter,lack
ofconfidence
inuseandoldageof
staffwerethou
ght
tobe
thereason
sbehind
thelower
scores
inthe
case
ofwardC.Thu
s,itwas
concludedthat
compu
terkn
owledg
eandprevious
acceptance
ofnu
rsingprocessesweretheprerequisitesforthe
high
results
regardinguseandacceptance
ofthe
system
.W.P.
Zhang
etal.[200
4][14]
Hospital
Survey
After
implem
entatio
nQuestionn
aire
open-end
edqu
estio
nsto
81ho
spitals
Toclarifytheim
plem
entatio
nandmaintenance
costsof
acompu
terizedpatient
record
(CPR)
system
inho
spitals
The
find
ings
ofthisstud
yindicatethatCPRsystem
sim
prov
edtheho
spitaladministrationandmedical
exam
inations
practices.Itwas
foun
dthat
the
design
erof
thesystem
hadthegreatestim
pact
ontheperbedim
plem
entatio
nam
ongthevariou
sfactorssurveyed.Num
berof
servers,institu
tion
type,andim
plem
entatio
ndate
also
played
anim
portantrole
intheperbedim
plem
entatio
ncost.
The
exam
inationof
perbedim
plem
entatio
ncost
byho
spitalsize
show
edthat
themeancosttend
edto
bethelowestformid-sizeho
spitalsbu
thigh
erforsm
allas
wellas
largehospitals.E
xaminationof
annu
almaintenance
costsagainsttotal
implem
entatio
n,ho
wever,show
edno
difference
amon
gthethreesizescatego
ries.Morethan
60%
oftheusers(hospitaldirectors,system
managers,
Tab
le2
(con
tinued)
Autho
rsDom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
ies
Finding
404 J Med Syst (2007) 31:397–432
nurses
andgeneralstaffmem
bers)respon
dedwith
“veryhigh”or
“high”
satisfaction.
Anov
erall
improv
ementin
themedical
care
was
observed
bytheintrod
uctio
nof
CPRsystem
.From
the
improv
edho
spitaladministration(w
ith80
%po
sitiv
erespon
se),user
satisfaction(w
ith54
%po
sitiv
erespon
se)andbetterpresentatio
nof
inform
ation(w
ith75
%po
sitiv
erespon
ses);the
overallop
timistic
attitud
ewas
foun
dto
beevident.
How
ever,abigpercentage
of82
%respon
ded
negativ
elyregardingthedisadv
antagesof
CPRi.e.
confusionon
installatio
n,po
oruser
friend
liness,
andincreasedstaffworkloaddu
ring
learning
and
training
.Laerum
Het
al.[200
4][15]
Hospital
Survey
After
implem
entatio
n–Q
uestionn
aire
completed
by79
medical
secretaries,
172nu
rses,70
physicians
–Interview
with
8–12
representativ
emedical
secretaries,nu
rses,
physicians
Toevaluate
useof
andattitud
esto
aho
spital
inform
ationsystem
bymedical
secretaries,
nurses,andph
ysicians
The
find
ings
show
thatthenu
rses
andtheph
ysicians
usetheHIS
muchless
than
themedicalsecretaries.
The
medical
secretariesrepo
rted,ho
wever,that
all
definedtasksregardingcomputer-based
applications
wereperformed
moreeasily
than
paper-basedapplication.
The
nurses
andph
ysicians
notedthat
theelectron
icho
spitalinform
ation
system
hasless
simplifiedtheirworkthan
medical
secretaries.The
medical
secretariesweremuch
moresatisfies
with
theuseof
theHIS
than
the
nurses
andph
ysicians,particularly
with
theuseof
scanneddo
cumentim
ages.The
secretariesgave
sign
ificantly
morepo
sitiv
erespon
sesthan
the
nurses
andtheph
ysicians
toHIS.Nursesand
physicians
wereless
satisfied
with
usingthe
system
.The
mostfrequently
repo
rted
prob
lems
amon
gtheph
ysicians
consistedof
variou
ssoftware
andhardware-relatedprob
lems,thesystem
working
tooslow
ly,andlack
ofcompu
ters
where
theclinical
workwas
beingdo
ne.Bothnu
rses
and
physicians
inthemedical
wardfoun
dthat
patient
data
weremoreaccessible
whenstored
electron
ically
than
whenstored
onpaper,in
particular
regardinglabtestdata.
DanielB.Hier
etal.[200
4][16]
Hospital
Survey
After
implem
entatio
nQuestionn
aire
distribu
tedto
330faculty
andho
use
staffph
ysicians
(closed
andop
enended
questio
ns)
Todeterm
inewhether
anydifferencesexist
betweenho
usestaffandfaculty
physician’s
acceptance
ofan
electron
ichealth
record
system
.
The
stud
yshow
sthat
theuser
acceptance
ofthe
EHRwas
high
(86%
)forbo
thfaculty
physicians
andho
usestaffph
ysicians.How
ever,ho
usestaff
physicians
show
edsign
ificantly
high
erfrequency;
88%
preferredtheEHRto
apaperrecord
and75
%said
they
enjoyedusingtheEHR.64
.7%
ofthe
faculty
mem
bersindicatedthey
prefer
itand60
.6%
J Med Syst (2007) 31:397–432 405
show
edtheirinterestin
itsuse.
88%
oftheho
use
staffand64
.7%
ofthefaculty
believedthat
the
EHR
increasedavailabilityof
medical
records,
easedthedeterm
inationof
thetreatin
gph
ysician,
increasedtheability
tocommun
icatewith
other
physicians,increasedthelegibilityof
documentatio
n,andeasedaccess
tolabo
ratory
results.A
majority
ofho
usestaffph
ysicians
believedthat
theEHRmadeworkflow
more
efficient,redu
ceddu
plicateor
unnecessarytesting,
savedtim
edo
cumentin
gcare,andredu
cedpatient
care
errors.Majority
ofthefaculty
shared
their
view
onthesequ
estio
nnaire
items.System
speed
andlack
ofcompu
ters
wereconsidered
tobe
the
greatestbarrierby
both
grou
ps.
Joseph
K.
Rotichet
al.
[200
4][17]
Primary
care
Tim
eseries
Beforeandafter
implem
entatio
nTim
emeasurement
Evaluationof
theim
pact
ofintrod
ucingCPRon
timeof
healthcare
deliv
erywith
intherural
health
centre
The
find
ings
ofthisstud
yshow
that
theMosoriot
Medical
RecordSystem
(MMRC)clearlychanged
theflow
ofhealthcare
intheMosoriotR
uralHealth
Center(M
RHC).Asubstantialdecrease
inthetim
eof
patient
interactionwith
healthcare
prov
ider
was
observed.The
usingof
thesystem
show
edaclear
redu
ctionof
10minutes
(from
41to
31minutes)in
each
visitforthepatients,i.e.patientswho
were
noticed
tospendless
timewaitin
g.For
the
healthcare
prov
iders(nursesandclinical
officers)
thetim
e–motionredu
ctionwas
also
observed.
Theyspentless
timewith
patientsandotherstaff
andhadas
aresult,
moretim
eforperson
alactiv
ities.For
thenu
rses
andclerks,ho
wever,it
show
edmoretim
espentin
registratio
nbu
tless
timeforwritin
grepo
rts(3%
vs.18
%).Sim
ilarly
therewas
anincrease
inthetim
eforsearchingthe
inform
ationthroug
hthesystem
(3%
vs.0,
5%).
Overall,
inthisstud
y,itwas
agreed
that
asimple,
inexpensive,
andeffectiveelectron
icmedical
record
system
couldbe
establishedin
adeveloping
coun
trywith
poor
resource;thebenefitsof
which
will
berealised
inthelong
runin
term
sof
finances
andtherewill
bebetterhealth
services
served
tothepeop
le.
Tab
le2
(con
tinued)
Autho
rsDom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
ies
Finding
406 J Med Syst (2007) 31:397–432
DrikSteng
elet
al.[200
4][18]
Hospital
Rando
mized
controlled
trial
After
implem
entatio
nDataob
tained
from
patient
medical
records
–Archivedecon
omic
recordsassessed
byexpert
observers
Todeterm
inewhether
theintrod
uctio
nhand
held
compu
ter-baseddo
cumentatio
ncouldim
prov
ebo
ththequ
antitativeandqu
alitativ
easpectsof
medical
records
From
thisstud
y,itwas
foun
dthattheintrod
uctio
nof
ahand
held
compu
terinto
daily
routines
increased
thenu
mberof
diagno
sisrecorded
andalso
itim
prov
edtheov
erallqu
ality
ofthepatient
records.
The
stud
ydemon
stratedthattheratin
gson
allthree
investigated
aspectsof
documentatio
nqu
ality
were
betterwith
theelectron
icdevice
than
with
conv
entio
nalhand
writtendo
cumentatio
n.The
timerequ
ired
forhand
held
compu
ter
documentatio
nwas
also
foun
dto
besign
ificantly
decreased.
Itwas
also
noticed,ho
wever,that
the
difference
inthenu
mberof
diagno
seswas
slightly
balanceby
theaccompany
ingload
ofredu
ndant
itemswhich
was
said
tobe
sign
ificantly
infavo
urof
theexperimentalhand
held
documentatio
nsystem
.The
conclusion
basedon
thestud
yclearly
affirm
sthepracticability,accurateness,and
accessibility
ofthedo
cumentatio
nas
comparedto
thetradition
almetho
ds.
Likou
rezosA
etal.[200
4][19]
Hospital
Survey
After
implem
entatio
n–Q
uestionn
aire
completed
by23
physicians,and21
nurses
Tomeasure
clinicians’compu
terbackgrou
ndand
experience,satisfactionwith
,perceptionof,and
concerns
aboutan
EMR
inem
ergency
department(ED)
The
find
ings
ofstud
yshow
that
allclinicians
were
know
nwith
compu
terandalso
hadused
compu
ter
before
than
thenewsystem
isinstalled.
Mosto
fthe
physicians
werefamiliar
with
theInternet
asthey
used
everyday.Allclinicians
weretrainedanduse
theem
ergencydepartment(ED)EMR.Eighty-
sevenpercentof
physicians
and85
.7%
ofnu
rses
useEMReveryday.
The
similarpercentage
has
access
totheEMR
from
anyplacein
theclinical
setting
.Easyto
enterdata,access
data,read
data
onthescreen
werefoun
dto
befacilitated
bythe
EMRrepo
rted
byclinicians.Mostnu
rses
butno
tph
ysicians
perceive
that
patient
inform
ationis
moreconfidentialwith
EMR
than
with
paper
record.A
high
erpercentage
ofnu
rses
than
physicians
perceive
that
anEMR
will
elim
inate
paperw
orkandwill
redu
cetherisk
oferrors.A
majorities
ofph
ysicians
andnu
rses
repo
rtthat
the
EMRwill:no
tim
prov
ethequ
ality
ofmedical
care
received
bythepatients,no
tmakepatient
care
less
expensive,
notdecrease
patient
waitin
gtim
e,no
tdecrease
thenu
mberof
ED
visits,no
tdecrease
the
crow
ding
intheED.
Rainu
Kaushal
etal.[200
4][23]
All
domains
Survey
Before
implem
entatio
n–Interview
with
expert
panel(10peop
le)
–Econo
mic
data
obtained
Toestim
atethecostsof
anNHIN
(National
health
inform
ationnetwork)
The
find
ings
show
that
ITisan
impo
rtanttool
used
toim
prov
ethesafety
andefficiency
oftheU.S.
healthcare
system
,bu
tits
adaptatio
nwas
J Med Syst (2007) 31:397–432 407
from
publishedcost
estim
ates
considered
toremainlim
itedlargelybecauseof
alack
ofalignedfinancialincentives
andnatio
nal
standards.The
estim
atefortheim
plem
entatio
nof
theNationalHealth
Inform
ationNetworkwas
regarded
tobe
expensive.
Itwas
estim
ated
that
NHIN
will
costabou
t$15
6billion
capitalcostsand
approx
imately2yearstim
eisrequ
ired
forreal
grow
thin
theU.S
healthcare
costs.Itwas
recogn
ized
that
benefitsfrom
investmentsgained
substantially
both
interm
sof
mon
eyandqu
ality.
The
dataob
tained
from
thestud
ysugg
estedthatIT
coulddecrease
costsof
healthcare.Atthesame
time,
ITcouldim
prov
equ
ality
ofcare
andwork
performance.The
stud
yalso
show
edthat
byinvestingin
differenttypesof
clinical
inform
ation
system
sandtheprov
idersof
healthcare,itwill
likelyto
maxim
izetheinvestment.
Claud
iaPagliariet
al.
[200
4][20]
Primary
hospital
Survey
After
implem
entatio
nInterview
with
stakeholders,
questio
nnaire
toproject
team
sandclinical
users
Toassess
thedevelopm
entof
theScottish
Nationalelectron
icclinical
commun
ications
system
afterim
plem
entatio
n
The
stud
yindicatesdifficultieswhile
attemptingto
implem
entatechnicalandbehaviou
ralchange
interventio
n.Moregeneralim
prov
ementsin
human
andtechno
logy
capabilitywerefoun
dto
befacilitated
bytheprog
ram.The
prog
ram
was
perceivedto
have
facilitated
aspectslik
ehu
man
infrastructure
andstim
ulated
jointworking
amon
gstakeholdergrou
ps.The
stud
yalso
show
sthat
the
implem
entatio
nof
thesystem
was
themost
successful,andelectron
icou
tpatient
book
ingthe
leastsuccessful
during
thestud
y.Analia
Baum
etal.[200
4][21]
Hospital
Survey
After
implem
entatio
nQuestionn
aire
to15
0administrativestaff,
doctors,andnu
rses
Toevaluate
theim
pact
oftheim
plem
entatio
nof
anew
mod
elof
helpdesk
andtechnicalsupp
ort
inHIS
usersat
theho
spital
The
stud
yshow
sapo
sitiv
eattitud
eof
theusers
towards
theim
plem
entatio
nof
HIS.A
greater
frequencyof
positiv
erespon
sesby
thedo
ctorsthan
anyothergrou
pwas
observed.The
administrative
staffandnu
rses
weresatisfied
concerning
the
accessibility
throug
htheinternet.Doctors
show
edmoresatisfactionhigh
erthan
thenu
rses.R
egarding
theanalysisof
theun
derstand
ingof
thehelpdesk,
allthreegrou
psof
usersshow
edtheirsatisfaction,
althou
ghtherespon
sefrom
thenu
rses
was
foun
dto
beless
satisfactory.
Asaresultof
thestud
yof
therespon
setim
egivento
theusers’
requ
ests,the
threegrou
psagainaffirm
edthesuccessof
the
Tab
le2
(con
tinued)
Autho
rsDom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
ies
Finding
408 J Med Syst (2007) 31:397–432
system
.There
was
with
agreaterrespon
sefrom
the
doctorswho
foun
dHIS
tobe
atool
with
fast
accessibility
andavailability24
hours.The
useof
software,
tokn
owtherequ
irem
entandto
measure
therespon
setim
e,contribu
tedto
improv
ingthe
system
sim
ageon
thepartof
theusers.Itwas
foun
dthat
respon
setim
eaffectsbo
thnu
rses
and
doctorssatisfactionandisacond
ition
that
isto
beim
prov
ed.
Laura
Zurita
etal.[200
4][29]
Hospital
Survey
After
implem
entatio
n–Interview
with
12patients
–Directob
servationof
daily
routines
Toprov
ideinform
ationabou
tpatient
values
and
commun
icationthatwill
beuseful
inthedesign
ofamorepatient
friend
lyhealth
record
system
.
The
results
ofthisresearch
underlie
thegeneral
societal
tend
encies
oftheusersacceptance
ofthe
useof
new
techno
logy.How
ever,itwas
also
observed
that
theuserswerevery
enthusiastic
onprotectin
gtheirprivacy.
The
useof
EHR
isperceivedto
protecterrors,prov
idemoreeffective
health
system
sandgive
betterserviceas
wellas
allow
thehealth
prov
ider
tohave
easier
access
totheinform
ation.
The
ease
ofaccessibility
ofinform
ationwas
also
expected
andthestud
yshow
sthat
theexpectations
weremet,as
94%
gained
access
tothesystem
throug
htheintranet
andthe
rest(6%)accessed
itviateleph
one.
The
intranet
(helpd
esk)
was
foun
dto
beuseful
forthemedical
grou
ps.Allusersalso
show
edsatisfactionabou
ttherespon
setim
e.Users’attitud
ewas
foun
dto
bepo
sitiv
etowards
thechange.
Emans
Evang
elJoel
Rajaet
al.
[200
4][24]
Hospital
Survey
After
implem
entatio
n–S
tructuredQuestionn
aire
distribu
tedto
120nu
rses
ToAssessCom
puterkn
owledg
e,attitud
estoward
CPR,andrelatedskillsam
ongnu
rses
The
find
ings
show
anov
erallpo
sitiv
eattitud
etowards
compu
terutilizatio
nby
thepatients.The
results
ofthisstud
yshow
that
themajority
ofthe
nurses
(75%
)hadgo
odcompu
terkn
owledg
eand
20.8%
ofthem
hadgo
odcompu
terkn
owledg
ein
aclinicalcare
setting
.Halfof
thenu
rses
hadaverage
compu
terskills.The
stud
y,ho
wever,show
sno
relatio
nshipbetweencompu
terkn
owledg
eand
compu
terattitud
eandskillsam
ongnu
rses.
Com
putertraining
typesdidno
tseem
toaffect
nurses’compu
terkn
owledg
e.Attitude
towards
compu
terusewas
foun
dto
beindepend
entof
sex,
age,
nurse’sexperiences,profession
alqu
alifications,andmon
thly
family
incomeon
compu
terusage.
Jonathan
C.
Hob
sonet
al.
[200
5][22]
Hospital
Coh
ort
Beforeandafter
implem
entatio
nDataextractedfrom
medical
hand
writtenno
tes
andcompu
terdata
Toevaluate
whether
orno
ttheuseof
acompu
terizedMicrosoftWordtemplatewou
ldincrease
theaccuracy
ofclinical
notes
The
find
ings
ofthisstud
yreveal
that
compu
terized
notesweremoreaccurate
andeasier
topu
tin.The
results
ofthestud
yshow
that
before
implem
entin
g
J Med Syst (2007) 31:397–432 409
anew
system
,theaccuracy
ofhand
writtenno
tes
didno
tmeetthe
predefined
standard
set.The
initial
stud
yfoun
dthat
impo
rtantdemog
raph
icdata
were
missing
from
ordinary
hand
writtenno
tes.After
implem
entin
gtheno
teswerefoun
dto
beaccurate
inmorethan
90%
ofthecases.The
form
sused
toenterdata
wereadop
tedeasily.Calendardata
entrieswereinserted
automatically
toenable
easier
andmoreaccurate
details.Itwas
concludedthat
compu
terizedno
tesshou
ldbe
introd
uced
inclinics
dealingwith
urgent
referrals.
GeorgeW.
Odh
iambo
-Otieno
etal.
[200
5][58]
Primary
care
Survey
After
implem
entatio
n–Interview
with
system
design
er’sop
erators,users
–Day-to-dayob
servationof
theop
erations
ofthe
system
–Review
ofrelevant
literature
Toevaluate
theextent
ofwhich
existin
ginform
ationsystem
shave
supp
ortedthe
operationalmanagem
entof
health
services
atthedistrict
levelin
Kenya.
The
authorsconclude
that
DistrictHealth
Managem
entInform
ationSystem
(DHMIS)were
notsup
portiveof
theDistrictH
ealth
System
(DHS)
Manager’sstrategy
andop
erationalmanagem
ent
functio
n.There
was
amarkeddifference
inthe
focusof
healthcare
workers
andDHMIS
staff.
The
health
workers
werefoun
dto
complainabou
tthequ
ality
ofinform
ationprod
uced.Ontheother
hand
,theDHMIS
staffs
wererepo
rted
tocomplain
abou
ttheinadequate
supp
liesof
basicresources
necessaryforeffectiveop
erationof
thesystem
.A
lack
ofintegrationwas
noticed
intheinform
ation
system
stud
ied;
they
weredisjointed
with
noeffectivecentralcontrolor
coordinatio
nto
ensure
that
theinform
ationcouldbe
madeavailable
wheneverandwherever.The
variationin
collected
data
also
madeitim
possible
toallow
comparison
interm
sof
performance
amon
gDHSs.The
results
also
revealed
that
thereweresomeinadequacies
ofperson
nel,equipm
ent,working
space,
storage
space,
trainedstaff,andmanagem
entsupp
ort.The
inform
ationprod
uced,as
aresult,
was
less
accurate,un
timely,
lackingconfidentiality,
and
incomplete.
Clayton
P.D
etal.[200
5][25]
Primary
care
Survey
Beforeandafter
implem
entatio
nDataob
tained
from
patient
medical
records
Todiscussthebenefitsof
usingEHRand
measure
physicianprod
uctiv
itybefore
andafter
implem
entatio
nof
theEHR.
The
find
ings
ofthisstud
yshow
that
currently,of
472em
ploy
edph
ysicians,32
1(68%
)routinely
entersomedata
directly
into
theEHRwith
out
coercion
.Twenty-fivepercent(80/32
1)of
the
physicians
usevo
icerecogn
ition
forsomedata
Tab
le2
(con
tinued)
Autho
rsDom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
ies
Finding
410 J Med Syst (2007) 31:397–432
entry.Tw
elve
of95
ambulatory
clinicshave
voluntarily
adoptedmeasuresto
elim
inatepaper
charts.O
fthe212physicians
who
entereddatain
2004,sixty-ninephysicians
(22%
)increasedtheir
levelo
fdataentry,while12
(6%)decreased.In
this
studyitwas
foundthattheaverageproductiv
ityincreasedover
theduratio
nof
theevaluatio
nperiod.
Mosto
fthesurvey
participantsreported
that
inform
ationretrievaltim
ewas
betterwith
theEHR
pilotsystem
than
theirprevious
system
andalso
nearlyallofthe
providersbelievedthatthey
provided
bettercare
fortheirpatientsby
usingtheEHRand
also
respondentsdidnotw
anttoreturn
totheir
previous
system
.Staffproductiv
ityim
proved
significantly.R
eportedbenefitsof
EHR
implem
entatio
nincluded
improved
intraoffice
communication,decreasedtranscriptions
costs,
perceivedim
proved
quality
ofpatient
care,and
improved
record
accessibility.
LisaPizziferri
etal.[200
5][65]
Primary
care
Tim
eseries
Beforeandafter
implem
entatio
n–Q
uestionn
aire
completed
by23
physicians
–Tim
emotionObservatio
nof
physicianactiv
ity
–Effectof
theEHR
ontheph
ysician’sov
erall
timein
clinical
sessions
–Evaluationof
physician’sperceptio
nsregarding
theEHR
The
results
ofthisstud
yshow
that
theov
eralltim
espentperpatient
during
clinical
sessions
decreased
with
thenew
system
.A
majority
ofph
ysicians
believedEHR
useresults
inqu
ality
improv
ement.
The
results
werecatego
rizedinto
twotypes:direct
patient
care
andindirectpatient
care.T
here
was
nosign
ificantchange
indirect
patient
care,for
exam
ple,
thetim
espentin
activ
ities
such
asthe
timespentin
exam
iningthepatient.The
physician
believedthat
theinpu
tof
thenew
system
oncommun
ication,
access,efficiency,workload,
and
quality
ofcare
indicatedthat
thenew
system
resultedin
anim
prov
ementin
manydo
mains.The
only
item
show
ingalower
ratin
gthan
averagewas
thenew
system
’sim
pact
onworkload.
Mandi
Why
teet
al.[200
5][26]
Hospital
Survey
After
implem
entatio
nDataob
tained
from
patient
medical
records
Toassess
thequ
ality
ofrecord
keepingby
nurses
inapaediatric
ward
Itwas
foun
dfrom
thestud
ythat
thecompu
terized
recordsarebetterforrecord
keepingin
thecontext
ofchild
protectio
n,as
alargeam
ount
ofinform
ationisstored
wheneverthenu
rses
logon
.How
ever,no
neof
therecordsthat
wereaudited
met
thepredetermined
standard.The
auditalso
indicatedthatbo
ththetwotypesof
record
keeping
intwowards
[HighDependencyUnit(H
DU)and
PaediatricIntensiveCareUnit(PICU)]hadtheir
faults.T
rainingandeducationissues
werefoun
dto
J Med Syst (2007) 31:397–432 411
bethebasicreason
behind
thesubstand
ardresults
achieved
bytheuseof
thetwosystem
s.Itwas
concludedthat
nursingtraining
prog
ramsmust
compriseof
such
ITtraining
too,
andtheresults
revealed
that
inthePICU,40
%of
thepatientsdid
notaccess
totheirprevious
data.
Tim
Scott
etal.[200
5][59]
Primary
care
and
hospital
Coh
ort
Beforeandafter
implem
entatio
nSem
istructureinterviews
with
senior
clinicians,
managers,projectteam
mem
bers
Toexam
ineusers’
attitud
estowards
implem
entatio
nof
anEMR
System
The
find
ings
ofthisstud
yshow
thattheadop
tionof
theelectron
icmedicalrecord
system
was
flaw
edanddetached
from
thelocalenv
iron
mentshow
ing
disapp
rovalof
theim
plem
entatio
n.Fou
rmajor
processeswereidentifiedin
theim
plem
entatio
nof
EMR:selectio
n,design
andearlytesting,adaptatio
nforwidespreaduse,andadaptatio
nof
the
organizatio
nto
thenewelectron
icmedicalrecords.
Regarding
theselectionof
theelectron
icmedical
record
system
,onlyon
erespon
dent
expressed
approv
alof
thesystem
choice,the
restfoun
dit
prob
lematicandrepo
rted
dissatisfactionwith
the
choice.T
heresistance
observed
towards
thedesign
andtestingwerethou
ghttobe
dueto
inadequate
earlytesting,
substantialsoftwareprob
lems,and
inadequateIT
know
ledg
e.Difficulties
werealso
noticed
regardingtheadop
tionof
theelectron
icmedicalrecordsdu
eto
differencesov
erprioritiesof
theusersanddu
eto
softwareprob
lemsitself.The
results
also
show
edthatthesystem
redu
cedclinical
prod
uctiv
itydu
etoextraworkinvo
lvingprocessing
labo
ratory
resultrepo
rts,entry’sordersand
navigatin
gthroug
hthesystem
.Maria
Bryson
etal.[200
5][66]
All domain
Survey
After
implem
entatio
nQuestionn
aire
completed
by20
20nu
rses,midwives
andhealth
visitors
workers
(Onlinesurvey)
Toinvestigatethenu
rses
perceptio
n,kn
owledg
eandexpectations
oftheNationalHealth
system
’sIT
prog
ram
Poo
raccess
andinadequate
know
ledg
eof
theplans
intheNationalH
ealth
Service
(NHS)was
observed
asaresultof
thisstud
y.The
survey
sugg
estedthat
thenu
rsingstaffs
wereno
tfully
awareof
the
currentIT
plans.How
ever,51
%of
thenu
rses
affirm
edthesign
ificantim
prov
ementbrou
ghtby
theintegrated
electron
ichealthcare
record
system
inclinical
care.Nohigh
levelof
enthusiasm
aspredictedwas
observed.Lackof
confidence
inusingadvanced
inform
ationtechno
logy
compo
undedwith
thelack
oftraining
andno
Tab
le2
(con
tinued)
Autho
rsDom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
ies
Finding
412 J Med Syst (2007) 31:397–432
invo
lvem
entof
thenu
rses
inthechange
process
werefoun
dto
bethereason
sbehind
thenegativ
erespon
ses.For
future
developm
ents,itwas
expected
that
earlyandregu
larinvo
lvem
entof
potentialuserswou
ldbe
madepo
ssible.Itwas
demon
stratedthatthenu
rses
wou
ldbe
ableto
share
decision
makingwith
clientsusingcredible,tim
ely,
andaccurate
inform
ation.
Wilcox
Aet
al[200
5][27]
Primary
care
Retrospectiv
ecoho
rtstud
y,Survey
Beforeandafter
implem
entatio
n–Interview
with
7ph
ysicians
–Trackingof
medicalrecord
Toanalyzetheusageof
summaryworksheets,
andtheirim
pact
onph
ysicianperformance.
Thisstud
yshow
sthat
four
physicians
forevery
patient
seen
usetheworksheet,andthreeof
them
stated
they
likeit.
Twoidentifiedho
wthey
useitto
trackeither
vitalsigns
orlabo
ratory
results
fortheir
patients.One
indicatedhe
only
used
itsometim
es,
butalso
stated
hisintentionto
useitmore.
One
physiciansaid
hereview
severyfield,
while
other
physicians
useiteither
forfollo
wingdisease-
specific
labvalues
orforreview
ingmedications
andtrending
bloo
dpressure
andweigh
t.One
physicianwho
indicatedlim
itedusesaid,“Hedo
esreview
thereminders
specifically”.
Alsothe
find
ingof
thisstud
yindicatedthat
usagehas
grow
nfrom
afew
hund
redto
over
25,000
unique
patientspermon
thdu
ring
a2-year
period
.The
find
ings
show
salso
diabetic
patientsforwho
mthe
worksheet
isaccessed
aresign
ificantly
morelik
ely
tobe
incompliancewith
accepted
testingregimens
forglycosolated
hemog
lobin.
KMäkelä
etal.[200
5][28]
Primary
care
Survey
After
implem
entatio
nQuestionn
aire
tohealthcare
profession
alin
15healthcare
center
Toevaluate
how
wellEPRsystem
smeetthe
needsof
physicians
andotherhealthcare
profession
alsandto
determ
ineifthereareany
sign
ificantdifferencesbetweenEPRsystem
s.
The
find
ings
show
that
despite
awideuseof
EPR
system
sin
healthcare
centres,thoseelectron
icdatabasescouldno
tyetcompetein
allaspectswith
conv
entio
nalpaper-basedrecords.The
fact
isa
reflectio
nof
thedifficulty
ofstructuringclinical
data
tomakethem
easily
accessible
while
still
retainingasimpleanduser-friendlymetho
dto
inpu
ttheinform
ationfor73
%of
thecases,allof
theinform
ationrequ
estedwas
madeavailablein
theEPR.Nosign
ificantcorrelationwas
observed
betweentheam
ount
ofpatientsforwho
mall
inform
ationwas
availablein
theEPRandthe
duratio
nforwhich
theEPRsystem
shadbeen
inuse.
Asign
ificantdifference
betweenthetype
ofEPRsystem
andthepercentage
ofpatientsfor
who
malltheinform
ationwas
availablethroug
htheEPRwas
demon
stratedthroug
hout
thestud
y.
J Med Syst (2007) 31:397–432 413
Tab
le3
Telem
edicineevaluatio
nstud
ies
Autho
rsDom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
yFinding
B.Larcheri
etal.
[200
3][35]
Hospital
Survey
Beforeand
after
implem
entatio
n
Questionn
aire
completed
by59
physicians
and63
nurses
(open
andclosed
respon
sequ
estio
n)
Toexploreuser
satisfactionwith
thenewsystem
after6mon
thsof
experimentatio
n
The
find
ings
ofthisstud
yshow
that
therewas
anov
erallpo
sitiv
eattitud
etowards
thesystem
.A
sign
ificantdifference
was
noticed
betweenthe
user
opinionbefore
andaftertheuseof
the
system
,thus
wemight
conclude
that
theuser
satisfactionwas
generally
high
.A
qualitativ
eanalysisshow
edthat
86%
ofph
ysicianusersfelt
that
theteleconsultatio
nwas
useful
asitoffered
direct
commun
icationwith
thecolleagues.The
averageof
consultatio
ntim
efortheequivalent
outpatient
clinicin
thesameho
spitalfor
thesame
consultant
was
15minutes.In
additio
n,having
diagno
stic
inform
ationallowspatientsto
bescreened
andup
to50
%canbe
elim
inated
asno
trequ
estin
gtheho
spitalinvestigation.
Itwas
expected
that
atechno
logy
-based
approach
could
redu
cethenu
mberof
outpatient
clinicsby
75%,
redu
cetheconsultatio
ntim
eandby
makingthe
fullinform
ationavailable,
ensure
that
each
consultatio
niseffective.
Tho
mas
L.
etal.
[200
3][39]
Hom
eand
prim
ary
care
Con
trol
trial
andsurvey
After
implem
entatio
n–Q
uestionn
aire
completed
by14
7stud
ents(con
sultant
survey),94
scho
olnu
rseevaluatio
n,77
stud
ent
evaluatio
ns,51
parentsevaluatio
n–E
cono
mic
recordsexistencein
the
centers
Toevaluate
thequ
ality
andcost
effectivenessof
health
care
prov
ided
inurbanandrural
elem
entary
scho
ol-based
telehealth
centers
The
findings
ofthisstudyshow
thatprovider,nurse,
child,and
parent
satisfactionallw
erehigh.T
heproviders’andnurses’decision
confidence
scores
ranged
from
alowof
4toahigh
of4.8on
a5-point
scale.The
high
meanscores
onthedecision
confidence
scales
indicatedthatthetelehealth
protocolsspeededthetim
ebetweendiagnosisand
starto
ftreatm
entand
helped
with
thenurses’
decision
aboutthe
disposition
whenthechild’s
problem
was
outsideof
astandard
protocol.T
hestudentswerealso
positiveaboutthe
interactionwith
thephysicianor
thenursepractitioneraverage
family
savingsperencounterwere3.4hoursofwork
time($43)and$177
inem
ergencydepartmento
r$54in
physiciancosts.Includingtravel,savings
for
families
ranged
from
$101
to$224
perencounter.
Lynne
Bladw
inet
al.[2003]
[40]
Primary
care
Coh
ort
Beforeandafter
implem
entatio
n–T
imemeasurement
–Dataob
tained
from
hospital
records
Tomeasure
theeffect
oftelemedicineon
consultatio
ntim
e
The
results
ofthisstud
yshow
that
with
theuseof
telemedicine;
thenu
mberof
registered
outpatientsin
clinicsdecreased.
The
duratio
nof
theconsultatio
ntim
ewas
redu
cedfrom
23to
10
414 J Med Syst (2007) 31:397–432
minutes
becausediagno
stic
inform
ationwas
beingsent
inadvanceandin
theconsultatio
ntim
e,theusersdiscussthemanagem
entof
the
disease.
Inotherwords
consultatio
nchanged
from
diagno
sisto
managem
ent.With
diagno
stic
inform
ation,
patientscanbe
screened
andabou
thalfof
them
canbe
considered
tobe
with
outany
need
ofho
spitalinvestigation.
Moreover,
accordingto
theresults
ofthisstud
ytelemedicine
couldredu
cethenu
mberof
outpatient
clinicsby
75%
aswellas
redu
cethedu
ratio
nof
the
consultatio
n.Finkelstein
Jetal[200
3][41]
Hospital
and
Hom
ecare
Survey
After
implem
entatio
n–Q
uestionn
aire
completed
by41
patients
–Interview
with
41patients
Toassess
user
acceptance
ofthe
homeautomated
telemanagem
ent(H
AT)system
The
find
ings
show
that
inthepretestin
gph
aseat
hospital,patientsdemon
stratedahigh
levelof
acceptance
ofuser
interface.
90%
ofthepatients
stated
nodifficultiesin
usingthesystem
and93
%of
thepatientsclaimed
they
wou
ldusethesystem
atho
meandadvise
others
thesystem
.The
patientsnamed
saving
timeby
redu
cing
number
ofvisitsandgetting
results
ontim
e,andthe
possibility
ofhaving
morecomprehensive
diseasemanagem
entas
high
utility
oftheHAT
system
.Atthepilottestingaftereigh
tweeks
ofusingHATat
homeshow
edstatistically
sign
ificantim
prov
ementin
disease-specific
quality
oflifedimension
sof
generalsatisfaction,
selfefficacy,distress,andho
mehassles.
And
also
thestud
yshow
edsign
ificant
improv
ementin
patient
satisfactionwith
treatm
entprocessandhigh
acceptance
ofthe
HATsystem
.Michael
J.Breslow
etal.
[200
4][30]
Hospital
Coh
ort
Beforeandafter
implem
entatio
nDatacollected
from
patients
medical
records
Tomeasure
theeffectsof
telemedicineon
clinical
and
econ
omic
outcom
esin
amulti
site
intensivecare
unit
The
results
ofthisstud
ydemon
strate
improv
edclinical
outcom
esandfinancialperformance.
Patient
mortalityratesforbo
ththeICU
stay
and
hospitalizationwerelower
(9.4%
vs.12
.9%)
whenremoteintensivemanagem
ent
supp
lementedroutinecare.The
averageleng
thof
stay
(LOS)in
ICU
during
thesupp
lemental
care
period
was
lower
(3.63days
vs.4.35
days).
There
was
nochange
intheho
spital’sLOScosts
werefoun
dto
beredu
cedtoo,
asaresultof
both
aredu
ctionin
LOSandadecrease
inthe
daily
costsof
ICU
care.In
additio
n,the
redu
ctionin
ICU’sLOScreatednew
capacity
J Med Syst (2007) 31:397–432 415
that
enabledadditio
nalpatientsto
becaredforin
thetwoICUs.
Marily
nne
Aet
al.
[200
4][67]
Hom
ecare
Survey
After
implem
entatio
nFocus
grou
pswith
9clientsand13
nurses
Interviewswith
9ph
ysicians
and9managers
Toassess
thereadinessof
stakeholders
toadop
tteleho
mecareservices
foradult
diabetic
The
find
ings
ofthisstud
yshow
differencesin
stakeholders’conceptio
nsof
techno
logy,
includ
ingcommon
them
esam
ongclients,
prov
iders,andorganizatio
ns.Nurses,ph
ysicians,
andmanagersshared
twosubjects:on
erelatedto
thebenefitsof
increasedsystem
access,
particularly
forruralclientsandthosewith
mob
ility
limitatio
nsandalso
with
respectto
increasedacuity
andnu
mbers
ofho
mecare
clientsseekingservices,n
ursesandclientsshared
practical
concerns
abou
tthemechanics
oftelemedicineinclud
ingsystem
failu
reandease
ofuse.
Nursesfocusedon
diseasemanagem
entand
supp
ort.Managersandph
ysicians
focusedon
overallethics,system
structured,system
evaluatio
n,andsustainability.
Kaufm
anD.
R.et
al.
[200
4][68]
Hom
ecare
Cog
nitiv
ewalkthrou
gh(usability
testing)
After
implem
entatio
nFiledusability
testingin
25subjects’ho
me:
a4subjectsin
New
YorkCity
and11
inUpstate
New
York
Toidentifybarriers
toeffective
homeTelem
edicineuse
(IDEATel)
The
find
ings
ofthisstud
yshow
that
thereisa
rang
eof
both
cogn
itive
andno
ncog
nitiv
ebarriers
that
wereun
anticipated
bythecogn
itive
walkthrou
ghanalysis.The
evaluatio
ninclud
eda
cogn
itive
walkthrou
ghanalysisto
characterize
task
complexity
andidentifypo
tentialprob
lems
aswellas
fieldusability
testingin
patients’
homes.The
stud
yrevealed
dimension
sof
the
interfacethat
impededop
timal
access
tosystem
resources.Itwas
foun
dsign
ificantob
stacles
correspo
ndingto
perceptual-m
otoric
skills,
mentalmod
elsof
thesystem
,andhealth
literacy.
Won
gY.K
etal[200
5][44]
Hom
ecare
Clin
ical
trials
After
implem
entatio
n–Q
uestionn
aire
completed
by22
patients.–S
econ
dary
outcom
emeasuresby
medical
equipm
entto
assess
patient
outcom
es
Toexplorethefeasibility
and
efficacy
ofan
exercise
prog
ram
forelderlypeop
lewith
knee
pain
cond
uctedviavideocon
ferencing
The
find
ings
ofthestud
yshow
that
therewere
sign
ificantim
prov
ementsin
theprim
aryou
tcom
emeasures.Significant
improv
ementswere
recorded
inallsecond
aryou
tcom
esexcept
rang
eof
motionof
thekn
ee.Anincrease
ofstreng
thof
50%
and46
%(Po0
.001
)was
foun
din
therigh
tandleftqu
adriceps,respectiv
ely.
A21
%redu
ctionin
thetim
etakento
completethe
TUGT(tim
edup
-and
-gotest)was
observed.
There
was
also
sign
ificantim
prov
ementin
the
Tab
le3
(con
tinued)
Autho
rsDom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
yFinding
416 J Med Syst (2007) 31:397–432
know
ledg
etest.Over80
%of
thesubjects
answ
ered
allthequ
estio
nscorrectly,with
anaverageim
prov
ementof
threemarks
inthefinal
evaluatio
n.Over80
%of
theelderlysubjectswho
joined
theprog
ram
agreed
orstrong
lyagreed
abou
tallaspectsof
usingvideocon
ferencing.
Mostof
them
feltthat
thesystem
was
user-
friend
lyandconv
enient.
BratanT
etal[200
5][36]
Hom
ecare
Clin
ical
trials
After
implem
entatio
nInterview
with
twoph
ysicians,4
managers,1career
and1nu
rse
Todeterm
inethefeasibility
and
acceptability
ofho
memon
itoring
The
find
ings
ofthestud
yshow
that
data
transm
ission
was
foun
dto
beun
reliableon
occasion
sandwas
improv
edby
extend
ingthe
wirelessnetworkin
theho
mes.Dataaccess
and
presentatio
nwereconsidered
acceptable,
althou
ghsugg
estio
nsforminor
changeswere
made.
Nodiscernibleim
pact
ondisease
managem
ent,diagno
sisor
care
was
observed
asa
resultof
themon
itoring
.Intervieweesno
ted
severalpo
tentialbenefits.Som
ekeyadvantages
from
interviewees’
pointof
view
that
were
mentio
nedwere:
facilitates
bettercommun
ication
with
thegeneralpractitioner;enablesearly
detectionof
deteriorations;canprov
idea
continuo
uspictureof
patients’
health;the
equipm
entwas
easy
touse.
The
mon
itoring
was
foun
dto
begenerally
acceptable
andfeasibility
was
considered
tohave
been
largelyprov
en.
MairF.S
etal[200
5][37]
Hom
ecare
Rando
mized
controlled
trial
After
implem
entatio
nQuestionn
aire
completed
by22
patientsand14
nurses
Toexplorepatient
andprov
ider
perspectives
onho
metelecare
The
results
ofthisstud
yshow
that
thereare
sign
ificantdifferencesin
perceptio
nbetween
patientsandprov
iderswith
regard
totelecare
encoun
ters
across
allthedo
mains
addressed.
Patientsdemon
stratedmorepo
sitiv
eview
sof
the
telecare
encoun
ters
than
prov
iders.The
prov
iders
weremoreconcernedabou
tthenegativ
eeffects
oftelecare
consultatio
nson
commun
ication.
With
regard
totheaccuracy
ofassessmentof
apatient’smedical
prob
lems,thepatientsbelieved
thatnu
rses
couldun
derstand
theirprob
lemsqu
itewellwhenusingthetelecare
system
,while
incontrastthenu
rses
werefarless
confidentabou
tthisaspect
ofthetelecare
encoun
ters.The
prelim
inarydata
obtained
inthepresentstud
yidentifyamarkeddifference
ofop
inionbetween
patientsandserviceprov
idersabou
ttheutility
andacceptability
oftheho
metelecare
consultatio
ns.
J Med Syst (2007) 31:397–432 417
SScalvini
etal.
[200
5][31]
Hom
ecare
Coh
ort
After
implem
entatio
nDatacollected
from
patient
medical
records
Toinvestigatetheuseand
econ
omic
aspectsof
home-based
telecardiology
inchronicheart
failu
repatients
The
find
ings
ofthestud
yshow
that
therewere
sign
ificantredu
ctionin
hospitalizationand
instability
intheho
me-basedtelecardiology
(HBT)grou
prelativ
eto
theusualcare
grou
p.There
was
aredu
ctionof
24%
inthetotalcosts
afteron
eyear
inthegrou
ps,which
underw
ent
telecardiology.Thiscostredu
ctionwas
presented
afteron
eyear.There
was
anincrease
inthe
quality
oflifein
HBTcomparedto
theusual-care
grou
ps(the
meanfortheMinnesota
living
Questionn
aire
scores
werefoun
dto
be29
and
23.5respectiv
ely).
Th
Von
tetsiano
set
al.
[200
5][42]
Hom
ecare
Quasi
experimental
Beforeandafter
implem
entatio
nThe
econ
omic
data
obtained
from
hospitalrecords
–Questionn
aire
topatients
–Toevaluate
theclinical
usefulness
ofatelemedicine
system
–Toestim
ateecon
omic
benefitsof
interventio
n
The
find
ings
show
asign
ificantim
prov
ement
regardingbo
thusefulness
andecon
omicbenefits.
There
was
adecrease
inho
spitalization,
which
declined
from
37to
only
6as
aconsequenceof
usethesystem
.A
similardecrease
inem
ergency
departmentvisitswas
also
observed;visits
drop
pedfrom
156to
86afterusingthesystem
.The
patient
know
ledg
eabou
tthediseaseand
self-m
anagem
entwerealso
observed
tobe
improv
ed.There
was
asign
ificantim
prov
ement
inpatient
quality
oflifeof
morethan
28%,while
thesatisfactionrate
was
32%
better.Empo
wered
patientsandcaregivers
wereable
tominim
ize
symptom
s,complications,anddisabilitiesandto
increase
autono
myin
everydayliv
ing.
The
total
costswerelower
during
thestud
ythan
before.
The
redu
cedcostsin
thepresentstud
ywere
mainlydu
eto
theredu
ctionin
hospitaladmission
andthelower
leng
thof
stay
ofpatients.
SianM
Nob
leet
al.
[200
5][32]
Hospital
Rando
mized
controlled
trial
Beforeandafter
implem
entatio
n–T
imemeasurement
–Interview
(exp
ertpanel)
–The
econ
omic
data
obtained
from
hospitalsrecords
Cost-consequences
analysisof
minor
injury
telemedicinein
aperiph
eral
emergency
department
The
stud
yshow
sthat
telemedicinehastheability
toredu
cecostssubstantially
only
where
transport
costsin
theabsenceof
telemedicineisvery
high
.The
find
ings
also
revealed
that
unlik
ethe
previous
stud
ies,itwas
foun
dto
have
been
more
costly
from
both
NHSandpatient
view
points.
The
mainreason
sbehind
itweretheneed
for
specialistequipm
ent,high
erstaffcosts,and
high
erfollo
w-upcosts.The
timetakenby
the
Tab
le3
(con
tinued)
Autho
rsDom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
yFinding
418 J Med Syst (2007) 31:397–432
twoconsultatio
nswas
considerably
long
erthan
forgeneralph
ysicianconsultatio
n.Itwas
concludedthat
theprocedural
changesmay
berequ
ired
iftelemedicinewas
tobe
putinto
practice(e.g.theneed
totriage
telemedicine
patientsatthemainho
spital).Itw
ould
affectbo
thnu
rses
andgeneralph
ysicians,bu
titistooearly
topredicttheeffects.
AGary
Linassi
etal.
[200
5][38]
Hospital
Survey
After
implem
entatio
nQuestionn
aire
(open/closeended
questio
ns)to
15patients
Toassess
user
(patient)
satisfactionwith
telemedicine
From
theresultof
thestud
y,itwas
observed
that
telemedicineprov
ides
sign
ificantadv
antagesov
ermoretradition
almetho
dsof
ampu
teeevaluatio
nandcare.Overall,
thepatientssurveyed
were
satisfied
with
theorganizatio
nalstructure.
Telem
edicineseem
edto
representacost-
effective,
practical
solutio
n.Itisacceptable
topatientsandprov
ides
areliableassessmentof
the
ampu
tee.
App
roximately97
%of
therespon
ses
werein
thego
odto
excellent
rang
eshow
inggreat
user
satisfaction.
How
ever,concerns
wereraised
abou
tease
ofaccess
tolocaltelemedicinesites,
conn
ectio
nwaitin
gtim
esandlack
offamiliarity
with
telemedicinetechno
logy.Minor
difficulties
relatedto
technicalaspectswereob
served.
How
ever,itwas
sugg
estedthat
future
developm
entsmight
includ
emorecomprehensive
evaluatio
nsandtreatm
entservices
inthe
commun
ity.
DDavid
Persaud
etal.
[200
5][33]
Primary
care
Survey
Beforeandafter
implem
entatio
nThe
econ
omic
data
extractedfrom
expenserepo
rt–Q
uestionn
aire
(openended)
to21
5patient
Toexam
inethecostof
Telehealth
from
asocietal
perspective
The
find
ings
ofthisstud
yshow
that
thesocietal
costof
anaverageTelehealth
consultatio
nwas
sign
ificantly
high
erthan
thecostof
face-to-face
consultatio
n,even
thou
ghfrom
theperspectiveof
thepatient
Telehealth
services
werecheaper.
How
ever,itwas
also
observed
that
from
asocietal
perspective,theov
erallcostof
prov
iding
Telehealth
was
high
erthan
prov
idingface-to-face
services.F
orthemajority
oftheFCof
Telehealth
thelargestart-up
capitalcostsaccountwas
noted
tobe
morecostly
perconsultatio
n.EricDillon
etal.
[200
5][34]
Primary
care
Coh
ort
After
implem
entatio
n–D
ataob
tained
from
projects
repo
rtsInterview
Toevaluate
costsof
theTelehealth
projectin
Australia
The
authorsconclude
that
theWestern
Australia
Telehealth
projectachieved
itsob
jectives
that
weresign
ificantclinical
utilizatio
n,educational
andtraining
supp
ort,andim
prov
edinform
ation
access.H
owever,itw
asfoun
dthatno
tallclinical
Telehealth
projectsprov
ided
sign
ificantcost
J Med Syst (2007) 31:397–432 419
saving
sto
thehealth
department.Som
eprog
rams
representedanetcostto
thehealth
department
andthesaving
sofferedby
someotherswereon
lymod
est.How
ever,mostprog
rammes
offered
sign
ificantqu
antitativebenefitsandotherlong
erterm
benefitsthan
costsaving
salon
e.Clin
ical
usegrew
by30
%of
allTelehealth
activ
ity.
Edu
catio
nalusewas
increasedby
40%
and
managem
entusefoun
dto
grow
by30
%.The
stud
yalso
demon
stratedthat
theaverage
overhead
costof
each
Telehealth
eventwas
$AU19
2which
comparedfavo
urably
with
the
averagecostpertrip
of$A
279.
Itwas
also
observed
thatthecoordinatorswerecriticalof
the
implem
entatio
nof
individu
alprojectsandplayed
asign
ificantrole
intheov
erallsuccessof
the
projects.
Brian
Jetal.
[200
5][43]
Hospital
and
Hom
eCare
Retrospectiv
erecord
review
After
implem
entatio
nReview
of49
outpatient
records
Tocompare
specific
treatm
entand
outcom
evariablesbetween
mentalhealth
care
via
videocon
ferencingto
care
prov
ided
inperson
The
find
ings
ofthisstud
yshow
thaton
egrou
pwas
seen
viavideoconferencing
(telem
entalhealth
care
[TMHC])while
asecond
grou
pwas
seen
face-to-face
care
(FTFC)andserved
asacontrol
grou
pshow
edthat
theGlobalAssessm
entof
Fun
ctioning
was
sign
ificantly
moreim
prov
edfor
theTMHCgrou
p.Meanchange
inGlobal
Assessm
ento
fFun
ctioning
forTMHC(15.3)
was
sign
ificantly
morethan
meanchange
forFTFC
(8.4).There
wereno
sign
ificantdifferences
betweenthegrou
psin
thenu
mberof
labo
ratories
orstud
iesordered,
selected
mentalstatus
elem
ents,self-helprecommendatio
nsmade,
ornu
mberof
patientsprescribed
twoor
more
psycho
trop
icmedications.The
rate
offull
compliancewith
themedicationplan
andfollo
w-
upappo
intm
entswas
sign
ificantly
betterfor
TMHC.The
find
ingof
stud
yalso
show
edthat
Providers
usingTMHCtold
morepatientsto
return
forfollo
w-upappo
intm
entsin
30days.
Tab
le3
(con
tinued)
Autho
rsDom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
yFinding
420 J Med Syst (2007) 31:397–432
Tab
le4
DSSevaluatio
nstud
ies
Autho
rsSystem
Dom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
yFinding
Sho
rtD
etal
[200
3][50]
CDSS
Primary
care
Survey
Beforeand
after
implem
entatio
n
Questionn
aire
andinterview
with
15General
practitioner
Todevelopandevaluate
inprim
ary
care
acompu
terizeddecision
supp
ort
system
forthemanagem
entof
stroke
patients
The
find
ings
show
that
practitioners
werevery
positiv
efortheCDSSto
addressun
certaintyam
ongGPs.
Practition
ersstated
severalpo
tential
benefitsof
thesystem
,includ
ing
helpingthem
toim
prov
eGP–patient
dialog
ueandreinforcingtheir
recommendatio
nmessage
topatients,
theapplicability
andacceptability
ofincorporatingthesystem
into
aprim
arycare
consultatio
n.The
system
was
foun
deasier
touseandto
navigate.The
timeto
access
the
system
was
seen
asapo
sitiv
efactor.
Lim
itatio
nsto
applicability
ofthe
system
wereidentifiedsuch
aspractitionersneedingbasicinform
ation
techno
logy
(IT)skillsandbo
thGPs
andpatientsrequ
iringan
understand
ingof
simplerisk
concepts
arenecessary.
Quantitativ
eresults
furthersugg
estedthatthesystem
made
decision
makingeasier,im
prov
edfeelings
ofbeingsupp
orted,
improv
edthequ
ality
ofdecision
makingand
increasedsatisfaction.
Cornelia
Met
al[200
3][51]
CDSS
Hospital
Con
trol
trial
After
implem
entatio
nClin
icaltrialo
n52
patientswho
assign
edto
interventio
nor
controlcond
ition
Questionn
aire
completed
bythosepatients
–Toevaluate
prelim
inaryeffectsof
acompu
terizedsupp
ortsystem
oncong
ruence
betweenpatients’
repo
rted
symptom
s–T
oinvestigatethesystem
’sease
ofuse,
timerequ
irem
ents,andpatient
satisfaction.
The
find
ings
ofthestud
yreveal
that
therewereno
sign
ificantdifferencesin
numbers
ofrepo
rted
symptom
sin
the
interventio
nandcontrolgrou
ps.There
werelargevariations
inpatients’
repo
rtsof
frequency,
severity,degree
ofbo
ther
ofsymptom
s,and
impo
rtance.Alsothestud
yshow
sthat
while
patientsin
both
grou
pswere
equivalent
atbaselin
ein
symptom
characteristics,thereweresign
ificantly
greatercong
ruence
betweenpatients’
repo
rted
symptom
sandthose
addressedby
theirclinicians
inthe
J Med Syst (2007) 31:397–432 421
experimentalgrou
p.The
system
scored
high
onease
ofuse.There
were
nosign
ificantgrou
pdifferencesin
patient
satisfaction.
Beuscart-
zeph
irM.
Cet
al[200
4][45]
CPOE
Hospital
Survey
After
implem
entatio
nSem
istructured
andstructured
interviewsof
physicians
and
nurses
Observatio
non
users’
interactions
with
patient
recordsDocum
entanalysis
Tostud
yof
CPOE’smedication
administrationfunctio
nswith
focus
impact
onph
ysician–
nurse
coop
eration.
The
find
ings
ofthestud
yreveal
that
the
nursegetsthemedicationorderson
her
care
plan
andmedication
administrationrecord.Whentheorder
seem
sun
clearor
difficultto
administer,shemay
asktheph
ysician,
ifhe
isavailable.
On-site
observations
inthepaper-do
cumentedsituation
sugg
estedthat
thedo
ctorsareno
tinterested
anddo
n’tfeel
committed
with
theexactplanning
ofdrug
administration,
which
they
ordinarily
leaveto
thenu
rses.The
comparisonof
coop
erationmod
elsin
both
situation
show
sthat
userstend
toadop
ta
distribu
teddecision
makingparadigm
inthepaper-basedsituation,
while
the
CPOEsystem
supp
ortsacentralized
decision
makingprocess.The
usability
assessmentun
coveredusability
prob
lemsfortheentryof
medication
administrationtim
eschedu
lingby
the
physicianandrevealed
that
the
inform
ationcanbe
ambigu
ousforthe
nurse.
Joan
S.
Ash
etal.
[200
4][69]
CPOE
Hospital
Survey
After
implem
entatio
n–S
emistructuredinterviewwith
housestaff(phy
sicians,
nurses,ph
armacists,
administrators,inform
ation
techno
logy
staff)accompany
with
observation
Toidentifysuccessfactorsconcerning
theim
plem
entatio
nof
CPOE
The
find
ings
ofthestud
yreveal
that
the
direct
entryof
medical
orders
bythehealthcare
decision
maker
andits
otheru
sersincreasedyearby
year
during
the5-yearstudy.Itwasalso
noted
thatCPO
Eisperceivedto
reduce
medicalerrors.T
hestudy
hasproduced
detaileddescriptions
offactorsrelatedto
CPOE’ssuccesssuch
as(1)computertechnology
principles
(which
include:temporalconcerns;
Tab
le4
(con
tinued)
Autho
rsSystem
Dom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
yFinding
422 J Med Syst (2007) 31:397–432
technology
andmeetin
ginform
ation
needs);m
ultid
imensionalintegration;
andcosts.(2)Personalprinciples:v
alue
tousers,training,and
support.(3)
Organizationalp
rinciplessuch
ascollaborativ
eprojectm
anagem
ent;
term
s,concept;im
provem
entthrough
evaluatio
n).(4)
Environmentalissues
provided
themotivationandcontextfor
implem
entin
gsuch
system
.How
ever,
good
training
andsupportw
eredefined
astheprerequisitesforbetterresults.
JosAarts
etal.
[200
4][60]
CPOE
Hospital
Survey
After
implem
entatio
n–U
nstructuredinterview
with
25staffmem
bers
–Observing
thesystem
inuse
Tostud
ytheInterrelationshipof
technicalandsocial
elem
entsand
how
CPOEfitinto
medical
work
practices
The
find
ings
ofthisstud
yweredivided
into
twocatego
ries:i.e.socioand
techno
.The
societal
aspectswere
prom
inentin
theexperience
ofthe
administrativeperson
alrelatedto
the
system
intheho
spitaland
theirattitud
etowards
it.Itwas
feared
that
greater
amou
ntsof
timeandcostwou
ldbe
requ
ired
intheim
plem
entatio
nprocess.Thisstud
yalso
show
sthatthe
implem
entatio
nof
CPOEin
twoDutch
hospitalsprov
edto
bevery
much
prob
lematic.In
oneho
spitalCPOE
neverbecamefunctio
nal,andin
the
other,CPOEwas
only
used
bythe
nursingandadministrativestaff.Toa
largeextent
socio-technicalissues
influenced
theim
plem
entatio
n,such
astheexistenceof
localinfrastructures
andworking
arrang
ements.These
are
noteasily
changedas
aresultof
new
techno
logies.Researchers
sugg
ested
that
introd
ucingnew
techno
logies
that
influencemedical
workpractices
and
organizatio
nalarrang
ementsshou
ldbe
carefully
plannedformeasurementof
organizatio
nalim
pact
tomakeiteasier
andaccurate.
Cho
iS.S
etal
[200
4][57]
CPOE
Hospital
Survey
Beforeandafter
implem
entatio
n–C
ompu
terandpaperbased
documentscomparison
–Interview
with
anu
rse
Toevaluatio
nof
aweb-based
nurse
orderentrysystem
fortuberculosis
patientsin
Peru
Inthisstud
yitwas
foun
dthatthereisno
sign
ificantdifference
inthecontrol
grou
ps’ou
tputs.The
interventio
ngrou
pshow
edasign
ificantdrop
inthe
J Med Syst (2007) 31:397–432 423
errorrate
afterthesametim
einterval.
Alsoin
thisstud
yitwas
foun
dthatthe
nurses
affirm
edthesystem
’susability
andease
touseandsimplicity
and
ability
toview
allof
theinform
ation
regardingapatient’smedication
historyon
onepage.The
web
system
show
sapatient’sentireregimen
historyfrom
theinitial
regimen
tothe
present.Add
ition
ally,theweb
system
allowed
thenu
rseto
seeallof
the
changesat
onetim
e.RobertL
.Ohsfeldt
etal.
[2004]
[54]
CPOE
Hospital
Sim
ulation
Before
implem
entatio
nQuestionn
aire
distribu
tedto
74generalho
spitals
–The
econ
omic
data
obtained
from
expenserepo
rts
–Toestim
atethecostof
implem
entin
gCPOEin
hospitalsin
aruralstate
–Toevaluate
thefinancialim
plications
ofCPOE’snatio
nwide
implem
entatio
n
The
findings
show
aconsistent
pattern
ofgreaterusein
urbanhospitals.In
additio
n,theextent
ofintegrationacross
clinicalfunctio
nswas
foundto
begreaterin
urbanhospitals.T
hefinancial
results
ofthisstudyshow
edthatcostsfor
CPO
Eim
plem
entatio
nwereabout$
1.3
millionforcriticalaccessandrural
hospitals,$2millionforruralreferral,
and$1.9millionforurbanhospitals
underthelowcostscenario.F
orrural
andcriticalaccesshospitals,theseCPOE
implem
entatio
ncostswould
representa
substantialexpansion
overall.CPO
Eim
plem
entatio
nwould
increase
operatingcostsforruraland
critical
access
hospitalsin
theabsenceof
substantialcostsavings
associated
with
improved
efficiency
orim
proved
patient
safety.Itw
asalso
observed
thatthecost
impactisless
dram
aticbutstill
substantialfor
theurbanandrural
referralhospitals.
Benjamin
Bog
ugki
etal.
[200
4][55]
CDSS
Hospital
Quasi-
experimental
Beforeandafter
implem
entatio
nDataob
tained
from
patient
medical
records
Toexam
inetheim
pact
ofacompu
ter-
basedreminderin
addressing
adrug
shortage
The
stud
yshow
sa55
%decrease
inthe
numberof
orders,thus
inthepo
stim
plem
entatio
nperiod
,thereisan
increase
inthealternativemedication.
Inadditio
n,thisstud
yshow
scost
Tab
le4
(con
tinued)
Autho
rsSystem
Dom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
yFinding
424 J Med Syst (2007) 31:397–432
redu
ction(ann
ualsaving
:$3
6,55
2)for
theinstitu
tiondu
eto
the
implem
entatio
nof
thedrug
reminder
system
.Resultsshow
that
CDSSisan
effectivetool
inconserving
medications
during
times
ofnatio
nal
drug
shortagesby
rapidlychanging
prescribingpractices.
William
L.
Galanter
etal.
[200
4][46]
CPOE
Hospital
Coh
ort
Beforeandafter
implem
entatio
nDataob
tained
from
documentatio
nTomeasure
theeffectsof
alertsfor
inpatient
digo
xinuse(w
ithCPOE)
The
find
ings
ofthisstud
yshow
that
the
alertsgenerally
actedto
improv
ebo
ththespeedandov
erallmagnitude
ofapprop
riateclinicianrespon
seto
aclinical
situation.
Thisstud
yshow
sthat
alerts,generatedby
unsafe
medication,
improv
edthesafe
useof
digo
xin.
DuringCPOE,alerts
associated
with
missing
levelswere
effective.
For
hypo
kalemia
and
hypo
magnesaem
ia,thealertsgiven
during
CPOEwereno
tas
effectiveas
thosegivenat
thetim
eof
newly
repo
rted
low
electrolytes.During
CPOE,supp
lementatio
nfor
hypo
kalemia
was
notim
prov
edwhereas
supp
lementatio
nfor
hypo
magnesaem
iaim
prov
edin
one
hour.Itwas
also
observed
that
asyn
chrono
usalertsweresuccessful
inchanging
clinicianbehaviou
r,whereas
synchron
ouswereno
t.Itwas
observed
that
therespon
seto
thealertwas
depend
entup
onbo
ththemannerin
which
thealertwas
commun
icated
totheclinician,
synchron
icor
asyn
chrono
us,andtheclinical
context
ofthealert.
Joan
S.
Ash
etal.
[200
4][70]
CPOE
Hospital
Cross
sectional
After
implem
entatio
nQuestionn
aire
(openended
questio
n)distribu
ted96
4ho
spitals
Toassess
the
–Availabilityof
inpatient
compu
terized
physicianorderentry
–The
degree
towhich
physicians
are
usingit.
Thisstud
ydemon
stratesthat
CPOEstill
does
notem
ploy
widespread
implem
entatio
nacross
theU.S.
hospitals.The
results
ofthisstud
yshow
that
CPOEwas
notavailableto
83.7%
oftheho
spitals.Only9.7%
ofU.S.ho
spitalspresently
have
it.In
approx
imatelyhalfof
thoseho
spitals,
J Med Syst (2007) 31:397–432 425
morethan
90%
ofph
ysicians
use
CPOEandin
onethirdof
them
,more
than
90%
oftheorders
areenteredvia
CPOE.T
hedataalso
show
edthatthere
isatrendtowards
ahigh
erpercentage
ofho
spitalswith
CPOErequ
iringits
useov
erthepast5years,mov
ing
towards
concom
itant
trends
inhigh
erparticipationandsaturatio
nprop
ortio
ns.
Rob Shu
lman
etal.
[200
5][47]
CPOE
Hospital
Coh
ort
Beforeandafter
implem
entatio
nDatafrom
themedicationerrors
recorded
before
andafter
implem
entatio
nCPOE
Tocompare
theim
pact
ofCPOEwith
hand
writtenprescriptio
nsbasedon
thefrequency,
type
andou
tcom
eof
medicationerrors
inICU
Throu
ghthisstud
yitwas
demon
strated
that
theintrod
uctio
nof
CPOEwas
associated
with
:aredu
ctionin
the
prop
ortio
nof
medicationerrors
(from
6.7%
to4.8%
);an
increase
insome
typesof
errors
likedo
seerror,
omission
ofrequ
ired
drug
s.Itwas
notedthat
theprescriber’ssign
ature
mustbe
added.
There
was
anim
prov
ementin
theov
erallpatient
outcom
escore(the
meanAcute
Phy
siolog
yAnd
Chron
icHealth
Evaluation—
APA
CHE—IIscores)
with
CPOE.
ParkR.W
.et
al[200
5][52]
CPOE
Hospital
Survey
After
implem
entatio
n–Q
uestionn
aire
completed
bydirectorsof
hospital
inform
ationsystem
sin
122
hospitals(30teaching
hospitalsand92
general
hospitals)
Todeterm
inetheavailabilityof
CPOE
andEMRsystem
sin
teaching
and
generalho
spitalsin
therepu
blic
ofKorea
From
thisstud
y,itwas
foun
dthat
the
CPOEsystem
swereavailableat
all
locatio
nwith
intheho
spitalforall
typesof
orders
in80
.3%
ofallthe
hospitals.93
,3%
ofthesurveyed
teaching
hospitalshadcompleted
CPOEsystem
sbu
tgeneralho
spitals
didin
76,1%.86
%of
hospitalsreplied
that
theuseof
CPOEwas
mandatory;
9,3%
said
that
theho
spitalencouraged
itsuse,
and4,7%
said
that
theusewas
optio
nal.96
,7%
oftheteaching
hospitalsrequ
ired
theirph
ysicians
toorderby
CPOE.A
majority
ofph
ysicians
enteredmostof
their
medical
orders
into
CPOEsystem
.
Tab
le4
(con
tinued)
Autho
rsSystem
Dom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
yFinding
426 J Med Syst (2007) 31:397–432
Only9%
oftheho
spitalsgetavailable
acompleteEMR
system
.Teaching
hospitalstend
edto
demon
strate
ahigh
erpercentage
ofEMRusethan
did
generalho
spitals.With
consideringall
form
sof
EMRs,includ
ingpartial,two
thirds
oftheteaching
hospitalshave
leastsomeform
ofEMRs.
Tejal
Ket
al[200
5][53]
CPOE
Primary
care
Prospectiv
ecoho
rtstud
yAfter
implem
entatio
n18
79Prescriptionreview
,66
1patient
survey,and65
3chart
review
Toassess
therates,types,andseverity
ofou
tpatient
prescribingerrors
and
understand
thepo
tentialim
pact
ofcompu
terizedprescribing.
The
results
ofthestud
yshow
thatof
the
prescriptio
ns,14
3(7.6%)containeda
prescribingerror.Three
errors
ledto
preventableADEsand62
(43%
;3%
ofallprescriptio
ns)hadpo
tentialfor
patient
injury
(potentialADEs);on
ewas
potentially
lifethreatening(2%)
and15
wereseriou
s(24%
).Errorsin
frequency(n=77
,54
%)anddo
se(n=
26,18
%)werecommon
.The
ratesof
medicationerrors
andpo
tentialADEs
wereno
tsignificantly
differentatbasic
compu
terizedprescribingsites(4.3%
vs11.0%,P=.31;
2.6%
vs4.0%
,P
=.16)
comparedto
hand
writtensites.
Adv
ancedchecks
(including
dose
and
frequencychecking
)couldhave
prevented95
%of
potentialADEs.
William
L.
Galanter
etal.
[200
5][48]
CPOE
Hospital
Coh
ort
Beforeandafter
implem
entatio
nDataob
tained
from
thepatient
records
Toevaluate
theutility
ofalertsas
well
asanalyzethefactorsthat
may
play
arole
inno
ncom
pliancewith
alert
recommendatio
nsam
ongho
usestaff
inorderto
improv
eCDSin
thefuture
The
results
ofthestud
yshow
that
the
alerts(generated
byun
safe
medication)
wereeffectivein
decreasing
theorder
andadministrationof
drug
scontradictions
dueto
renalinsuffiency.
The
likelihoo
dof
apatient
receivingat
leaston
edo
seof
contradicted
medicationdecreasedfrom
89%
to47
%afteralertim
plem
entatio
n,which
show
sa42
%redu
ctionin
administration.
Itwas
notedthat
this
redu
ctionwas
almostentirelydu
eto
thecancellatio
nof
theorderafter
view
ingthealert,which
occurred
in41
%of
themistakesin
which
analert
was
given.
The
timeof
thedaywas
not
foun
dto
besign
ificantly
associated
with
thelik
elihoo
dof
alert
J Med Syst (2007) 31:397–432 427
compliance.
The
stud
yalso
revealed
that
compliancewith
thealertswas
high
erin
malepatients,was
notedto
increase
with
thedu
ratio
nof
house
stafftraining
,andincreasedin
patients
with
moresevere
renaldy
sfun
ction.
Ross
Kop
pel
etal.
[200
5][61]
CPOE
Hospital
Survey
After
implem
entatio
n–Interview
with
housestaffand
inform
ationtechno
logists,
nurses,ph
armacists,and
physicians
–Observatio
n(hou
sestaff,
nurses
andph
ysicians)–
Questionn
aire
toho
usestaff
Toidentifytherole
ofCPOEin
facilitatingprescriptio
nerrorrisk
The
find
ings
ofthestud
ydemon
strate
that
awidelyused
CPOEsystem
facilitated
22typesof
errorrisks.For
exam
ple,
fragmentedCPOEdisplays
mistakesfordo
sage
guidelines,
igno
redantib
iotic
renewals,no
tices
placed
onpaperchartsrather
than
intheCPOEsystem
,separatio
nof
functio
nsthat
facilitatedo
uble
dosing
andincompatib
leorders,and
inflexible
ordering
form
atsgeneratin
gwrong
orders.Three
quarters
oftheho
use
staffrepo
rted
observingeach
ofthese
errorrisksandindicatedthatthey
were
occurringweeklyor
moreoften.
Itwas
also
observed
that
manyerrors
occurred
frequently.Itwas
sugg
ested
that
asCPOEsystem
sare
implem
ented,
clinicianandho
spitals
mustattend
toerrorsthatthesesystem
scausein
additio
nto
errors
that
they
prevent.
Jaap
L.
Van
den
brink
etal.
[200
5][56]
DSS
Hospital
Coh
ort
After
implem
entatio
n–Q
uestionn
aire
topatientsdata
obtained
from
automated
logg
ingon
system
Todeterm
ineuse,
appreciatio
n,and
effectivenessof
aninform
ation
system
The
results
ofthestudyshow
thepatient
used
thesystem
intensively(averaging
4–5tim
esaweek).Itw
asobserved
that
thepatient
appreciatedthesystem
highly.89%
werenoticed
toadvise
other
patients.Patient
satisfactionquestio
nsscored
high
percentagesbetween91–
94%.M
ostp
atientswerenotedto
learn
thesystem
easily
(with
apercentage
of61%).The
system
detected
patient
problemsthathadnotb
eendiscovered
during
regularoutpatient
visits.
Tab
le4
(con
tinued)
Autho
rsSystem
Dom
ain
Study
design
Tim
eof
evaluatio
nSou
rceof
evidence
Aim
ofstud
yFinding
428 J Med Syst (2007) 31:397–432
analysis should be used to investigate influences of the newsystems on the organisation and its personnel.
In our review we found that during the period 2003–2005, most of the evaluations aimed to include issues suchas the effectiveness of the systems, the quality of care, userand patient satisfaction, and the system’s usability. And alsothe evaluation studies tended to use subjective approachescombined with quantitative studies in order to analysingcost and benefits.
Most of the studies included in this paper have usedsurvey methodology as their research method. Some of thestudies focused upon financial analysis, such as a cost andbenefit or consequences approach. Some of the studies useda clinical trial or cohort study to research the systems’outputs. It is, however, important to note that it has beendifficult to find generalized models and methods to evaluateIT-based applications in clinical settings that cover all as-pects such as economic and inter- and intra-organizationalapproaches. It seems that there are no clear guidelines forevaluation studies to develop evaluations and obtain moreclear and exact feedback about the implemented systems.
A key requirement for analytical work is a cleardefinition of what constitutes an IT-based application.Today there are different terminologies relating to similarsystems. Analysis of the effects has predominantly takenplace at an organizational level. However, this does not takeaccount of the differences in the settings in which analysisis taking place.
A significant barrier to investment in IT in healthcare isthe widely recognized fact that any cost saving resultingfrom technology changes is not always seen by theimplementer, but is rather passed on to a third party. Inessence, benefits appear at one site and in one budget, whilea large share of the cost commitments appear at another siteand in another budget. To our best knowledge, theevaluation studies performed do not include any discussionabout this important issue, or how lack of incentive to adoptsystems can influence the organization and its personnel.
As in other areas, such as healthcare for the elderly [62,63], evaluation studies in general pay more attention to thetechnical aspects and/or direct costs of the technology inuse. Further, the potential effects of the implementation ofIT-based applications are identified without analysing themfrom an inter-organizational and economic perspective.
In addition to difficulties in quantifying the output of theuse of IT in healthcare, evaluative challenges in assessingthe impact of IT include isolating its impacts from others. Itis difficult to find a clear relationship between IT,organizational improvements, quality of care and benefitsrealization.
The most common type of analytical approach usedtakes the form of a case study, and usually retrospectiveones that try to measure the actual impact of an ITV
itali
Sintchenk
oet
al.
[200
5][49]
DSS
Hospital
Coh
ort
Beforeandafter
implem
entatio
nDataob
tained
from
thepatient
records
Toassess
theeffect
ofacompu
ter-
baseddecision
supp
ortsystem
onantib
iotic
useandpatient
outcom
ein
acriticalcare
unit
The
results
ofthisstud
ysugg
estthatthe
useof
theDSScontribu
tedto
the
redu
ctionof
patient
leng
thof
stay
intheICU.The
introd
uctio
nof
theDSS
was
foun
dto
have
been
associated
with
aredu
ctionin
antib
iotic
usagein
theICU.Itwas
also
foun
dto
have
brou
ghtachange
inpatternsof
antib
ioticsusein
theICU.Itwas
also
demon
stratedthroug
hthisstud
ythat
therewas
difficulty
with
theuseof
the
DSSam
ongclinicians
with
different
roles.
J Med Syst (2007) 31:397–432 429
investment or prospective assessments. The evaluationstherefore suffer two main weaknesses. First, it is difficult toattribute IT investments to observed effects, because it isusually not possible to apply case-control design to theevaluative study. Second, it is difficult to generalize fromthe particular study to the broader context. The results ofour review show that the study of the impact of IT inhealthcare is currently limited. No studies have beenconducted to explore the impact of IT on the system as awhole. Few organizational-level studies exist that focusexplicitly upon the impact of IT in a specific sector of care.This review therefore suggests that there is an increasingneed to share knowledge and find methods to evaluate theimpact of investments, and in parallel with this, formulateindicators for success. It is therefore interesting to developor extend evaluation methods that can be applied to thisarea, but also that have a multi-actor perspective in order tounderstand the effects, consequences and prerequisites thathave to be achieved for the successful implementation anduse of IT in healthcare.
References
1. Kaplan, B., Evaluating informatics applications—some alternativeapproaches: theory, social interactionism, and call for methodo-logical pluralism. Int. J. Med. Inform. 64:39–56, 2001.
2. Ammenwerth, E., Brender, J., Nykanen, P., Prokosch, H. U.,Rigby, M., and Talmon, J., Visions and strategies to improveevaluation of health information systems. Reflections and lessonsbased on the HIS-EVAL workshop in Innsbruck. Int. J. Med.Inform. 73:479–491, 2004.
3. Ammenwerth, E., Iller, C., and Mansmann, U., Can evaluationstudies benefit from triangulation? A case study. Int. J. Med.Inform. 70:237–248, 2003.
4. Andersen, J. G., Aydin, C. E., and Jay, S. J., EvaluatingHealthcare Information Systems: Methods and Applications.Thousand Oaks, CA: Sage publication, 1994.
5. Friedman, C. P., and Wyatt, J. C., Evaluation Methods in MedicalInformatics. New York: Springer, 1996.
6. Kaplan, B., and Shaw, N. T., Future directions in evaluationresearch: People, organizational, and social issues. Methods Inf.Med. 43:215–231, 2004.
7. Phil, W., An assessment of strategies for information systemevaluation: Lessons for education. Int. J. Educ. Res. 25:361–371,1996.
8. Delpierre, C., Cuzin, L., Fillaux, J., Alvarez, M., Massip, P., andLang, T., A systematic review of computer-based patient recordsystems and quality of care: More randomized clinical trials or abroader approach. Int. J. Qual. Health Care. 16:407–416, 2004.
9. Wang, S. J., Middleton, B., Bardon, C. G., Spurr, C. D., Carchidi, P. J.,Kittler, A. f., Goldszar, R. C., Fairchild, D. G., Sussman, A. J.,Kuperman, G. J., andBates, D.W., A cost-benefit analysis of electronicmedical records in primary care. Am. J. Med. 114:397–403, 2003.
10. Laerum, H., Karlsen, T. H., and Faxvaag, A., Effects of scanning andeliminating paper-based medical records on hospital physicians’clinical work practice. J. Am. Med. Inform. Assoc. 10:160–588, 2003.
11. Ruland, C. M., and Ravn, I. H., Usefulness and effects on costsand staff management of a nursing resource managementinformation system. J. Nurs. Manag. 11:208–215, 2003.
12. Houston, T. K., Ray, M. N., Crawford, M. A., Giddens, T., andBerner, E. S., Patient perceptions of physician use of handheldcomputers. AMIA Annu. Symp. Proc. 299–303, 2003.
13. Ammenwerth, E., Mansmann, U., Iller, C., and Eichstädter, R.,Factors affecting and affected by user acceptance of computer-based nursing documentation: Results of a two-year study. J. Am.Med. Inform. Assoc. 10:69–84, 2003.
14. Zhang, W. P., Yamauchi, K., Mizuno, S., Zhang, R., and Huang,D. M., Analysis of cost and assessment of computerized patientrecord systems in Japan based on questionnaire survey. Med.Inform. Internet Med. 29:229–238, 2004.
15. Laerum, H., Karlsen, T. H., and Faxvaag, A., Use of and attitudesto a hospital information system by medical secretaries, nursesand physicians deprived of the paper-based medical record: Acase report BMC Med. Inform. Decis. Mak. 4:18, 2004.
16. Hier, D. B., Rothschild, A., LeMaistre, A., and Keeler, J.,Differing faculty and housestaff acceptance of an electronic healthrecord one year after implementation. Medinfo. 11:1300–1303,2004.
17. Rotich, J. K., Hannan, T. J., Smith, F. E., Bii, J., Odero, W. W.,Vu, N., Mamlin, B. W., Mamlin, J. J., Einterz, R. M., and Tierney,W. M., Installing and implementing a computer-based patientrecord system in sub-Saharan Africa: The Mosoriot MedicalRecord System. Med. Inform. Internet Med. 10:295–303, 2003.
18. Stengel, D., Bauwens, K., Walter, M., Kopfer, T., and Ekkernkamp,A., Comparison of handheld computer-assisted and conventionalpaper chart documentation of medical records. A randomized,controlled trial. J. Bone Joint Surg. Am. 86:553–560, 2004.
19. Likourezos, A., Chalfin, D. B., Murphy, D. G., Sommer, B.,Darcy, K., and Davidson, S. J., Physician and nurse satisfactionwith an Electronic Medical Record System. J. Emerg. Med.27:419–424, 2004.
20. Pagliari, C., Gilmour, M., and Sullivan, F., Evaluating nationalelectronic clinical communications implementation in Scotland.Medinfo. 11:1794, 2004.
21. Baum, A., Figar, S., Serverino, J., Assale, D., Schachner, B.,Otero, P., Luna, D., and de Quiros, F. G., Assessing the impact ofchange in the organization of a technical support system for ahealth information systems (HIS). Medinfo. 11:1367–1370, 2004.
22. Hobson, J. C., Khemani, S., and Singh, A., Prospective audit ofthe quality of ENT emergency clinic notes before and afterintroduction of a computerized template. J. Laryngol. Otol.119:264–266, 2005.
23. Kaushal, R., Blumenthal, D., Poon, E. G., Jha, A. K., Franz, C.,Middleton, B., Glaser, J., Kuperman, G., Christino, M., Fernandopulle,R., Newhouse, J. P., and Bates, D. W., The costs of a nationalhealth information network. Ann. Intern. Med. 143:165–173,2005.
24. Raja, E. E., Mahal, R., and Masih, V. B., An exploratory study toassess the computer knowledge, attitude and skill among nurses inhealthcare setting of a selected hospital in Ludhiana, Punjab,India. Medinfo. 11:1304–1307, 2004.
25. Clayton, P. D., Naus, S. P., Bowes, W. A., Madsen, T. S., Wilcox,A. B., Orsmond, G., Rocha, B., Thornton, S. N., Jones, S.,Jacobsen, C. A., Udall, M. R., Rhodes, M. L., Wallace, B. E.,Cannon, W., Gardner, J., Huff, S. M., and Leckman, L., Physicianuse of electronic medical records: issues and successes with directdata entry and physician productivity. AMIA Annu. Symp. Proc.141–145, 2005.
26. White, M., Computerised versus handwritten records. Paediatr.Nurs. 17:15–18, 2005.
27. Wilcox, A. B., Jones, S. S., Dorr, D. A., Cannon, W., Burns, L.,Radican, K., Christensen, K., Brunker, C., Larsen, A., Narus, S.P., Thornton, S. N., and Clayton, P. D., Use and impact of acomputer-generated patient summary worksheet for primary care.AMIA Annu. Symp. Proc. 824–828, 2005.
430 J Med Syst (2007) 31:397–432
28. Makela, K., Virjo, I., Aho, J., Kalliola, P., Koivukoski, A. M.,Kurunmaki, H., Kahara, M., Uusitalo, L., Valli, M., Vuotari, V.,and Ylinen, S., Electronic patient record systems and the generalpractitioner: an evaluation study. J. Telemed. Telecare. 11:S2:66–68, 2005.
29. Zurita, L., and Nohr, C., Patient opinion–EHR assessment fromthe users perspective. Medinfo. 11:1333–1336, 2004.
30. Breslow, M. J., Rosenfeld, B. A., Doerfler, M., Burke, G., Yates,G., Stone, D. J., Tomaszewicz, P., Hochman, R., and Plocher, D.W., Effect of a multiple-site intensive care unit telemedicineprogram on clinical and economic outcomes: an alternativeparadigm for intensivist staffing. Crit. Care Med. 32:31–38, 2004.
31. Scalvini, S., Capomolla, S., Zanelli, E., Benigno, M., Domenighini,D., Paletta, L., Glisenti, F., and Giordano, A., Effect of home-basedtelecardiology on chronic heart failure: Costs and outcomes. J.Telemed. Telecare. 11:S1:16–18, 2005.
32. Noble, S. M., Coast, J., and Benger, J. R., A cost-consequencesanalysis of minor injuries telemedicine. J. Telemed. Telecare.11:15–19, 2005.
33. Persaud, D. D., Jreige, S., Skedgel, C., Finley, J., Sargeant, J., andHanlon, N., An incremental cost analysis of telehealth in NovaScotia from a societal perspective. J. Telemed. Telecare. 11:77–84,2005.
34. Dillon, E., Loermans, J., Davis, D., and Xu, C., Evaluation of thewesterrn Australian Department of Health telehealth project.J. Telemed. Telecare. 11:S2:19–S2:121, 2005.
35. Larcher, B., Arisi, E., Berloffa, F., Demichelis, F., Eccher, C.,Galligioni, E., Galvagni, M., Martini, G., Sboner, A., Tomio, L.,Zumiani, G., Graiff, A., and Forti, S., Analysis of user-satisfactionwith the use of a teleconsultation system in oncology. Med.Inform. Internet. Med. 28:73–84, 2003.
36. Bratan, T., Clarke, M., Jones, R., Larkworthy, A., and Paul, R.,Evaluation of the practical feasibility and acceptability of homemonitoring in residential homes. J. Telemed. Telecare. 11(Suppl 1):29–31, 2005.
37. Mair, F. S., Goldstein, P., May, C., Angus, R., Shiels, C., Hibbert,D., O’Connor, J., Boland, A., Roberts, C., Haycox, A., andCapewell, S., Patient and provider perspectives on home telecare:preliminary results from a randomized controlled trial. J. Telemed.Telecare. 11(Suppl 1):95–97, 2005.
38. Linassi, A. G., and Li Pi Shan, R., User satisfaction with atelemedicine amputee clinic in Saskatchewan. J. Telemed. Tele-care. 11:414–418, 2005.
39. Young, T. L., and Ireson, C., Effectiveness of school-basedtelehealth care in urban and rural elementary schools. Pediatrics.112:1088–1094, 2003.
40. Baldwin, L., Clarke, M., Hands, L., Knott, M., and Jones, R., Theeffect of telemedicine on consultation time. J. Telemed. Telecare.9:S1:71–73, 2003.
41. Finkelstein, J., Khare, R., and Ansell, J., Feasibility and patients’acceptance of home automated telemanagement of oral anti-coagulation therapy. Proc. AMIA Symp. 230–234, 2003.
42. Vontetsianos, T., Giovas, P., Katsaras, T., Rigopoulou, A.,Mpirmpa, G., Giaboudakis, P., Koyrelea, S., Kontopyrgias, G.,and Tsoulkas, B., Telemedicine-assisted home support for patientswith advanced chronic obstructive pulmonary disease: preliminaryresults after nine-month follow-up. J. Telemed. Telecare. 11:S1:86–S1:88, 2005.
43. Grady, B. J., and Melcer, T., A retrospective evaluation ofTeleMental Healthcare services for remote military populations.Telemed J E Health. 11:551–558, 2005.
44. Wong, Y. K., Hui, E., and Woo, J., A community-based exerciseprogramme for older persons with knee pain using telemedicine.J. Telemed. Telecare. 11:310–315, 2005.
45. Beuscart-Zephir, M. C., Pelayo, S., Degoulet, P., Anceaux, F.,Guerlinger, S., and Meaux, J. J., A usability study of CPOE’s
medication administration functions: impact on physician-nursecooperation. Medinfo. 11:1018–1022, 2004.
46. Galanter, W. L., Polikaitis, A., and DiDomenico, R. J., A trial ofautomated safety alerts for inpatient digoxin use with computerizedphysician order entry. J. Am. Med. Inform. Assoc. 11:270–277, 2004.
47. Shulman, R., Singer, M., Goldstone, J., and Bellingan, G.,Medication errors: a prospective cohort study of hand-writtenand computerised physician order entry in the intensive care unit.Crit. Care. 9:R516–R521, 2005.
48. Galanter, W. L., Didomenico, R. J., and Polikaitis, A., A trial ofautomated decision support alerts for contraindicated medicationsusing computerized physician order entry. J. Am. Med. Inform.Assoc. 12:269–274, 2005.
49. Sintchenko, V., Iredell, J. R., Gilbert, G. L., and Coiera, E.,Handheld computer-based decision support reduces patient lengthof stay and antibiotic prescribing in critical care. J. Am. Med.Inform. Assoc. 12:398–402, 2005.
50. Short, D., Frischer, M., and Bashford, J., The development andevaluation of a computerised decision support system for primarycare based upon ‘patient profile decision analysis’. Inform. Prim.Care. 11:195–202, 2003.
51. Ruland, C. M., White, T., Stevens, M., Fanciullo, G., and Khilani,S. M., Effects of a computerized system to support shared decisionmaking in symptom management of cancer patients: Preliminaryresults. J. Am. Med. Inform. Assoc. 10:573–579, 2003.
52. Park, R. W., Shin, S. S., Choi, Y. I., Ahn, J. O., and Hwang, S. C.,Computerized physician order entry and electronic medical recordsystems in Korean teaching and general hospitals: Results of a2004 survey. J. Am. Med. Inform. Assoc. 12:642–647, 2005.
53. Gandhi, T. K., Weingart, S. N., Seger, A. C., Borus, J., Burdick,E., Poon, E. G., Leape, L. L., and Bates, D. W., Outpatientprescribing errors and the impact of computerized prescribing.J. Gen. Intern. Med. 20:837–841, 2005.
54. Ohsfeldt, R. L., Ward, M. M., Schneider, J. E., Jaana, M., Miller,T. R., Lei, Y., and Wakefield, S., Implementation of hospitalcomputerized physician order entry systems in a rural state:Feasibility and financial impact. J. Am. Med. Inform. Assoc.12:20–27, 2005, Jan.
55. Bogucki, B., Jacobs, B. R., and Hingle, J., Computerizedreminders reduce the use of medications during shortages.J. Am. Med. Inform. Assoc. 11:278–280, 2004.
56. Van den Brink, J. L., Moorman, P. W., de Boer, M. F., Pruyn, J. F.,Verwoerd, C. D., and van emmel, J. H., Involving the patient: aprospective study on use, appreciation and effectiveness of aninformation system in head and neck cancer care. Int. J. Med.Inform. 74:839–849, 2005.
57. Choi, S. S., Jazayeri, D. G., Mitnick, C. D., Chalco, K., Bayona,J., and Fraser, H. S., Implementation and initial evaluation of aWeb-based nurse order entry system for multidrug-resistanttuberculosis patients in Peru. Medinfo. 11:202–206, 2004.
58. Odhiambo-Otieno, G. W., Evaluation of existing district healthmanagement information systems a case study of the districthealth systems in Kenya. Int. J. Med Inform. 74:733–744, 2005.
59. Scott, J. T., Rundall, T. G., Vogt, T. M., and Hsu, J., KaiserPermanente’s experience of implementing an electronic medicalrecord: A qualitative study. BMJ. 331:1313–1316, 2005.
60. Aarts, J., and Berg, M., A tale of two hospitals: a sociotechnicalappraisal of the introduction of computerized physician orderentry in two Dutch hospitals. Medinfo. 11:999–1002, 2004.
61. Koppel, R., Metlay, J. P., Cohen, A., Abaluck, B., Localio, A. R.,Kimmel, S. E., and Strom, B. L., Role of computerized physicianorder entry systems in facilitating medication errors. JAMA.293:1197–1203, 2005.
62. Vimarlund, V., and Olve, N. G., Economic analyses for ICT inelderly healthcare: Questions and challenges. Health InformaticsJ. 11:309–321, 2005.
J Med Syst (2007) 31:397–432 431
63. Olve, N. G., Vimarlund, V., and Agerbo, M. Evaluation as multi-actor trade-off—A challenge in introducing ICT innovations in thehealth sector. Invited paper, 4th WSEAS International Conferenceon E-Activities. Miami, Florida, USA, November 17–19, 2005.
64. Adams, W. G., Mann, A. M., and Bauchner, H., Use of anelectronic medical record improves the quality of urban pediatricprimary care. Pediatrics. 111:626–632, 2003.
65. Pizziferri, L., Kittler, A. F., Volk, L. A., Honour, M. M., Gupta, S.,Wang, S., Wang, T., Lippincott, M., Li, Q., and Bates, D. W.,Primary care physician time utilization before and after imple-mentation of an electronic health record: A time-motion study.J. Biomed. Inform. 38:176–188, 2005.
66. Bryson, M., Tidy, N., Smith, M., and Levy, S., An online surveyof nurses’ perceptions, knowledge and expectations of theNational Health Service modernization programme. J. Telemed.Telecare. 11:S1:64–66, 2005.
67. Hebert, M. A., and Korabek, B., Stakeholder readiness fortelehomecare: implications for implementation. Telemed J EHealth. 10:85–92, 2004.
68. Kaufman, D. R., Starren, J., Patel, V. L., Morin, P. C., Hilliman,C., Pevzner, J., Weinstock, R. S., Goland, R., and Shea, S. Acognitive framework for understanding barriers to the productiveuse of a diabetes home telemedicine system. AMIA Annu. Symp.Proc. 356–360, 2003.
69. Ash, J. S., Sittig, D. F., Seshadri, V., Dykstra, R. H.,Carpenter, J. D., and Stavri, P. Z., Adding insight: aqualitative cross-site study of physician order entry. Medinfo.11:1013–1017, 2004.
70. Ash, J. S., Gorman, P. N., Seshadri, V., and Hersh, W. R.,Computerized physician order entry in U.S. hospitals: results of a2002 survey. J. Am. Med. Inform. Assoc. 11:95–99, 2004.
71. Ammenwerth, E., and de Keizer, N., An inventory ofevaluation studies of information technology in healthcaretrends in evaluation research 1982–2002. Methods Inf. Med.44:44–56, 2005.
72. Kaplan, B., Evaluating informatics applications—Clinical deci-sion support systems literature review. Int. J. Med. Inform. 64:39–56, 2001.
432 J Med Syst (2007) 31:397–432