Mobilizing clinical decision support to facilitate knowledge translation:A case study in China

Yinsheng Zhang a,c, Haomin Li b,n, Huilong Duan a, Yinhong Zhao d

a College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, Chinab Children's Hospital, Institute of Translational Medicine, Zhejiang University, Hangzhou 310027, Chinac School of Computer Science and Information Engineering, Zhejiang Gongshang University, Hangzhou 310018, Chinad China National Center for Biotechnology Development, Beijing 100036, China

a r t i c l e i n f o

Article history:Received 4 August 2014Accepted 14 February 2015

Keywords:Knowledge translationClinical decision supportKnowledge representationKnowledge acquisitionInference engineContext-aware knowledge retrievalNatural language processing

a b s t r a c t

Background: A wide gulf remains between knowledge and clinical practice. Clinical decision support hasbeen demonstrated to be an effective knowledge tool that healthcare organizations can employ to deliverthe “right knowledge to the right people in the right form at the right time”. How to adopt variousclinical decision support (CDS) systems efficiently to facilitate evidence-based practice is one challengefaced by knowledge translation research.Method: A computer-aided knowledge translation method that mobilizes evidence-based decisionsupports is proposed. The foundation of the method is a knowledge representation model that is ableto cover, coordinate and synergize various types of medical knowledge to achieve centralized andeffective knowledge management. Next, web-based knowledge-authoring and natural language proces-sing based knowledge acquisition tools are designed to accelerate the transformation of the latest clinicalevidence into computerized knowledge content. Finally, a batch of fundamental services, such as dataacquisition and inference engine, are designed to actuate the acquired knowledge content. These servicescan be used as building blocks for various evidence-based decision support applications.Results: Based on the above method, a computer-aided knowledge translation platform was constructedas a CDS infrastructure. Based on this platform, typical CDS applications were developed. A case study ofdrug use check demonstrates that the CDS intervention delivered by the platform has producedobservable behavior changes (89.7% of alerted medical orders were revised by physicians).Discussion: Computer-aided knowledge translation via a CDS infrastructure can be effective in facilitatingknowledge translation in clinical settings.

& 2015 Elsevier Ltd. All rights reserved.

1. Introduction

1.1. The knowledge-practice chasm

Globally, healthcare organizations fail to use evidence optimally. In1973, Wennberg and Gittelsohn published their landmark articledemonstrating substantial variation among different healthcare ser-vice providers [1]. Forty years later, this situation has not changed. Alarge gulf remains betweenwhat we know and what we practice. Onestudy in 2003 showed that only 54.9% of US patients receivedevidence-based care [2]. Similar findings have been reported globallyin both developed and developing countries [3].

Several reasons have caused this knowledge-practice gap. First,knowledge grows exponentially. It is estimated that each year 10,000

diseases, 3000 drugs, and 400,000 articles are added to the biomedicaldomain [4], and every 19 years, the volume of the literature in thebiomedical domain is doubled [5]. This explosive knowledge growthhas inflicted a heavy cognitive load on clinicians. Second, medicalknowledge itself is evolving. Former conclusions can be overturned bythe latest evidence. According to a 2001 survey, 50% of guidelineknowledge becomes invalid in 5.8 years, and the validity period of 90%guidelines is only 3.6 years [6]. The dynamic nature of knowledgeinflicts great pressure on physicians to respond to the latest clinicalevidence in a timely manner. Third, the knowledge translation courseis long. The course could take as long as 10 years for newly discoveredknowledge to enter textbooks [7], and it takes 10–17 years for newknowledge to be transformed into routine practice [8].

1.2. Clinical decision support as a knowledge tool to facilitateknowledge translation

To close the chasm between knowledge and practice, knowledgetranslation (KT) in healthcare has beenwidely recognized and studied

Contents lists available at ScienceDirect

journal homepage: www.elsevier.com/locate/cbm

Computers in Biology and Medicine

http://dx.doi.org/10.1016/j.compbiomed.2015.02.0130010-4825/& 2015 Elsevier Ltd. All rights reserved.

n Corresponding author. Tel.: þ86 571 87951792.E-mail addresses: elevenjohns@163.com (Y. Zhang), hmli@zju.edu.cn (H. Li),

duanhl@zju.edu.cn (H. Duan), zhaoyh@cncbd.org.cn (Y. Zhao).

Computers in Biology and Medicine 60 (2015) 40–50

in the past decade. According to the Canadian Institutes of HealthResearch (CIHR), KT has been defined as “a dynamic and iterativeprocess that includes the synthesis, dissemination, exchange andethically sound application of knowledge to provide more effectivehealth services and products and strengthen the healthcare system”

[9]. Because knowledge accumulates and evolves rapidly, KT isconstantly faced with the challenge of synthesizing the vast body ofhighly dynamic knowledge content and utilizing it at the point of care[10]. To meet this challenge, healthcare organizations need effectiveknowledge management tools to organize and utilize the vast knowl-edge content to maintain competitive performance in such a highlydynamic environment [11]. CDS (clinical decision support) is such atool that a healthcare organization can employ to deliver the “rightknowledge to the right people in the right form at the right time”[12]. Many studies [13–19] have demonstrated CDS's capability toimprove evidence-based practice and facilitate knowledge translation.

1.3. Needs of CDS mobilization infrastructure

Traditional CDS systems are developed as independent systemsby different vendors. They usually target a specific application (suchas diagnostic CDS [20], lab test surveillance and alerting [21,22], ordrug use checking [23]) and use private knowledge representationand implementation methods. This has led to fragmented, disinte-grated and inconsistent knowledge, therefore introducing complex-ity in developing CDS front ends [24] and managing back-endknowledge content [16]. Because of such heterogeneity, coordinatedknowledge content and unified knowledge management cannot beeasily achieved in healthcare organizations [25]. As an amendmentto traditional CDS systems, the 2000 AMIA symposium recom-mended “maintainable technical and methodological foundationsfor computer-based decision support” as a key factor for implement-ing “evidence-based (i.e., based on timely and latest knowledge)”CDS [26,27]. Therefore, the mission of this study is to design andconstruct a mobilization infrastructure or foundation where compu-terized clinical decision support is used as the major knowledge toolthat facilitates knowledge translation. The infrastructure should beable to provide centralized knowledge management and efficientknowledge acquisition such that the rapidly changing clinicalevidence can be quickly and effectively applied to computerizeddecision support.

2. Methods

As mentioned above, a CDS infrastructure is a promising alter-native to traditional CDS systems in facilitating knowledge transla-tion (Fig. 1). The computer-aided knowledge translation processenabled by computerized decision support tools can be summarizedas three procedures: 1. Knowledge representation. Before medicalknowledge can be used in various computerized decision supportsystems, it needs to be represented in a computer-interpretableformat. Knowledge representation determines the knowledge baseschema and what types of computerized decision supports can beprovided. 2. Knowledge acquisition. Knowledge acquisition trans-forms medical knowledge embedded in the literature and textbooksinto computer-interpretable formats determined by knowledgerepresentation formalisms. This procedure is usually performed byknowledge engineers. 3. Knowledge application. Computer-inter-pretable knowledge contents are applied in healthcare decisionmaking in the form of computerized decision support.

To support knowledge translation better, this paper aims to designa CDS infrastructure that could benefit all of the above threeprocedures. Implementation of such an infrastructure would entailseveral distinctive characteristics compared with traditional CDSsystems. First, the infrastructure needs a general-purpose knowledge

representationmodel to cover, coordinate and synergize various typesof medical knowledge in computer-interpretable forms, which laysthe foundation for centralized knowledge management and subse-quent knowledge dissemination via computerized decision supportservices [28,29]. Second, the infrastructure should provide highlyefficient knowledge acquisition and authoring tools to reduce theknowledge acquisition time and cost. Knowledge acquisition is atime-consuming process and can take as much as 50% of the totaleffort for building a CDS system [30]. Efficient and timely knowledgeacquisition ensures that the provided CDS intervention is based onthe latest clinical evidence. Third, the infrastructure should provide amechanism to transfer acquired knowledge contents to evidence-based decision supports efficiently.

To fulfill the above requirements, this paper designs a computer-aided knowledge translation framework as such a CDS infrastructure.As shown in Fig. 2, the framework is composed of five components.The first component is knowledge representation, which defines aunified ontology to cover and coordinate multiple knowledge types.The second component is knowledge acquisition, where knowledgecontent is acquired and authored using computer-aided methods. Thethird component is the knowledge base, which stores various types ofknowledge and their inter-relationships as defined by the unifiedontology. The fourth is a layer of knowledge-driven services such asthe inference engine, treatment recommendation, and context-awareknowledge retrieval, which are used as fundamental service modulesby diverse CDS applications. The fifth is a layer of CDS applications thatinteract with end users. The above framework covers the threeprocedures (knowledge representation, knowledge acquisition andknowledge application) of computer-aided knowledge translation.The framework supports knowledge translation by the followingworkflow (Fig. 2).

1. Design a knowledge representation model suited for knowledgetranslation, and construct the corresponding knowledge base.2. Knowledge content is continuously synthesized, curated andmaintained by an expert panel through a knowledge-authoring webportal. 3. In addition to the aforementioned authoring web portal,

CDS1

KE1

EMRS

KB1

CDS2

KE2

CDS3

KE3

CDS1

CDS2

CDS3

CDS Infrastructure

KEKBEMRS

vendor1

vendor2

vendor3

KnowledgeExecution

Environment

KR

KR1

KB2

KR2

KB3

KR3

Fig. 1. Computer-aided knowledge translation process by computerized decisionsupport tools. (A) Knowledge translation facilitated by traditional CDS systemsdeployed in a healthcare organization. (B) Knowledge translation facilitated by aCDS infrastructure. KR¼knowledge representation; KB¼knowledge base; KE¼-knowledge engineer.

Y. Zhang et al. / Computers in Biology and Medicine 60 (2015) 40–50 41

computer-aided knowledge acquisition is enabled by technologiessuch as NLP (natural language processing), which extracts structuredcontent from free text resources. 4. An integration engine [31] whichconnects various information systems and routes messages betweenthem acts as an ideal clinical event source for the inference engine.For example, when a new lab test report is generated in LIS(laboratory information system), the integration engine will notifythe inference engine to start a reasoning process. 5. A CDR (clinicaldata repository), which aggregates various clinical data through theintegration engine acts as a central data source. 6. A data acquisitionservice retrieves both structured and free text data from CDR forvarious CDS applications. 7. NLP tasks for specific types of clinicaldocuments can be defined to extract structured data from free text. 8.Structured data provided by the data acquisition service are fed intothe inference engine as facts. 9. Drug use rules or diagnostic rules areloaded into the inference engine to generate drug alerts or clinicalproblems. 10. Care protocols associated with a detected clinicalproblem are recommended as normalized interventions. 11. HL7Infobutton based context-aware knowledge retrieval provideshuman-readable knowledge for CDS interventions to improve useracceptance. 12. An application store that manages registeredcomputer-aided tools pushes and executes hardcoded knowledgeapplications under specific contexts by a plug-in technical framework[32]. 13. The above services can be co-delivered in a specific sequenceto provide collaborative decision support. For example, the dataacquisition, inference engine and treatment recommendation servicescan be orchestrated to implement the “patient data, diagnosis,treatment” decision support process. 14. Various CDS applicationsenabled by the above services interact with physicians to improvedecision making.

In the following manuscript, we will introduce each componentof the framework in detail and explain how they assist knowledgetranslation through the above workflow.

2.1. Knowledge representation

As mentioned before, the heterogeneity of knowledge repre-sentation in traditional CDS systems has limited their ability toprovide full-scale collaborative knowledge services. To address thisproblem, we constructed a unified ontology that incorporatesvarious types of knowledge as well as their semantic relationships,

as shown in Fig. 3. The ontology contains the following knowledgecategories:

The diagnostic knowledge category defines diagnostic rules andrelated concepts for clinical problems. Context items (e.g., systolicblood pressure and diastolic blood pressure) that are classified intoseveral semantic groups (e.g., lab test, vital sign and symptom) areused to define diagnostic rules (e.g., systolic blood pressur-e4140 mmHg). An inference engine processes the rules to drivediagnostic decision support.

The treatment knowledge category defines recommended careprotocols for clinical problems, such as standard order sets andclinical pathways. Care protocols are designed by clinical expertsbased on the latest evidence. In this ontology, the clinical pathwayis modeled as a multi-phase care plan, while a standard order setis a single-phase care plan. Each phase of the care plan contains apredefined order set such as nursing, diet, medication, lab testrequest, and surgery request.

The pharmaceutical knowledge category defines various drug userules, such as dosage, frequency, administration route, solvent,indication, contraindication, allergy, pregnancy class, DDI (drug–druginteractions), and insurance policy, for medication-related decisionsupport. Other pharmaceutical resources, such as ADR (adverse drugreactions) and DDD (defined daily doses) are also included.

The above three categories are formally represented knowl-edge. However, there also exists much knowledge directly hard-coded in various clinical applications where explicit knowledge isnot available. Traditionally, such hardcoded knowledge is alwaysexcluded from knowledge management. In our approach, thehardcoded knowledge embedded in various computer-aided toolscan also be managed and linked to other knowledge. In this way,the hardcoded knowledge serves as a supplement to formallyrepresented knowledge. For example, a personalized drug dosingtool can be linked to medication orders in a care protocol. Whenthe care protocol is applied, the tool will be invoked at the point ofcare to fulfill personalized medication. The primary goal ofincluding hardcoded knowledge is to overcome the limitations ofthe formal knowledge representation's scalability and capability[33]. In this computer-aided knowledge translation framework,these tools are managed by an application store service (refer to“Section 2.4.5”) and are integrated into the clinical workflow by aplug-in mechanism [32].

Knowledge Base

physicians

docfreetext Integration Engine

Inference Engine

CDR

LIS RIS CIS

Clinical problemDiagnostic rulesFacts

Diagnosis

Knowledge-authoringweb portal

Context-awareknowledge retrieval

Drug usechecking Diagnostic CDS

Clinical Decision Support ApplicationsOther CDSapplications

Data Acquisition TreatmentRecommendation

Knowledge Driven ServicesApplicationStore

NLP-basedknowledgeacquisition

Knowledge Acquisition

GuidelinesSystematic reviewDrug fact sheets

LiteratureDosing advisorFormulary supportRisk evaluator

HardcodedTreatmentStandard order setsClinical pathways

Drug use rulesADEDDD

PharmLexiconAlgorithmsTerminologies

NLP

Public KnowledgeDrug labelsCochrane reviewsMOH clinical pathways

Clinical expertsPharmacistsMedical insurance staff

Expert Panel

11

2 3

12107 9

5

46 8

14

UnifiedKnowledgeOntology

Knowledge Representation

1

13

Fig. 2. Computer-aided knowledge translation framework. Each number in the figure represents a data communication: (1) database schema according to unified knowledgeontology, (2,3) knowledge content, (4) clinical event, (5,6) clinical data, (7) NLP task, (8) facts (structured clinical data), (9) drug use rules and diagnostic rules, (10) careprotocol, (11) human-readable literature, (12) computer-aided tools, (13) fundamental services for building CDS applications, (14) CDS interventions.

Y. Zhang et al. / Computers in Biology and Medicine 60 (2015) 40–5042

Both formally represented knowledge and hardcoded knowl-edge are “machine-doable knowledge”, which are based on“human-readable knowledge”, such as clinical practice guidelines,meta-analyses, systematic reviews, and randomized clinical trials.Machine-doable knowledge is processable by computers andmakes knowledge application automatic, efficient and withoutbias. The human-readable knowledge explains what happened inblack boxes and can effectively improve user acceptance [34].Moreover, the adverse effect of inappropriately complied knowl-edge could be suppressed if human-readable knowledge is avail-able at point of care. Both machine-doable knowledge and human-readable knowledge can be co-delivered to clinicians.

This ontology not only defines various knowledge types butalso the semantic relations between them. Three relations thatcoordinate these knowledge types should be emphasized: 1. Theclinical problem acts as an intermediary between diagnosticand treatment knowledge. When a clinical problem is detectedby a diagnostic rule, corresponding care protocols can be recom-mended. 2. Hardcoded knowledge “supplements” formally repre-sented knowledge. 3. Machine-doable knowledge “references”human-readable knowledge. The above semantic relations linkdifferent knowledge categories and provide synergetic knowledgeresources for various clinical applications.

In summary, the unified ontology defines what types of knowl-edge are incorporated and managed. It determines the knowledgebase schema and lays the foundation for computer-aided knowl-edge translation.

2.2. Knowledge acquisition

For reliable knowledge acquisition, the involvement of clinicalexperts is required. In our collaborator hospital (DaYi hospital, 2600beds, ShanXi Province, China), an expert panel was established underthe supervision of the hospital director. The expert panel currentlycontains 35 clinicians, 6 pharmacists, and 1 medical insurance staffmember. The clinicians were selected from 35 clinical departments(one for each department). They are responsible for maintainingclinical problems, literature, order sets, and clinical pathways used ineach specialty. Pharmaceutical knowledge, such as drug use rulesand drug labels are maintained by the pharmacists. One medical

insurance staff member was also recruited for editing insurancepolicies, such as required indications for specific drugs.

Two knowledge acquisition modes are provided to help expertsefficiently create and update knowledge content. The first is aknowledge authoring web portal and the other is computer-aidedknowledge acquisition, which extracts computer-processable knowl-edge from public resources by technologies such as Chinese languageoriented NLP.

The knowledge-authoring web portal is a knowledge acquisitiontool that acts as a bridge between humans and the knowledge base.These tools usually provide structured data entry templates or formsfor experts to create computerized knowledge content. Fig. 4 shows theknowledge-authoring web portal developed using ASP.Net MVC tech-nology. The web portal is hosted on an intranet IIS server, and contentcontributors can access the portal by web browser. The web portalsupports management of following knowledge types: clinical prob-lems (Fig. 4A and B), care protocols (Fig. 4C), literature, pharmacy,and hardcoded knowledge. Hardcoded knowledge includes externalcomputer-aided tools, such as dosing advisors and risk evaluators,which can be registered to specific contexts in the clinical workflow.For example, they can be registered to specific medical orders in a careprotocol, as shown in Fig. 4D. When physicians use the care protocol,the corresponding tools will be automatically launched.

In addition to knowledge-authoring via the web portal, it is now atrend to automatically extract knowledge from narrative resources[35,36]. In this study, we designed an NLP system that depends on aChinese medical lexicon. The lexicon is maintained by the knowledge-authoring web portal (Fig. 5). With the help of this NLP system, wehave extracted 39,324 drug-ADR associations from 3683 Chinese drugfact sheets [37]. The details of this study are provided in Supplement-I.

2.3. Knowledge base

For constructing a knowledge base, this paper employed the EntityFramework [38] to generate a knowledge base schema from anontology model. Entity Framework is Microsoft's implementationof ORM (object relation mapping) [39] technology, which enablesmapping and conversion between an object-oriented model and arelational database schema. We used the visual model editor provi-ded by Entity Framework to construct the unified ontology. Then, the

include

testrequest

drug userule

checks

dosage

route

allergy

constrain

frequency

constrain

constrainconstrain

include

is a

Human-readableKnowledge

is a

systemreview

meta-analysis

is aclinicalpracticeguideline

DomainKnowledge

is a

Machine-doable

Knowledge

reference

is a

clinicalproblem

diagnosticrule

has

contextitem

has

is a

lab testresult

is a

vital sign

instance offact

has

protocol is aorder set

phase

medicalorder

has

has

is aclinicalpathway

drug factsheet

is ais a

symptom

is a

is a

medication

3rd-party /legacy tools

dosingadvisor

evaluationtools

formularysupport

is a

is a

is ais a

other externalapplications

FormallyRepresentedKnowledge

is a

HardcodedKnowledge

is ais a

is a supplem

ent

Fig. 3. A unified ontology that contains various types of knowledge.

Y. Zhang et al. / Computers in Biology and Medicine 60 (2015) 40–50 43

corresponding SQL server database schema was automatically gener-ated from the ontology. We use this database as the persistent storagefor our knowledge base. The ontology model and database schema areprovided in Supplement-V.

Until now, through the above knowledge acquisition approaches,27,960 clinical problems from ICD-10 (Chinese version), 463 standardcare protocols created according to clinical practice guidelines andChina MOH (Ministry of Health) clinical pathways [40], 5378 medicalarticles from public resources such as China Guideline Clearinghouse,1986 drugs from the hospital's pharmacy system, 490 DDIs (drug-druginteractions) from related work [41], 161 drug DDDs (defined daily

doses) from WHO ATC/DDD Index, and 39,324 drug-ADR associationsfrom drug labels have been maintained in the knowledge base. Inaddition, 11 computer-aided tools were collected and uploaded,including dosing advisor, formulary support, risk evaluator, BMI (bodymass index) calculator, BSA (body surface area) calculator, and CrCl(creatinine clearance) calculator.

2.4. Knowledge driven services

To apply the knowledge content in computerized decision supportefficiently, a “knowledge execution environment” is needed to actuate

Fig. 4. Knowledge-authoring web portal. The web portal supports creating various knowledge types, such as (A) clinical problems, (B) diagnostic rules, and (C) careprotocols. D is a dialog of computer-aided tools that can be registered to a specified medical order in the care protocol.

Y. Zhang et al. / Computers in Biology and Medicine 60 (2015) 40–5044

the knowledge content in the knowledge base (Fig. 1). As differentknowledge types in the unified knowledge ontology use differentrepresentation formulisms, such as the rule syntax used by diagnosticrules and the multi-phase data structure used by care protocols, it isnecessary to provide corresponding reasoning techniques or proces-sing methods [42]. As a consequence, the following services areprovided to actuate the knowledge content.

2.4.1. Data acquisitionIn clinical settings, both structured data and narrative text exist.

A task-specific NLP framework was proposed to extract structureddata from narrative text (Fig. 6). The framework allows a user todefine specific NLP tasks for different data requirements. For eachNLP task, the user can subset the homegrown domain ontology toform a task-specific lexicon composed of only task-relevant conceptsand relationships. The user can also specify a targeted documenttype and NLP algorithms for the task. At runtime, the integrationengine acts as a task monitor and notifies the NLP system when aclinical document or report was generated in the clinical setting.Then the corresponding NLP task will extract structured conceptsand values to drive the subsequent services. A related case studyusing this NLP framework is provided in Supplement-II.

2.4.2. Inference engineA rule-based inference engine has been developed based on

ANTLR [43] (ANother Tool for Language Recognition). The enginesupports diagnostic rules, such as “If ([Glucose]47.0 || [OGTT120']411.1) then [Diabetes]”. To help clinicians edit rules, a web-based GUI

rule editor (Fig. 4 B) is provided in the knowledge-authoring webportal. In this specific implementation, the inference engine dependson a shared domain ontology between external data providers, suchas CDR and NLP. This partly solves the difficulty of mapping externalclinical data to referenced variables in rule expression, which is acommon problem faced by existing rule engines [44]. The rule engineis published as open source and can be found on the project homepage. Meanwhile, we are considering using Drools [45] as a morepromising rule engine for future requirements, which supportsfeatures such as rule chaining, decision table, DSL (domain specificlanguage) extension and temporal reasoning.

2.4.3. Treatment recommendationCare protocols, such as order set and clinical pathway, are

evidence-based therapeutic treatment plans. Collecting, organizing,and distribution of these interventions are important for the mean-ingful use of CDS [46]. Experts from different specialties can use theknowledge-authoring web portal to define care protocols (Fig. 4C),and associate them with corresponding clinical problems. Suchprotocols can be recommended to physicians when specific clinicalproblems are detected (Fig. 7G). The knowledge-authoring web portaluses the same medical order dictionaries of EMRS, so the created careprotocols can generate native medical orders in CPOE (computerizedphysician order entry) without term conversion.

2.4.4. Context-aware knowledge retrievalFor context-aware retrieval of human-readable literature, both

the local knowledge base and external knowledge sources are

Fig. 5. Editing concepts and relationships in the Chinese language lexicon. In the editing area, a concept was defined by its name, semantic type, coding, value type andclinical significance. Each concept was allowed to be assigned with several terms, which could be synonyms, abbreviations, and frequent typos. A pop-up dialog box thatprovided semantic relations defined in UMLS was used to construct relationships around this concept. The concepts and relationships also constitute a shared domainontology among different systems, such as the inference engine and data acquisition service.

Y. Zhang et al. / Computers in Biology and Medicine 60 (2015) 40–50 45

used. The local knowledge base contains 5378 articles uploadedthrough the knowledge-authoring web portal, including 2352drug labels and 2822 clinical practice guidelines collected fromthe Internet, 126 MOH-published clinical pathways [40], and 78drug adverse event reports from the SFDA (China State Food andDrug Administration) [47]. Such local literature can be tagged tomachine-doable knowledge, e.g., diagnostic rules and drug userules. In addition, the integration of external reference sources issupported, including both HL7 Infobutton standard [48] compliantsources (such as BMJ Best Practices, MedlinePlus, UpToDate, MerckManual and Micromedex) and non-HL7 compliant knowledgesources (such as Cochrane Reviews, FDA DailyMed, PubMed, ChinaGuideline Clearinghouse [49] and China Cochrane/Evidence BasedMedicine Center [50]). To handle various external knowledgesources, an Infobutton manager was developed to translate anInfobutton query into a resource-specific SOA or URL request. Untilnow, 6 HL7 compliant sources and 22 non-HL7 compliant sourceshave been integrated. We keep the integration of external knowl-edge sources as an important feature of this platform because webelieve shareable and public knowledge bases will prevail in thenear future. The knowledge sources are managed through theknowledge-authoring web portal (⟨http://www.cktp.org:8006/KnowledgeSource⟩).

2.4.5. Application storeLegacy decision support systems and various third-party applica-

tions have become commonplace within the medical field as bothpersonal and professional tools [51]. Typical applications includedosing advisor, formulary support and risk evaluator. These toolsassist physicians with decision making in daily activities. This platform

designed an application store service that allows users to upload,manage and register (Fig. 4D) such tools through the knowledge-authoring web portal. In runtime, the application store will pushcorresponding tools to the specific context and execute them througha plug-in mechanism [32].

2.5. Clinical decision support

The above knowledge driven services can be used as buildingblocks for CDS applications. Based on these services, clinical decisionsupport applications (Fig. 7) have been developed to effectivelydisseminate the knowledge content to influence healthcare decisionmaking. Two approaches are used to disseminate diverse formattedknowledge to clinicians by computerized decision support:

The first one is the on-demand approach, in which users query ortrigger knowledge positively. All human-readable knowledge can bedisseminated in this way via context-aware knowledge retrieval (Cand D in Fig. 7) from both local and external knowledge sources.Similarly, hardcoded knowledge in the form of computer-aided toolscan also be downloaded and triggered by the user via the applicationstore service that plays the role of “Apple App Store” or “Android AppMarket” in EMRS. For example, E in Fig. 7 shows a micropump speedcalculator invoked by the user.

The other is a proactive approach in which data drives theknowledge dissemination. The formally represented knowledge suchas diagnostic rules and drug use rules has the potential to beimplemented in this way. Notified by clinical events and newlygenerated data, the inference engine fetches and executes relatedrules from the knowledge base. Then the inference results are pushedto users as alerts (A and B in Fig. 7) or clinical problems (F in Fig. 7).Associated care protocols (G in Fig. 7) are also provided as actionable

Fig. 6. Task specific NLP framework. The user defines NLP tasks based on diverse CDS data requirements and enrolls new concepts to complement the homegrown domainontology. In each NLP task instance, targeted concepts will be extracted from specific documents and fed to CDS tasks.

Y. Zhang et al. / Computers in Biology and Medicine 60 (2015) 40–5046

interventions that can be directly applied in the order entry work-bench. One challenge for this approach is that much of the informa-tion needed to actuate the rules is recorded in narrative text. Theaforementioned task-specific NLP framework (Fig. 6) was designed toaddress this challenge. With this NLP framework, several free-textdriven CDS applications have been developed, such as extractingmedication and adverse events from clinical documents for ADE(adverse drug event) detection and extracting fetal biometry mea-surements (e.g., biparietal diameter and femur length) from preg-nancy ultrasound reports (refer to Supplement-II).

3. System implementation and case study

3.1. System implementation

According to the above framework, we initiated a knowledgetranslation platform (KTP) project in DaYi Hospital. KTP wasdeployed in the hospital in September 2013. As seen fromTable 1, KTP has been extensively used through various CDSapplications in clinical settings. During the period from 2013/09/01 to 2014/10/01, KTP processed 1,345,364 clinical events notifiedby the integration engine, and retrieved 2,427,133 facts using thedata acquisition service. These events and acquired clinical datawere further used by the inference engine to drive diagnostic CDS,

drug use check CDS, and free-text driven CDS. Infobutton isanother frequently used CDS application.

Among the above KTP-based clinical decision support applica-tions, some (e.g., drug use check) are able to produce observablephysician behavior changes, while others (e.g., Infobutton) help toalter awareness, attitudes, and knowledge, but did not necessarilyinfluence behavior. Considering the variety of knowledge typesand decision supports covered by KTP, it is not easy to evaluate theentire platform as a whole. However, we are able to do evaluationby specific use cases. The system log shows that drug use check isfrequently used by clinicians and can be analyzed as a typical casestudy of the platform.

3.2. Case study

The drug use check CDS provides alerts in two modes: asynchro-nous notification (A in Fig. 7) and synchronous message (B in Fig. 7).In the current phase, synchronous mode was implemented, and thebasic rules were supported, including dosage, administration route,frequency and skin test. During the period from 2013/12/25 to 2014/10/01, 9596 alerts were detected from 2,667,016 medical orders(1,113,705 were medication orders) by the drug use check.

Among the 9596 alerts generated, we need to find out howmany medication orders were actually corrected. A corrected orderis a direct proof that a physician's behavior was influenced by the

Fig. 7. CDS applications in CPOE. (A) Notification area for clinical events, such as drug alert and ADT (admit, discharge, transfer); (B) drug alert message; (C) Infobutton fordrug; (D) retrieved drug label by Infobutton; (E) micropump speed calculator as a hardcoded knowledge application; (F) clinical problems generated by diagnostic CDS;(G) recommended care protocols. This screen cut was produced by the Chinese CPOE system in DaYi hospital. English readers can refer to Supplement-VI for a more detaileddescription.

Y. Zhang et al. / Computers in Biology and Medicine 60 (2015) 40–50 47

CDS intervention. To measure how many alerts were actuallycorrected, we carried out a retrospective analysis that appliedthe same rule set to the 2,667,016 medical orders. In this analysis,if an alert was overridden by the physician and the correspondingmedication order was not modified, the same drug alert can stillbe detected using the same rules. In the end, we found that 987alerts remained. That means that 8609 (9596-987) alerts werecorrected by physicians. The 89.7% (8609/9596) acceptance rateindicates that the CDS intervention provided by KTP significantlyinfluenced clinicians' behavior, i.e., led to the modification ofmedical orders. Because these CDS interventions are backed bydrug rules that are constantly updated and curated by pharma-cists, such intervention acts as an efficient knowledge tool thatdelivers the latest maintained evidence. In this sense, KTP hassuccessfully assisted in translating evidence-based rules intoobservable clinician behavior changes. This demonstrates thecapability of KTP in facilitating knowledge translation throughevidence-based decision support tools. The detailed report anddata set of this case study can be found in Supplement-IV.

4. Discussion

The primary contribution of this study is a computer-aided knowl-edge translation framework that acts as a CDS mobilization infrastruc-ture. The framework can also be extended to more hospitals andscenarios. Many modules and knowledge assets in the framework canbe customized according to different local requirements. The followingare some examples: 1. The current inference engine can be replaced orextended by other engines such as Drools. 2. Through a customizablelexicon and algorithms, the NLP system (Fig. 6) is expected to handlevarious situations, i.e., different domains or languages. A secondarycontribution is a scalable and extensible knowledge managementmechanism. With the knowledge-authoring tool, the knowledge con-tent can be periodically updated to include the latest evidence andtailored to local settings, which makes dependent decision supportflexible and adaptive. Through this mechanism, clinicians are bothauthors and users of the knowledge content. They can form anautonomous and collaborative community where knowledge contentis continuously improved by a “create, use, evaluate, revise” loop. Athird contribution is the integration andmanagement of the hardcodedknowledge by the application store service. This enhances the reusa-bility of legacy and external tools. By the knowledge-authoring webportal, users can upload various computer-aided tools into the knowl-edge base and register them for specific contexts. In this way, suchhardcoded knowledge is integrated into the clinical workflow and canbe invoked in a context-aware style. The inclusion of hardcodedknowledge into the unified ontology also extended the knowledgerepresentation (KR) capability. As discussed in the article “Why we need

many knowledge representation formalisms” [42], different KRs servespecific needs and inference methods, therefore enabling more power-ful artificial intelligence. The inclusion of hardcoded knowledge fulfillsthe necessity of the coexistence of various KR methods.

One point that readers should bear in mind is that technology“assists” but will not “substitute” for humans [52]. The knowledgetranslation platform provides evidence-based CDS in EMRS to assistphysicians in making better decisions. CDS interventions (such asdrug alerts) are informative, rather than mandatory. Moreover, theplatform supports the co-delivery of relevant human-readable litera-ture with CDS interventions, so that users have more references forconsideration at the point of care. This also helps to prevent over-reliance on the system.

There are several limitations to this study. First, a more systematicevaluation methodology is needed for this knowledge translationplatform. One challenge in evaluation is that for certain scenarios it isdifficult to determine whether user altered their behavior solelybecause of the provided CDS intervention. For example, after a newclinical problem is detected, the physician may take action, such ascreating new medical orders or revising existing medical orders.However, there is no straightforward means to measure to whatdegree the actionwas influenced by CDS. To address this problem, weare considering several evaluation approaches, such as an observa-tional method [53] (i.e., when a CDS alert is fired, if such correspond-ing behavioral changes are observed, knowledge translation isregarded as successful), KPI (key performance indicator) comparisons,and questionnaires. Second, in the current implementation of KTP,lack of standardization can be an obstacle for sharing knowledgecontent among different healthcare organizations. For example, thereferenced variables in diagnostic rules and the medical ordersdefined in care protocols are based on the hospital's private terminol-ogies. Although using institution-specific terminologies saves theefforts of data mapping between KTP and external systems, it makesthe knowledge content difficult to be used and shared across differenthospitals. A viable solution may be using standard terminologies forknowledge-authoring, and publishing the knowledge content as anopen resource. Each hospital can use a terminology service to map thestandardized knowledge content to their local terminologies.

In the end, readers should be aware that knowledge translation (KT)is not merely a technical issue. The KTP proposed in this paper is notintended to be an “all-around” solution. As mentioned in [54], KT is acomplex process that involves physicians' cognitive and behavioralchanges. It is influenced by various psychological, social, economic,legislative and regulatory factors. Because KT is such a complicated andbroad subject, the topic covered by this study is just “the tip of theiceberg”, and more research must be carried out in the future, such asintegrating more types of knowledge. In recent years, genomicsknowledge and high-throughput testing technology have graduallybeen applied in clinical practice [55]. This type of knowledge is also

Table 1Log data collected from the knowledge translation platform.

Log items Period Count Description

Inpatient visits 2013/09/01–2014/10/01

27,179 Inpatient visits enrolled by the new EMRS that is supported by KTP

Clinical events notified by theintegration engine

2013/09/01–014/10/01

1,345,364 Including 8 event types, such as lab test (trigger diagnostic CDS), medical order submission (trigger drugcheck use), and clinical document (trigger free-text driven CDS), etc

Facts retrieved by the dataacquisition service

2013/09/01–2014/10/01

2,427,133 Facts retrieved from clinical events by the data acquisition service

Clinical problems generated bythe inference engine

2013/09/01–2014/10/01

1621 In this period, 20 diagnostic rules are defined

Drug alerts generated by theinference engine

2013/12/25–2014/10/01

9596 In this period, the basic rules such as dosage, administration route, frequency and skin test are supported

Infobutton queries 2013/09/01–2014/10/01

23,883 User queries for drug labels and clinical practice guidelines through the Infobutton

nMore details on the usage data are provided in Supplement-III.

Y. Zhang et al. / Computers in Biology and Medicine 60 (2015) 40–5048

faced with the urgent pressure for knowledge translation [56,57]. Theinclusion of such knowledge can further extend the capability of KTP inguiding personalized diagnosis and drug use.

5. Conclusion

Translating knowledge into clinical practice has always been anurgent need for healthcare organizations. In this paper, we report oureffort to design a computer-aided knowledge translation frameworkthat uses computerized decision support as a knowledge tool toimprove evidence-based practice. This report describes a pioneerstudy in China, and we hope it can promote and inspire relatedresearch in more hospitals.

Conflict of interest

None declared.

Acknowledgment

This study was supported in part by the National High-techR&D Program of China (2012AA02A601) and the National NaturalScience Foundation of China (30900329). Sponsors have no role inthis study that could have influenced its outcome.

Many thanks to the journal editor and reviewers who haveprovided us with valuable advice and helped us to substantiallyimprove the manuscript.

Appendix A. Supporting information

Supplementary data associated with this article can be found in theonline version at http://dx.doi.org/10.1016/j.compbiomed.2015.02.013.

References

[1] J.E. Wennberg, A.M. Gittelsohn, Small area variations in health care delivery,Science 182 (1973) 1102–1108.

[2] E.A. McGlynn, S.M. Asch, J. Adams, J. Keesey, J. Hicks, A. DeCristofaro, E.A. Kerr,The quality of health care delivered to adults in the United States, N. Engl. J.Med. 348 (2003) 2635–2645.

[3] R. Grol, Successes and failures in the implementation of evidence-basedguidelines for clinical practice, Med. Care 39 (2001) II-46–II-54.

[4] O. Hirakis, S. Karakounos, Goals and benefits of knowledge management inhealthcare, in: A.A. Lazakidou (Ed.), Handbook of Research on Informatics inHealthcare and Biomedicine, Idea Group Inc., Hershey, Pennsylvania, US, 2006,pp. 193–200.

[5] T. Hongsermeier, Clinical Informatics Opportunities in Healthcare and Lifescie-nces 2008. Available: ⟨http://www.healthcare-informatics.com/sites/healthcare-informatics.com/files/archive/www.healthcare-informatics.com/Media/BlogTopics/2008%20Hongsermeier%20-%20Medical%20Informatics%20Careers%20SEAK.pdf⟩.

[6] P.G. Shekelle, E. Ortiz, S. Rhodes, S.C. Morton, M.P. Eccles, J.M. Grimshaw,S.H. Woolf, Validity of the Agency For Healthcare Research and Quality clinicalpractice guidelines: how quickly do guidelines become outdated? JAMA 286(2001) 1461–1467.

[7] E.M. Antman, J. Lau, B. Kupelnick, F. Mosteller, T.C. Chalmers, A comparison ofresults of meta-analyses of randomized control trials and recommendations ofclinical experts: treatments for myocardial infarction, JAMA 268 (1992)240–248.

[8] T. Thompson, Health Care Information Technology, 2003. Available: ⟨http://aspe.hhs.gov/sp/nhii/Conference03/SpeechText.htm⟩.

[9] J. Tetroe, Knowledge translation at the Canadian Institutes of Health Research:a primer, Focus Technical Brief, 18, 2007, 1–8.

[10] R.B. Haynes, Of studies, syntheses, synopses, summaries, and systems: the 5Sevolution of information services for evidence-based healthcare decisions,Evid. Based Med. 11 (2006) 162–164.

[11] M. Stefanelli, Knowledge and process management in health care organiza-tions, Methods Inf. Med. 43 (2004) 525–535.

[12] G. Schreiber, Knowledge engineering and management: the CommonKADSmethodology, MIT Press, Cambridge, Massachusetts, USA, 2000.

[13] T.J. Bright, A. Wong, R. Dhurjati, E. Bristow, L. Bastian, R.R. Coeytaux, G. Samsa,V. Hasselblad, J.W. Williams, M.D. Musty, Effect of clinical decision-supportsystemsa systematic review, Ann. Intern. Med. 157 (2012) 29–43.

[14] M.W. Jaspers, M. Smeulers, H. Vermeulen, L.W. Peute, Effects of clinicaldecision-support systems on practitioner performance and patient outcomes:a synthesis of high-quality systematic review findings, J. Am. Med. Inform.Assoc. 18 (2011) 327–334.

[15] M.E. Johnston, K.B. Langton, R.B. Haynes, A. Mathieu, Effects of computer-based clinical decision support systems on clinician performance and patientoutcome: a critical appraisal of research, Ann. Intern. Med. 120 (1994)135–142.

[16] J. H. Weber-Jahnke,G. McCallum, A light-weight component for adding decisionsupport to electronic medical records, in: Proceedings of the 41st Annual HawaiiInternational Conference on System Sciences, 2008, pp. 251–251.

[17] D.W. Bates, G.J. Kuperman, S. Wang, T. Gandhi, A. Kittler, L. Volk, C. Spurr,R. Khorasani, M. Tanasijevic, B. Middleton, Ten commandments for effectiveclinical decision support: making the practice of evidence-based medicine areality, J. Am. Med.l Inform. Assoc. 10 (2003) 523–530.

[18] D. Bates, M. Cohen, L. Leape, J.M. Overhage, M.M. Shabot, T. Sheridan, Reducingthe frequency of errors in medicine using information technology, J. Am. Med.Inform. Assoc. 8 (2001) 299–308.

[19] B. Middleton, K. Renner, M. Leavitt, Ambulatory practice clinical informationmanagement: problems and prospects, Healthc. Inf. Manag.: J. Healthc. Inf.Manag. Syst. Soc. Am. Hosp. Assoc. 11 (1997) 97.

[20] P.L. Elkin, M. Liebow, B.A. Bauer, S. Chaliki, D. Wahner-Roedler, J. Bundrick,M. Lee, S.H. Brown, D. Froehling, K. Bailey, The introduction of a diagnosticdecision support system (DXplain™) into the workflow of a teaching hospitalservice can decrease the cost of service for diagnostically challengingDiagnostic Related Groups (DRGs), Int. J. Med. Inform. 79 (2010) 772–777.

[21] ICNet International. ABX Alert. Available: ⟨http://icnetplc.com/en-gb/products/abx-alert⟩.

[22] E.G. Poon, S.J. Wang, T.K. Gandhi, D.W. Bates, G.J. Kuperman, Design andimplementation of a comprehensive outpatient results manager, J. Biomed.Inform. 36 (2003) 80.

[23] G.J. Kuperman, A. Bobb, T.H. Payne, A.J. Avery, T.K. Gandhi, G. Burns,D.C. Classen, D.W. Bates, Medication-related clinical decision support incomputerized provider order entry systems: a review, J. Am. Med. Inform.Assoc. 14 (2007) 29–40.

[24] R. Bose, Knowledge management-enabled health care management systems:capabilities, infrastructure, and decision-support, Expert Syst. Appl. 24 (2003)59–71.

[25] W. Sujansky, Heterogeneous database integration in biomedicine, J. Biomed.Inform. 34 (2001) 285–298.

[26] I. Sim, P. Gorman, R.A. Greenes, R.B. Haynes, B. Kaplan, H. Lehmann, P.C. Tang,Clinical decision support systems for the practice of evidence-based medicine,J. Am. Med. Inform. Assoc. 8 (2001) 527–534.

[27] M. Peleg, S. Tu, Decision support, knowledge representation and managementin medicine, Yearb. Med. Inform. 45 (2006) 72–80.

[28] A.A. Boxwala, B.H. Rocha, S. Maviglia, V. Kashyap, S. Meltzer, J. Kim,R. Tsurikova, A. Wright, M.D. Paterno, A. Fairbanks, A multi-layered frameworkfor disseminating knowledge for computer-based decision support, J. Am.Med. Inform. Assoc. 18 (2011) i132–i139.

[29] A.A. Boxwala, S. Tu, M. Peleg, Q. Zeng, O. Ogunyemi, R.A. Greenes,E.H. Shortliffe, V.L. Patel, Toward a representation format for sharable clinicalguidelines, J. Biomed. Inform. 34 (2001) 157–169.

[30] T.S. Field, P. Rochon, M. Lee, L. Gavendo, S. Subramanian, S. Hoover, J. Baril,J. Gurwitz, Costs associated with developing and implementing a computer-ized clinical decision support system for medication dosing for patients withrenal insufficiency in the long-term care setting, J. Am. Med. Inform. Assoc. 15(2008) 466–472.

[31] Z. Chenhui, D. Huilong, L. Xudong, An integration approach of healthcareinformation system, 2008, 606–609.

[32] Y. Zhang, Z. Li, H. Li, X. Zheng, X. Lu, H. Duan, A knowledge representation andtranslation method for clinical protocols, Chin. J. Biomed. Eng. 33 (2) (2014)139–147.

[33] C. Weng, S.W. Tu, I. Sim, R. Richesson, Formal representations of eligibilitycriteria: a literature review, J. Biomed. Inform. 43 (2010) 451.

[34] A. L. Russ, A. J. Zillich, M. S. McManus, B. N. Doebbeling,J. J. Saleem, A humanfactors investigation of medication alerts: barriers to prescriber decision-making and clinical workflow, in: Proceedings of the AMIA Annual Sympo-sium, 2009, 548.

[35] S. L. Achour, M. Dojat, C. Rieux, P. Bierling,E. Lepage, Knowledge acquisitionenvironment for the design of a decision support system: application in bloodtransfusion, in: Proceedings of the AMIA Symposium, 187–191, 1999.

[36] S. Achour, M. Dojat, C. Rieux, P. Bierling, E. Lepage, A UMLS-based knowledgeacquisition tool for rule-based clinical decision support system development,J. Am. Med. Inform. Assoc. 8 (2001) 351–360.

[37] C. Ge, Y. Zhang, W. Chen, H. Duan and H. LI, Construction and statisticalanalysis of an ADR (Adverse Drug Reactions) knowledgebase, Chin. J. Hosp.Pharm. In press.

[38] A. Adya, J. A. Blakeley, S. Melnik,S. Muralidhar, Anatomy of the ado. net entityframework, in: Proceedings of the 2007 ACM SIGMOD International Con-ference on Management of Data, 2007, 877–888.

[39] H.Z.C. Zhigang, Research and Application about object/relation mappingframework, Comput. Eng. Appl. 26 (2003) 058.

Y. Zhang et al. / Computers in Biology and Medicine 60 (2015) 40–50 49

[40] China Ministry of Health. The Medical Administrative Department of theMinistry of Health, in: Proceedings of the Clinical Pathway Management WorkConference, 2012. Available: ⟨http://www.gov.cn/gzdt/2012-05/14/content_2136918.htm⟩.

[41] B.L. Hug, D.J. Witkowski, C.M. Sox, C. Yoon, M.E. Matheny, D.W. Bates, Adversedrug event rates in six community hospitals and the potential impact ofcomputerized physician order entry for prevention, J. Gen. Intern. Med. 25(2010) 31–38.

[42] A. Sloman, Why we need many knowledge representation formalisms, in:Proceedings BCS Expert Systems Conf, (1985), pp. 163-183.

[43] T.J. Parr, R.W. Quong, ANTLR: a predicated-LL (k) parser generator, Softw.:Pract. Exp. 25 (1995) 789–810.

[44] C.Y. Jung, K.A. Sward, P.J. Haug, Executing medical logic modules expressed inArdenML using Drools, J. Am. Med. Inform. Assoc. 19 (2012) 533–536.

[45] P. Browne, JBoss Drools business rules, Packt Publishing Ltd., Birmingham, UK,2009.

[46] J. Osheroff, J. Teich, B. Middleton, A roadmap for national action on clinicaldecision support, J. Am. Med. Inform. Assoc 14 (2) (2007) 141–145.

[47] SFDA. China State Food and Drug Administration. Available: ⟨http://www.sda.gov.cn/WS01/CL0001/⟩.

[48] G. Del Fiol, V. Huser, H.R. Strasberg, S.M. Maviglia, C. Curtis, J.J. Cimino,Implementations of the HL7 context-aware knowledge retrieval (infobutton)standard: challenges, strengths, limitations, and uptake, J. Biomed. Inform. 45(2012) 726–735.

[49] Evidence-Based Medicine Committee of China Medical Association. ChinaGuideline Clearinghouse. Available: ⟨http://cgc.bjmu.edu.cn:820⟩.

[50] Chinese Journal of Evidence-Based Medicine. China Cochrane/Evidence BasedMedicine Center. Available: ⟨http://www.cjebm.org.cn/docs/yx/details.aspx?documentid=101⟩.

[51] O.I. Franko, T.F. Tirrell, Smartphone app use among medical providers inACGME training programs, J. Med. Syst. 36 (2012) 3135–3139.

[52] C.P. Friedman, A fundamental theorem of biomedical informatics, J. Am. Med.Inform. Assoc. 16 (2009) 169–170.

[53] Centers for Disease Control and Prevention, Data Collection Methods forProgram Evaluation: Observation, 2008. Available: ⟨http://www.cdc.gov/healthyyouth/evaluation/pdf/brief16.pdf⟩.

[54] J. Lomas, Diffusion, dissemination, and implementation: who should do what?Ann. NY Acad. Sci. 703 (2006) 226–237.

[55] X. Li, R.J. Quigg, J. Zhou, W. Gu, P.N. Rao, E.F. Reed, Clinical utility ofmicroarrays: current status, existing challenges and future outlook, Curr.Genomics 9 (2008) 466.

[56] M.T. Scheuner, P. Sieverding, P.G. Shekelle, Delivery of genomic medicine forcommon chronic adult diseases: a systematic review, JAMA 299 (2008)1320–1334.

[57] F.W. Frueh, D. Gurwitz, From pharmacogenetics to personalized medicine: avital need for educating health professionals and the community, Pharmaco-genomics 5 (2004) 571–579.

Y. Zhang et al. / Computers in Biology and Medicine 60 (2015) 40–5050