alightweightapi-basedapproachforbuildingflexible...

Research ArticleA Lightweight API-Based Approach for Building FlexibleClinical NLP Systems

Zhengru Shen Hugo van Krimpen and Marco Spruit

Department of Computing and Information Sciences Utrecht University Utrecht Netherlands

Correspondence should be addressed to Zhengru Shen zshenuunl

Received 18 February 2019 Revised 20 June 2019 Accepted 26 July 2019 Published 15 August 2019

Academic Editor Haihong Zhang

Copyright copy 2019 Zhengru Shen et al)is is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

Natural language processing (NLP) has become essential for secondary use of clinical data Over the last two decades manyclinical NLP systems were developed in both academia and industry However nearly all existing systems are restricted to specificclinical settings mainly because they were developed for and tested with specific datasets and they often fail to scale up)ereforeusing existing NLP systems for onersquos own clinical purposes requires substantial resources and long-term time commitments forcustomization and testing Moreover the maintenance is also troublesome and time-consuming )is research presents alightweight approach for building clinical NLP systems with limited resources Following the design science research approach wepropose a lightweight architecture which is designed to be composable extensible and configurable It takes NLP as an externalcomponent which can be accessed independently and orchestrated in a pipeline via web APIs To validate its feasibility wedeveloped a web-based prototype for clinical concept extraction with six well-known NLP APIs and evaluated it on three clinicaldatasets In comparison with available benchmarks for the datasets three high F1 scores (0861 0724 and 0805) were obtainedfrom the evaluation It also gained a low F1 score (0373) on one of the tests which probably is due to the small size of the testdataset )e development and evaluation of the prototype demonstrates that our approach has a great potential for buildingeffective clinical NLP systems with limited resources

1 Introduction

Todayrsquos technologies allow the accumulation of vast textualdata which consequently has boosted the popularity of NLPresearch )ere has been a huge amount of papers publishedand a variety of NLP systems or toolkits crafted in multipledomains over the last two decades Among them clinicalNLP occupies a large portion )ere are clinical NLP sys-tems such as Apache cTAKES that integrate different NLPtools to process clinical documents [1 2])ere are also NLPtools which target certain specific clinical needs includingextracting medication information [3] identifying locationsof pulmonary embolism from radiology reports [4] andcategorizing pain status [5]

Figure 1 presents a general architecture of a clinical NLPsystem that contains two main components backgroundknowledge and framework [1] Background knowledgecontains ontologies domain models domain knowledgeand trained corpora )e widely used clinical domain

knowledge is the Unified Medical Language System (UMLS)[6] Framework refers to a software platform that integratesvarious NLP tasks or modules either sequentially or hier-archically into NLP pipelines GATE and UIMA are theleading open-source frameworks [7 8] )ere are two levelsof NLP tasks low-level tasks and high-level tasks Low-leveltasks include tokenization part of speech tagging sentenceboundary detection and so on High-level tasks refer to thesemantic level processing such as named entity recognitionrelation extraction and sentiment analysis

History has shown that building a successful clinical NLPsystem requires a tremendous amount of resources Forinstance it took a team from Columbia University 14 yearsto commercialize the MedLEE system [9] )e developmentof cTAKES started at the Mayo Clinic in 2006 and furtherexternal collaborations with four other universities in 2010resulted in the first release of the current Apache project [2])erefore creating reusable NLP pipelines based on open-source modular frameworks like GATE and UIMA becomes

HindawiJournal of Healthcare EngineeringVolume 2019 Article ID 3435609 11 pageshttpsdoiorg10115520193435609

more reasonable [9 10] Although it dramatically reducesresources and level of expertise we argue that it is not anefficient and effective solution for two main reasons Firstlynearly every NLP pipeline that is created to address a singlespecific clinical need either rule or machine-learning basedhas been proven to be useful for only its designated purposes[11] )us reusability is difficult given the properties Sec-ondly deploying cTAKES-based NLP pipelines implies ahigh cost of operation which requires installation andconfiguration of multiple components by NLP experts [12]Besides maintenance of a deployed NLP system requires acontinuous investment

With the purpose of simplifying and outsourcing theNLP implementation software as a service or SaaS has beenintroduced to the NLP world during recent years [13] SaaSgenerally refers to the mode of software delivery where end-users are charged with a monthly or annual subscription feeto utilize a set of functionalities over the Internet [14] NLPsystems distributed in the SaaS model are often availablethrough web application programming interfaces (APIs)and named as NLP APIs or cloud-based NLP APIs [13 15]Many NLP APIs have emerged from both companies anduniversities and are growing popularly [7] A few prominentexamples are IBMWatson Aylien Lexalytics and TextRazor[13] From the cost-benefit perspective these NLP APIsallow developers to rapidly create NLP-enabled toolswithout investing abundant resources on implementingnecessary NLP techniques in codes and on regular main-tenance A number of applications based on NLP APIs werebuilt [16ndash18]

To utilize NLP APIs API-based frameworks have beenproduced [1519ndash21] API-based systems also known ascloud-based refer to tools that are built on external webAPIs and having their functionalities partially or fully ac-complished with one or a pipeline of APIs Due to thegrowing popularity of web APIs in the software industryAPI-based tools are abundant in companies For instance anAPI-based CMS (content management system) is utilized tosave development resources and follow-up maintenance[22] Furthermore researchers have also investigated theapproach in recent years Rizzo and Troncy proposed theNamed Entity Recognition and Disambiguation (NERD)framework that incorporates the result of ten different publicAPIs-based NLP extractors [21] A web-based tool called

TeXTracT was devised to support the setup and deploymentof NLP techniques on demand [15] Abdallah et al de-veloped a flexible and extensible framework for integratingnamed entity recognition (NER) web APIs and assessed itacross multiple domains [19] Although these tools exhibitpromising results few were built for clinical NLP or eval-uated on clinical datasets )erefore it is safe to say thatadopting these tools in clinical settings would be problematicdue to the unique characteristics of the clinical domain Forexample privacy is considered to be of the utmost impor-tance but none of the above tools have taken it intoconsideration

)is paper thus proposes a lightweight framework whichenables a rapid development of clinical NLP systems withexternal NLP APIs )e approach has the following ad-vantages compared to traditional NLP frameworks (1) fastdevelopment (2) lower costs (3) flexibility and (4) pro-gramming language independent )e deployment is min-imized by outsourcing both NLP tasks and backgroundknowledge to external API services )us NLP systems canbe quickly and cost-efficiently developed based on theproposed framework )e framework is flexible in manyaspects To begin with it supports the flexible combinationof different NLP tasks from external APIs Secondly usershave the freedom of choosing their preferred NLP APIvendors and multiple APIs can be integrated to achievebetter results To evaluate the framework we have built aweb-based open-source clinical NLP application

2 Methods

21DesignScience Our research followed the design scienceresearch as we built and evaluated the framework because ofits strength and popularity in solving a real-world problemby designing and building an innovative IT artifact [23] Inour case the artifact is a lightweight framework that facil-itates clinical NLP systems development We follow thedesign science research methodology (DSRM) proposed byPeffers et al which consists of six steps problem identifi-cation and motivation definition of the objectives for asolution design and development demonstration evalua-tion and communication [24]

)e DSRM is initiated by the (I) problem identificationand motivation which we addressed by literature studyPrevious studies have described the general architecture ofclinical NLP systems and how expensive it is to build themEven though the introduction of modular NLP frameworksreduced the complexity of NLP systems it is still challengingto create clinical NLP systems for many healthcare in-stitutions due to limited resources Based on the identifiedproblem we inferred the (II) objectives for a solutioncreating a lightweight NLP framework that enables a rapiddevelopment of an API-based clinical NLP system In the(III) design and development we developed the frameworkbased on the general architecture we identified after whicheach of its components is explained in detail To (IV)demonstrate and (V) evaluate the framework a web-basedopen-source clinical NLP application was developedMoreover experiments were carried out with three clinical

Backgroundknowledge

Framework

Ontologies domain models domain knowledge trained corpora

NLP pipelines

Task 1 Task 2 Task N

Text Structured form

Figure 1 A general architecture of clinical NLP systems

2 Journal of Healthcare Engineering

datasets to primarily examine whether external NLP APIswould deliver the state-of-the-art performance )e finalstep of the DSRM is the communication )e paper serves asthe start of our (VI) communication on this topic

22 Evaluation Design )ree English anonymized clinicaldatasets were used in our evaluation Two of the datasets areobtained from the Informatics for Integrating Biology andthe Bedside (i2b2) center 2008 Obesity Challenge and 2009Medication Challenge )e third dataset comes from aEuropean clinical trial called OPERAM Since the primarygoal of our evaluation is to prove that external general-purpose NLP APIs can yield good performance on clinicaldata we only used a subset of the two large i2b2 datasets

(i) 2008 Obesity Challenge )is dataset consists of 611discharge letters All discharge letters are annotatedwith 16 different medical condition terms in thecontext of obesity including asthma gastroesoph-ageal disorder and depression Terms could beeither annotated as being in the document notbeing in the document or undecidedunknownwhich was treated as a not being in the document)e strength of this dataset concerning the aim ofthese tests is that there are a lot of documents snf itsweakness is that it is only annotated for 16 abstractterms in the context of obesity To simplify theexperiment we randomly selected 100 dischargeletters and labeled each document with the medicalconditions that are annotated as ldquopresentrdquo

(ii) 2009 Medication Challenge 947 out of 1243 in totaldeidentified discharge letters have the gold standardannotations Medication names in the annotationsare used for the evaluation By comparing the an-notated medication names with those generatedfrom our application we calculate the evaluationmetrics We also randomly select 100 out of the 947documents

(iii) OPERAM Dataset )e dataset consists of fivedischarge letters that have been used during thepilot of the OPERAM clinical trial [25] Medicalexperts of the trial annotated these letters by bothmedical conditions and pharmaceutical drugsMoreover standardized clinical codes for eachannotation are included With this dataset we aimto demonstrate the performance of our NLP ap-plication with clinical documents from practiceseven though it is clear that the small size limits ourfindings

We extracted entities of ldquomedical conditionrdquo or ldquophar-maceutical drugrdquo from the in total 205 clinical documentsand then encoded themwith UMLS Based on the encodingsextracted entities were filtered so that distinct entities wereextracted for each clinical document In order tomeasure theperformance of our extraction we have used well-knownmetrics precision recall and F1 score )ey are computedfrom true positives (TP) false positives (FP) and falsenegatives (FN) for each document As stated above

annotations of the 2008 Obesity Challenge are different fromthe other two datasets To simplify the identification ofpositives and negatives we divided annotations into twogroups positives that are in the text and negatives which arenot mentioned )erefore comparing clinical entitiesextracted by our application to the ground truth we cal-culate the following

(i) TP entities that were both extracted and annotatedas positives

(ii) FP entities that were extracted as positives but wereannotated as negatives

(iii) FN entities that were not extracted but were an-notated as positives

Precision (1) represents the proportion of extractedpositives that are annotated positives On the contrary recall(2) is the proportion of annotated positives that were cor-rectly extracted as such F1 score (3) is the harmonic mean ofprecision and recall

precision TP

TP + FP (1)

recall TP

TP + FN (2)

F1 score 2lowastprecisionlowast recallprecision + recall

(3)

3 Results

)e section presents results in two parts the framework anda web-based open-source clinical NLP application )earchitecture lays down the technical groundwork uponwhich the application was constructed )e following ex-plains each of them in details

31 A Lightweight NLP Architecture for Clinical NLP )earchitecture addresses the issues of existing clinical NLPapplications including interoperability flexibility andspecific restrictions within the clinical field such as privacyand security )e strength of our proposed architecture isshown in its capabilities (1) freedom of assembling suitableNLP APIs either sequentially or hierarchically based onscenarios (2) encoding clinical terms with comprehensiveand standardized clinical codes (3) the built-in deidentifi-cation function to anonymize clinical documents Figure 2depicts its four main components external APIs in-frastructure NLP pipelines and Apps

311 External APIs In this architecture two types of APIsnamely an NLP API and a domain knowledge API areincluded to parse unstructured medical text and map parsedterms against a medical metathesaurus respectively )eNLP API provides various cloud-based NLP services thatparse unstructured text for different purposes includingentity recognition and document classification )e domain

Journal of Healthcare Engineering 3

knowledge API supports the mapping of medical text toconcepts from the UMLS metathesaurus As the most usedbiomedical database UMLS contains millions of biomedicalconcept names and their relations In addition domainmodels and training corpora are available for specific clinicaldocuments such as radiology reports pathology reports anddischarge summaries [1] )e UMLS is a major part of thesolution for standardization and interoperability as it mapsterms extracted by multiple APIs to standardized codes suchas ATC and ICD10

312 Infrastructure )e infrastructure layer preparesclinical data before sending them to external APIs by dei-dentification and adding authentications Furthermore itprocesses results received from external APIs for later in-tegration An optional component locally implementedNLP techniques is also incorporated

(i) API Processing )e purposes of API processing aretwo-fold (1) prepare clinical text before sendingthem to external APIs and (2) process resultsreturned from external APIs Given the differencebetween multiple APIs data processing is inevitableto achieve interoperability Specific API processingtasks include formatting clinical text for APIs re-quests filtering results returned fromAPIs and dataconversion

(ii) Privacy Privacy protection is a critical issue inclinical data sharing for both research and clinicalpractices and privacy violations often incur legalproblems with substantial consequences )e pri-vacy component embedded in the infrastructureoffers technical solutions to deidentify or ano-nymize patient-level data such as CRATE [26] andDEDUCE [27] CRATE is an open-source softwaresystem that anonymizes an electronic health recordsdatabase to create a research database with ano-nymized patientsrsquo data With CRATE implementedour approach can directly use patientsrsquo data In

comparison with CRATE DEDUCE is morelightweight As a Python package it processessensitive patient information with commands likeldquodeducedeidentify_annotations()rdquo

(iii) Security )e security component controls the ac-cess of clinical data and all external APIs Au-thentication and encryption are added to safeguarddata sharing via the Internet

(iv) (Optional) Local NLP Tasks As discussed pre-viously an external NLP API grants no control ofwhat NLP techniques to employ In case somespecific NLP techniques are required our local NLPtechnique component provides a choice of imple-menting your own NLP techniques locally in apreferred language

313 NLP Pipelines )is layer provides a list of NLP ser-vices from which clinical applications can select the mostsuitable ones on demand First of all differences among NLPAPI providers in terms of their available NLP services areapparent However as shown in Table 1 there are also anumber of common NLP services Secondly systematicstudies have summarized some commonly used NLPtechniques in clinical NLP applications [11] By combiningthe common NLP services of various APIs and the usefulNLP techniques in clinical settings a shortlist of NLP ser-vices is selected for the architecture

Moreover multiple NLP services from different APIscan be integrated either sequentially or hierarchically for asingle clinical NLP task )is enables clinical NLP appli-cations to address the limitations of individual APIs causedby particular NLP techniques implemented and dataemployed to build it More importantly having a config-urable NLP pipeline brings scalability and flexibility Forinstance a clinical concepts extraction enabled applicationcan support combining entity extraction service from two ormore of the NLP APIs in Table 1 However interoperabilitybetween different NLP APIs becomes a challenge as both

Text

NLP pipelines

Apps

(i)(ii)

(iii)

Concept extractionSummarizationSentiment analysis

Structured form

External APIs UMLS API NLP APIs

API NAPI 2API 1

Locally implementedNLP tasks

Infrastructure

API processing

Privacy Security

Figure 2 A lightweight NLP architecture for clinical NLP


their inputs and outputs might vary considerably )ereforethe NLP pipelines contain an integration component whichfacilitates the interoperability by implementing a properintegration strategy

314 Apps In the application layer clinical NLP-enabledapplications for various needs can be created )ey areproduced either for performing a specific NLP task such asextracting diagnoses from discharge summaries and iden-tifying drugs and dosage information from medical recordsor with a general purpose of processing unstructured clinicaltext Existing NLP applications in clinical domains arecategorized into the following groups

(i) Concept Extraction Kreimeyer et al conducted asystematic literature review of NLP systems con-structed to extract terms from clinical documentsand map them to standardized clinical codes [11]

(ii) Text Classification Classification of free text inelectronic health record (EHR) has surfaced as apopular topic in clinical NLP research Koopmanet al devised a binary classifier to detect whether ornot death is related to cancer using free texts of

death certificates [28] Other text classification ex-amples in clinical settings cover classifying acomplete patient record with respect to its eligibilityfor a clinical trial [29] categorizing ICU riskstratification from nursing notes [30] assessinginpatient violence risk using routinely collectedclinical notes [31] and among others

(iii) Sentiment Analysis Unlocking the subjectivemeaning of clinical text is particularly helpful inpsychology A shared task for sentiment analysisof suicide notes was carried out as an i2b2challenge [32]

32 Prototype API-Based Clinical Concept Extraction Toevaluate the architecture a prototype that extracts clinicalconcepts from clinical free texts has been developed )issection first illustrates the design of its main components)en the prototype itself is presented

321 External NLP APIs As described above web NLPAPIs have gained wide popularity over the last few yearsBoth academics and companies recognized the importanceand extended their NLP systems with web APIs As shown inTable 2 the prototype incorporates six leading NLP APIsfrom both academia and industry in its implementation)eselection is based on three criteria (1) free or free trialavailable (2) industrial APIs supported by big companiesteams (3) academic APIs verified by peers

322 NLP Technique Implemented Locally Studies haverevealed that negation is very common in clinical reports[33 34] For instance ldquono fracturerdquo ldquopatient denies aheadacherdquo and ldquohe has no smoking historyrdquo often appear inclinical texts In order to correctly extract clinical termsnegation detection becomes inevitable However given thatmost of the selected NLP APIs are tools for text processingand analysis in the general domain the negation issue ofclinical documents is not properly tackled and they cannotfilter out irrelevant information )erefore negation de-tection is implemented locally for the prototype As the mostwell-known negation detection algorithm NegEx has beenadopted by a number of biomedical applications [35ndash37]We implemented the algorithm to handle negation in thisprototype

323 API Processing NLP APIs first extract clinical termswhich will be filtered by the local negator )en the UMLSAPI transforms the filtered clinical terms to the standardizedcodes such as ATC codes ICD-10 or SNOMED whichensures that the extracted clinical terms are interoperableafter integration

For each extracted term the UMLS API returns its top 10matched codes )ese top matches are ranked on theirsimilarity to the extracted term with the first as the mostsimilar one )e prototype captures the unique identifier ofeach matched code for later use

Table 1 NLP services of common NLP API providers

NLP API Available NLP services

IBM WatsonNLU

Entity extraction concept extraction relationextraction text classification language

detection and sentiment analysis

Aylien

Article extraction entity extraction conceptextraction summarization text classification

language detection semantic labelingsentiment analysis hashtag suggestion image

tagging and microformat extraction

LexalyticsSentiment analysis concept extraction

categorization named entity extraction themeextraction and summarization

Meaning Cloud

Topic extraction text classification sentimentanalysis language detection and linguistic

analysis (POS tagging parsing andlemmatization)

Alchemy API

Entity extraction concept tagging keywordsextraction relation extraction text

classification language detection sentimentanalysis microformat extraction feed

detection and linked data

TextRazorEntity extraction disambiguation linkingkeywords extraction topic tagging and

classification

Developer Cloud Concept extraction translation personalityinsights and classification

Open Calais Entity extraction relation extraction andsentiment analysis

Dandelion APIEntity extraction text classification language

detection sentiment analysis and textsimilarity

HavenOnDemand

Autocomplete concept extraction documentcategorization entity extraction languagedetection sentiment analysis and text

tokenization


As discussed above when multiple APIs are applied forone task results need to be integrated )e prototype em-ploys a double weight system to integrate multiple APIs )efirst weight system determines whether an extracted term issimilar to another extracted term from the same document)e weight of a pair of two extracted terms is calculatedbased on their top 10 matches from the UMLS API and thenis compared with the similarity threshold c if the weight ishigher than the threshold we consider it to be an equal term)e weight formula is shown as follows

α4

1113874 1113875 +3β4

1113888 1113889≻c (4)

where α refers to the percentage of equal terms over all 10terms and β is the percentage of equal terms over the top 3terms α and β are calculated based on the UMLS APImatches of two extracted terms )e weight is a value be-tween 0 and 1 0 being that the terms are not similar at all and1 being exactly the same For a given NLP task an initialvalue of c 01 is recommended and then according to thenumber of false positives and false negatives we adjust thevalue of c to achieve optimal output )e strategy of tuningthese parameters is discussed further in Section 33

)e second weight system determines whether anextracted clinical term has enough cumulative weight fromall NLP APIs Since the performance of NLP APIs varies aweight for each individual API is estimated by using the F1scores calculated after testing each API on a small subset ofclinical documents )e F1 score for each API is normalizedto an extractor-weight ω For each clinical term extracted wesum the weights of the extractors the term was extracted byIf the weight is over the extractor threshold θ it is consideredto be actually extracted If it is less it is considered to be afalse extraction )e weight is computed as follows

1113944n

i1ωi (5)

where ω is the weight of an NLP API and n refers to thenumber of API used )e pseudocode of the integrationprocess is shown in Algorithm 1

324 Prototype Figure 3 shows the overall functionalcomponents of the prototype which is an instantiation ofthe proposed architecture )e prototype is a web appli-cation with a minimalistic user interface developed withHTML5 CSS JavaScript and PHP for the back end Giventhat many existing NLP APIs use JSON as the default

format JSON is the chosen format for data transferringbetween different components Figure 4 presents ascreenshot of the application Users need to provideclinical documents they want to process in the upper inputfield and then select APIs and coding standards Afterclicking the Extract button the results will be displayed inthe table at the bottom ldquoDiseases Remoterdquo lists the ex-tractions of external NLP APIs while ldquoDiseases Localrdquorepresents results of combining external NLP APIs thelocal negation handler and the UMLS API Unfortunatelythe application is not accessible online due to a lack of APItoken management Sharing our tokens online might incura charge when there are a large number of API requestsNevertheless researchers are able to deploy their ownversion of the system with the source codes we share onGitHub at httpsgithubcomianshan0915MABNLP Ademo video is also available at httpsyoutubedGk9NQGWYfI

33 Evaluation Results As explained before the prototypecomes with three hyperparameters that adjust the extractionoutputs negation (κ) term similarity threshold (c) andextractor threshold (θ) )e hyperparameter tuning wasmanually conducted by the researchers in the experiments

)e impacts of the controlling hyperparameters on theoutputs of our experiments vary First of all negationsurprisingly shows little positive influence as shown inTable 3 Its main reason probably lies in the fact that theimplemented negation algorithm NegEx only uses negationcue words without considering the semantics of a sentence[34] Implementation of more advanced algorithms such asDEEPEN and ConText will be conducted in future research)e higher c value means a higher similarity threshold forentities to be merged which results in a lower false positiveand higher false negative numbers By increasing the θ valuewe want entities to be extracted by more APIs and sub-sequently lower the number of false positives and increasethe number of false negatives However higher values bringdown the number of true positives )e aim is to strive forthe best combination of these hyperparameters for eachspecific NLP task )e experiments suggested that the valuesof c 01 and θ 035 are a decent starting point for furtherexploration

Results have shown that the performance of the pro-totype is not consistent Datasets like the obesity challengecan rely on our approach but its reliability on datasets suchas the medication challenge and OPERAM dataset needfurther improvement and evaluation

Table 2 NLP APIs selected for the prototype

API Fee Companyteam References

IBM Watson NLU Free trial IBM httpswwwibmcomwatsondevelopercloudnatural-language-understandingapiv1

MeaningCloud Free trial MeaningCloud LLC httpswwwmeaningcloudcomdeveloperdocumentationOpen Calais Free trial )omson Reuters httpwwwopencalaiscomopencalais-apiHaven OnDemand Free trial Hewlett Packard httpsdevhavenondemandcomapisTextRazor Free trial TextRazor Ltd httpswwwtextrazorcomdocsrestDandelion API Free trial Spaziodati httpsdandelioneudocs


Many NLP systems have been tested on the two i2b2datasets and there are benchmark performance metricsbeing published in the literature [38 39] We calculated theaverages of top 5 best systems as the baselines As displayedin Table 4 the prototype performs well and has great po-tential of being adopted for clinical concept extraction Incase of the OPERAM dataset there is no benchmark

)erefore its performance is evaluated from an expert in-tervention perspective By comparing the automatedextracted clinical concepts with the annotations we estimatehow well the prototype can be used to assist physiciansduring their manual extraction process Unfortunatelyfeedbacks from physicians indicate that the prototype isnot yet considered practically useful Firstly its poor

Input X [X1 X2 Xn] returns of n APIsW [ω1 ω2 ωn] weights of n APIsc similarity thresholdθ extractor threshold

Output T a list of clinical termsInitialisation ωα 025 and ωβ 075Filter out samesimilar terms extracted by one API

(1) for i 1 to n do(2) for xa in Xi do(3) Get the rest of terms Xj Xi minus Xa(4) for xb in Xj do(5) calculate the percentage of equal terms over all 10 terms α(6) calculate the percentage of equal terms over top 3 terms β(7) calculate the pairwise similarity δ ωαlowastα + ωβlowastβ(8) if δge c then(9) discard samesimilar term Xi Xi minus Xb(10) end if(11) end for(12) end for(13) end for(14) Get filtered arrays of terms Xδ [X1δ X2δ Xnδ]

Filter out extracted terms by the weights over all APIs(15) Compute weights over all APIs Xω 1113936

ni1XδW

(16) for ωsum x in Xω do(17) if ωsumge θ then(18) Add the term the final list T+ [x](19) end if(20) end for(21) return T

ALGORITHM 1 Pseudocode of the API integration algorithm

Infrastructure

API integration

Domain knowledge UMLS API

JSON

JSON

JSON JSON

Extraction pipeline

Watson MeaningCloud TextRazor

CRATE HTTPSNegation handler

Documents Annotations

Concept extraction

PHP framework Laravel

Nodejs express server

Figure 3 Prototype architecture


performance in extracting medical conditions requiresphysicians to spend more time filtering out incorrect ex-tractions Secondly the prototype fails to identify the as-sociated dosages and frequencies of medications

4 Discussion

We argue that outsourcing NLP tasks offers efficient NLPsolutions for processing unstructured clinical documents Tobegin with outsourcing often leads to a reduction of both ITdevelopment and maintenance costs Furthermore a lowerlevel of NLP expertise is required when external NLP ser-vices are used A developer with limited knowledge of NLPcould develop a clinical NLP application such as our pro-totype Lastly the architecture supports NLP services be-yond clinical concept extraction By adding a sentimentanalysis NLP pipeline constructed by external NLP APIs ourprototype can perform sentiment analysis on clinical doc-uments For instance changing from concept extractionto sentiment analysis can be accomplished by adjusting theAPI request parameters from ldquoldquofeaturesrdquo ldquoentitiesrdquordquo toldquoldquofeaturesrdquo ldquosentimentrdquordquo

41 Evaluation Results In comparison with the popularbiomedical NLP component collections listed in [40] themain advantage of our proposed approach is its lightweightnature )e popular component collections such ascTAKES Bluima and JCoRe require an intensive IT re-sources investment including Java developers NLP spe-cialists with experience in the UIMA framework and localhardware support On the contrary clinical institutionscould start to process unstructured text with as little re-sources as possible due to the fact that our cloud-basedapproach outsources NLP to external NLP services More-over Bluima has not been updated for four years Instead ofreplacing the popular NLP tools our approach should beconsidered as an alternative approach in the face of time andresource constraints

42 Error Analysis An error analysis has been carried outin order to better understand the performance of theprototype As explained in Section 22 there are two typesof errors namely FPs and FNs Figure 5 shows the per-centage of FP and FN errors in all experiments First of allone major source of errors in the two i2b2 datasets is falsenegatives which means many annotated terms in thedatasets are not extracted by our prototype )e highproportion of FNs is in great part attributed to the entity-type detection errors Since some NLP APIs (Mean-ingCloud and Open Calais) are unable to extract phar-maceutical drug entities it results in a lower amount ofextracted entities and higher false negatives )erefore toenhance the performance NLP APIs such as Mean-ingCloud and Open Calais might as well be excluded

Nevertheless the higher number of false positives led toan overall performance loss in the OPERAM medicalconditions extraction We found out that the problem liesin the annotation For example the sentence ldquoFall during

Figure 4 Prototype user interface of the multiple NLP API extraction pipeline A demo video and source code are available online

Table 3 Impact of negation from the experiments

Dataset Negation(κ) Recall Precision F1

score

Obesity challenge True 0733 0939 0823False 0805 0925 0861

Medication challenge True 062 0835 0712False 0636 0838 0724

OPERAM medicalconditions

True 0594 0271 0373False 0594 0271 0373

OPERAM medications True 0795 0816 0805False 0795 0816 0805


the night multiple hematomas Orthostatic hypotensionprovenrdquo contains two medical conditions hematoma andorthostatic hypotension Hematoma was found by two out ofsix extractors orthostatic hypotension was found by five outof six However neither of these two was annotated mostlikely because the context of the sentence was in past tenseand potentially not applicable to the current state of thepatient

43 Limitations and Future Research )ere are a number ofhurdles that prevent the adoption of our approach in dailypractice Further research is necessary to sufficiently addressthese concerns First of all practical implementation re-quires a more thorough privacy and security component)e privacy and security component is part of the proposedarchitecture and currently implemented in the prototypeusing CRATE [26] and HTTPS However since only ano-nymized datasets are used in the evaluation the deidenti-fication toolkit CRATE was not validated Before thepractical adoption we need to first evaluate the performanceof the privacy and security component with real-worldclinical data

Another concern lies in the computational efficiency ofour approach namely execution time As shown in thedemo video it takes about 20 seconds to process a dischargeletter In specific the majority of time (15 seconds) goes toannotation in which extracted terms are first encoded withUMLS and then pairwise similarity between them is

calculated Since the prototype was running locally on alaptop with 8GB RAM we think it would become faster if weimplement it on a larger server

In practice clinical NLP is employed to solve variousclinical problems ranging from entity extraction to cohortdetection Our research demonstrates that the proposedapproach performs well on clinical concept extraction It iscrucial to conduct further evaluation on other tasks such ascohort detection and sentiment analysis before adopting theapproach in practice

Last but not least due to the wide adoption of healthinformation systems (HIS) in healthcare institutions de-veloping a simple method that supports the integration ofour approach with HIS would facilitate its implementation

5 Conclusion

)e proposed NLP architecture offers an efficient solution todevelop tools that are capable of processing unstructuredclinical data in the healthcare industry With our approachless time and resources are required to create and maintainNLP-enabled clinical tools given that all NLP tasks areoutsourced Moreover the prototype built upon the ap-proach produces satisfactory overall results and its per-formance on certain datasets indicates that its practicalapplication in clinical text processing particularly clinicalconcept extraction is promising Nevertheless high varianceamong different datasets brings concerns on its general-ization and practicability

Table 4 Overall results on three datasets

Dataset κ c θ Recall Precision F1 score

Obesity challenge False 01 02 0805 0925 0861Baselinelowast 0771 0815 0787

Medication challenge False 01 035 0636 0838 0724Baselinelowast 0794 0845 0818

OPERAM medical conditions True 01 05 0594 0271 0373OPERAM medications False 0 035 0795 0816 0805lowastAverage of the top 5 best systems from the challenge

0

20

40

60

80

Obesity challenge Medication challenge OPERAM medical conditions OPERAM medications

FP ()

FN ()

Figure 5 Error distribution of all the experiments false positives vs false negatives


Data Availability

Source code and the OPERAM dataset are available atthe GitHub repository httpsgithubcomianshan0915MABNLP )e two i2b2 datasets are accessible fromhttpswwwi2b2orgNLPDataSetsMainphp Finallya demo video of the prototype is available at httpsyoutubedGk9NQGWYfI

Conflicts of Interest

)e authors declare that they have no conflicts of interest

Acknowledgments

)is work is part of the project ldquoOPERAM OPtimisingthERapy to prevent Avoidable hospital admissions in theMultimorbid elderlyrdquo supported by the European Com-mission (EC) HORIZON 2020 proposal 634238 and by theSwiss State Secretariat for Education Research and In-novation (SERI) under contract number 150137 )eopinions expressed and arguments employed herein arethose of the authors and do not necessarily reflect the officialviews of the EC and the Swiss government

References

[1] S Doan M Conway T M Phuong and L Ohno-MachadoldquoNatural language processing in biomedicine a unified systemarchitecture overviewrdquo in Methods in Molecular Biologyvol 1168 pp 275ndash294 Springer Clifton NJ USA 2014

[2] G K Savova J J Masanz P V Ogren et al ldquoMayo clinicaltext analysis and knowledge extraction system (cTAKES)architecture component evaluation and applicationsrdquo Jour-nal of the American Medical Informatics Association vol 17no 5 pp 507ndash513 2010

[3] J Patrick andM Li ldquoHigh accuracy information extraction ofmedication information from clinical notes 2009 i2b2medication extraction challengerdquo Journal of the AmericanMedical Informatics Association vol 17 no 5 pp 524ndash5272010

[4] T Cai A A Giannopoulos S Yu et al ldquoNatural languageprocessing technologies in radiology research and clinicalapplicationsrdquo Radiographics vol 36 no 1 pp 176ndash191 2016

[5] M Kreuzthaler and S Schulz ldquoDetection of sentenceboundaries and abbreviations in clinical narrativesrdquo BMCMedical Informatics and Decision Making vol 15 no S2pp 1ndash13 2015

[6] O Bodenreider ldquo)e unified medical language system(UMLS) integrating biomedical terminologyrdquo Nucleic AcidsResearch vol 32 no 1 pp D267ndashD270 2004

[7] H Cunningham V Tablan A Roberts and K BontchevaldquoGetting more out of biomedical documents with GATErsquos fulllifecycle open source text analyticsrdquo PLoS ComputationalBiology vol 9 no 2 Article ID e1002854 2013

[8] D Ferrucci and A Lally ldquoUIMA an architectural approach tounstructured information processing in the corporate re-search environmentrdquo Natural Language Engineering vol 10no 3-4 pp 327ndash348 2004

[9] J-H Chiang J-W Lin and C-W Yang ldquoAutomated eval-uation of electronic discharge notes to assess quality of carefor cardiovascular diseases using medical language extractionand encoding system (MedLEE)rdquo Journal of the American

Medical Informatics Association vol 17 no 3 pp 245ndash2522010

[10] R E de Castilho and I Gurevych ldquoA broad-coverage col-lection of portable NLP components for building shareableanalysis pipelinesrdquo in Proceedings of the Workshop on OpenInfrastructures and Analysis Frameworks for HLT pp 1ndash11Dublin Ireland August 2014

[11] K Kreimeyer M Foster A Pandey et al ldquoNatural languageprocessing systems for capturing and standardizing un-structured clinical information a systematic reviewrdquo Journalof Biomedical Informatics vol 73 pp 14ndash29 2017

[12] D Carrell ldquoA strategy for deploying secure cloud-basednatural language processing systems for applied researchinvolving clinical textrdquo in Proceedings of the System Sciences(HICSS) 2011 44th Hawaii International Conference on 2011pp 1ndash11 Kauai HI USA January 2011

[13] R Dale ldquoNLP meets the cloudrdquo Natural Language Engi-neering vol 21 no 4 pp 653ndash659 2015

[14] W L Currie B Desai and N Khan ldquoCustomer evaluation ofapplication services provisioning in five vertical sectorsrdquoJournal of Information Technology vol 19 no 1 pp 39ndash582004

[15] A Rago F M Ramos J I Velez J A Dıaz Pace andC Marcos ldquoTeXTracT a web-based tool for building NLP-enabled applicationsrdquo in Proceedings of the Simposio Argen-tino de Ingenierıa de Software (ASSE 2016)-JAIIO 45 BuenosAires Argentina September 2016

[16] G Haffari M Carman and T D Tran ldquoEfficient bench-marking of NLP APIs using multi-armed banditsrdquo in Pro-ceedings of the 15th Conference of the European Chapter of theAssociation for Computational Linguistics vol 1 pp 408ndash416Valencia Spain April 2017

[17] S Hellmann J Lehmann S Auer and M Brummer ldquoIn-tegrating NLP using linked datardquo in Proceedings of the 12thInternational Semantic Web Conference Sydney AustraliaOctober 2013

[18] P Martınez J L Martınez I Segura-Bedmar J Moreno-Schneider A Luna and R Revert ldquoTurning user generatedhealth-related content into actionable knowledge through textanalytics servicesrdquo Natural Language Processing and TextAnalytics in Industry vol 78 pp 43ndash56 2016

[19] Z S Abdallah M Carman and G Haffari ldquoMulti-domainevaluation framework for named entity recognition toolsrdquoComputer Speech amp Language vol 43 pp 34ndash55 2017

[20] K Chard M Russell Y A Lussier E A Mendonca andJ C Silverstein ldquoA cloud-based approach to medical NLPrdquoAMIA Annual Symposium Proceedings pp 207ndash216 2011

[21] G Rizzo and R Troncy ldquoNERD a framework for unifyingnamed entity recognition and disambiguation web extractiontoolsrdquo in Proceedings of the System Demonstration at the 13thConference of the European Chapter of the Association forComputational Linguistics (EACLrsquo2012) Avignon FranceApril 2012

[22] API-Based CMS Buyerrsquos Guide May 2018 httpsnordicapiscomapi-based-cms-buyers-guide

[23] A R Hevner S T March J Park and S Ram ldquoDesignscience in information systems researchrdquo Management In-formation Systems Quarterly vol 28 no 1 p 6 2008

[24] K Peffers T Tuunanen M A Rothenberger andS Chatterjee ldquoA design science research methodology forinformation systems researchrdquo Journal of Management In-formation Systems vol 24 no 3 pp 45ndash77 2007

[25] Z Shen M Meulendijk and M Spruit ldquoA federated in-formation architecture for multinational clinical trials


STRIPA revisitedrdquo in Proceedings of the 24th EuropeanConference on Information Systems (ECIS) Istanbul TurkeyJune 2016

[26] R N Cardinal ldquoClinical records anonymisation and textextraction (CRATE) an open-source software systemrdquo BMCMedical Informatics and Decision Making vol 17 no 1 p 502017

[27] V Menger F Scheepers L M van Wijk and M SpruitldquoDEDUCE a pattern matching method for automatic de-identification of Dutch medical textrdquo Telematics and In-formatics vol 35 no 4 pp 727ndash736 2018

[28] B Koopman G Zuccon A Nguyen A Bergheim andN Grayson ldquoAutomatic ICD-10 classification of cancers fromfree-text death certificatesrdquo International Journal of MedicalInformatics vol 84 no 11 pp 956ndash965 2015

[29] Y Ni S Kennebeck J W Dexheimer et al ldquoAutomatedclinical trial eligibility prescreening increasing the efficiencyof patient identification for clinical trials in the emergencydepartmentrdquo Journal of the American Medical InformaticsAssociation vol 22 no 1 pp 166ndash178 2015

[30] B J Marafino W John Boscardin and R Adams DudleyldquoEfficient and sparse feature selection for biomedical textclassification via the elastic net application to ICU riskstratification from nursing notesrdquo Journal of Biomedical In-formatics vol 54 pp 114ndash120 2015

[31] V Menger M Spruit R van Est E Nap and F ScheepersldquoMachine learning approach to inpatient violence risk as-sessment using routinely collected clinical notes in electronichealth recordsrdquo JAMA Network Open vol 2 no 7 Article IDe196709 2019 2019

[32] J P Pestian P Matykiewicz M Linn-Gust et al ldquoSentimentanalysis of suicide notes a shared taskrdquo Biomedical In-formatics Insights vol 5S1 2012

[33] W W Chapman W Bridewell P Hanbury G F Cooperand B G Buchanan ldquoEvaluation of negation phrases innarrative clinical reportsrdquo in Proceedings of the AMIA Sym-posium pp 105ndash109 Washington DC USA November 2001

[34] S Mehrabi A Krishnan S Sohn et al ldquoDEEPEN a negationdetection system for clinical text incorporating dependencyrelation into NegExrdquo Journal of Biomedical Informaticsvol 54 pp 213ndash219 2015

[35] W W Chapman G F Cooper P Hanbury B E ChapmanL H Harrison and M M Wagner ldquoCreating a text classifierto detect radiology reports describing mediastinal findingsassociated with inhalational anthrax and other disordersrdquoJournal of the American Medical Informatics Associationvol 10 no 5 pp 494ndash503 2003

[36] S Meystre and P J Haug ldquoNatural language processing toextract medical problems from electronic clinical documentsperformance evaluationrdquo Journal of Biomedical Informaticsvol 39 no 6 pp 589ndash599 2006

[37] K J Mitchell M J Becich J J Berman et al ldquoImple-mentation and evaluation of a negation tagger in a pipeline-based system for information extract from pathology reportsrdquoStudies in Health Technology and Informatics vol 107 no 1pp 663ndash667 2004

[38] O Uzuner ldquoRecognizing obesity and comorbidities in sparsedatardquo Journal of the American Medical Informatics Associa-tion vol 16 no 4 pp 561ndash570 2009

[39] O Uzuner I Solti and E Cadag ldquoExtracting medication in-formation from clinical textrdquo Journal of the American MedicalInformatics Association vol 17 no 5 pp 514ndash518 2010

[40] P Przybyła M Shardlow S Aubin et al ldquoText mining re-sources for the life sciencesrdquo Database vol 2016 2016


International Journal of

AerospaceEngineeringHindawiwwwhindawicom Volume 2018

RoboticsJournal of

Hindawiwwwhindawicom Volume 2018


Active and Passive Electronic Components

VLSI Design



Shock and Vibration


Civil EngineeringAdvances in

Acoustics and VibrationAdvances in



Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawiwwwhindawicom

Volume 2018

Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom

The Scientific World Journal

Volume 2018

Control Scienceand Engineering

Journal of



Journal ofEngineeringVolume 2018

SensorsJournal of



RotatingMachinery


Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018


Chemical EngineeringInternational Journal of Antennas and

Propagation




Navigation and Observation


Hindawi

wwwhindawicom Volume 2018

Advances in

Multimedia

Submit your manuscripts atwwwhindawicom

more reasonable [9 10] Although it dramatically reducesresources and level of expertise we argue that it is not anefficient and effective solution for two main reasons Firstlynearly every NLP pipeline that is created to address a singlespecific clinical need either rule or machine-learning basedhas been proven to be useful for only its designated purposes[11] )us reusability is difficult given the properties Sec-ondly deploying cTAKES-based NLP pipelines implies ahigh cost of operation which requires installation andconfiguration of multiple components by NLP experts [12]Besides maintenance of a deployed NLP system requires acontinuous investment

With the purpose of simplifying and outsourcing theNLP implementation software as a service or SaaS has beenintroduced to the NLP world during recent years [13] SaaSgenerally refers to the mode of software delivery where end-users are charged with a monthly or annual subscription feeto utilize a set of functionalities over the Internet [14] NLPsystems distributed in the SaaS model are often availablethrough web application programming interfaces (APIs)and named as NLP APIs or cloud-based NLP APIs [13 15]Many NLP APIs have emerged from both companies anduniversities and are growing popularly [7] A few prominentexamples are IBMWatson Aylien Lexalytics and TextRazor[13] From the cost-benefit perspective these NLP APIsallow developers to rapidly create NLP-enabled toolswithout investing abundant resources on implementingnecessary NLP techniques in codes and on regular main-tenance A number of applications based on NLP APIs werebuilt [16ndash18]

To utilize NLP APIs API-based frameworks have beenproduced [1519ndash21] API-based systems also known ascloud-based refer to tools that are built on external webAPIs and having their functionalities partially or fully ac-complished with one or a pipeline of APIs Due to thegrowing popularity of web APIs in the software industryAPI-based tools are abundant in companies For instance anAPI-based CMS (content management system) is utilized tosave development resources and follow-up maintenance[22] Furthermore researchers have also investigated theapproach in recent years Rizzo and Troncy proposed theNamed Entity Recognition and Disambiguation (NERD)framework that incorporates the result of ten different publicAPIs-based NLP extractors [21] A web-based tool called

TeXTracT was devised to support the setup and deploymentof NLP techniques on demand [15] Abdallah et al de-veloped a flexible and extensible framework for integratingnamed entity recognition (NER) web APIs and assessed itacross multiple domains [19] Although these tools exhibitpromising results few were built for clinical NLP or eval-uated on clinical datasets )erefore it is safe to say thatadopting these tools in clinical settings would be problematicdue to the unique characteristics of the clinical domain Forexample privacy is considered to be of the utmost impor-tance but none of the above tools have taken it intoconsideration

)is paper thus proposes a lightweight framework whichenables a rapid development of clinical NLP systems withexternal NLP APIs )e approach has the following ad-vantages compared to traditional NLP frameworks (1) fastdevelopment (2) lower costs (3) flexibility and (4) pro-gramming language independent )e deployment is min-imized by outsourcing both NLP tasks and backgroundknowledge to external API services )us NLP systems canbe quickly and cost-efficiently developed based on theproposed framework )e framework is flexible in manyaspects To begin with it supports the flexible combinationof different NLP tasks from external APIs Secondly usershave the freedom of choosing their preferred NLP APIvendors and multiple APIs can be integrated to achievebetter results To evaluate the framework we have built aweb-based open-source clinical NLP application

2 Methods

21DesignScience Our research followed the design scienceresearch as we built and evaluated the framework because ofits strength and popularity in solving a real-world problemby designing and building an innovative IT artifact [23] Inour case the artifact is a lightweight framework that facil-itates clinical NLP systems development We follow thedesign science research methodology (DSRM) proposed byPeffers et al which consists of six steps problem identifi-cation and motivation definition of the objectives for asolution design and development demonstration evalua-tion and communication [24]

)e DSRM is initiated by the (I) problem identificationand motivation which we addressed by literature studyPrevious studies have described the general architecture ofclinical NLP systems and how expensive it is to build themEven though the introduction of modular NLP frameworksreduced the complexity of NLP systems it is still challengingto create clinical NLP systems for many healthcare in-stitutions due to limited resources Based on the identifiedproblem we inferred the (II) objectives for a solutioncreating a lightweight NLP framework that enables a rapiddevelopment of an API-based clinical NLP system In the(III) design and development we developed the frameworkbased on the general architecture we identified after whicheach of its components is explained in detail To (IV)demonstrate and (V) evaluate the framework a web-basedopen-source clinical NLP application was developedMoreover experiments were carried out with three clinical

Backgroundknowledge

Framework

Ontologies domain models domain knowledge trained corpora

NLP pipelines

Task 1 Task 2 Task N

Text Structured form

Figure 1 A general architecture of clinical NLP systems













precision TP

TP + FP (1)

recall TP

TP + FN (2)


(3)

3 Results














Text

NLP pipelines

Apps

(i)(ii)

(iii)


Structured form


API NAPI 2API 1


Infrastructure

API processing

Privacy Security
















IBM WatsonNLU



Aylien






Meaning Cloud



Alchemy API





classification





HavenOnDemand


tokenization



α4

1113874 1113875 +3β4

1113888 1113889≻c (4)



1113944n

i1ωi (5)


















ni1XδW



Infrastructure

API integration


JSON

JSON

JSON JSON

Extraction pipeline




Concept extraction






4 Discussion








score




True 0594 0271 0373False 0594 0271 0373









5 Conclusion







0

20

40

60

80


FP ()

FN ()



Data Availability




Acknowledgments


References















































RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia













precision TP

TP + FP (1)

recall TP

TP + FN (2)


(3)

3 Results














Text

NLP pipelines

Apps

(i)(ii)

(iii)


Structured form


API NAPI 2API 1


Infrastructure

API processing

Privacy Security
















IBM WatsonNLU



Aylien






Meaning Cloud



Alchemy API





classification





HavenOnDemand


tokenization



α4

1113874 1113875 +3β4

1113888 1113889≻c (4)



1113944n

i1ωi (5)


















ni1XδW



Infrastructure

API integration


JSON

JSON

JSON JSON

Extraction pipeline




Concept extraction






4 Discussion








score




True 0594 0271 0373False 0594 0271 0373









5 Conclusion







0

20

40

60

80


FP ()

FN ()



Data Availability




Acknowledgments


References















































RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia











Text

NLP pipelines

Apps

(i)(ii)

(iii)


Structured form


API NAPI 2API 1


Infrastructure

API processing

Privacy Security
















IBM WatsonNLU



Aylien






Meaning Cloud



Alchemy API





classification





HavenOnDemand


tokenization



α4

1113874 1113875 +3β4

1113888 1113889≻c (4)



1113944n

i1ωi (5)


















ni1XδW



Infrastructure

API integration


JSON

JSON

JSON JSON

Extraction pipeline




Concept extraction






4 Discussion








score




True 0594 0271 0373False 0594 0271 0373









5 Conclusion







0

20

40

60

80


FP ()

FN ()



Data Availability




Acknowledgments


References















































RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia















IBM WatsonNLU



Aylien






Meaning Cloud



Alchemy API





classification





HavenOnDemand


tokenization



α4

1113874 1113875 +3β4

1113888 1113889≻c (4)



1113944n

i1ωi (5)


















ni1XδW



Infrastructure

API integration


JSON

JSON

JSON JSON

Extraction pipeline




Concept extraction






4 Discussion








score




True 0594 0271 0373False 0594 0271 0373









5 Conclusion







0

20

40

60

80


FP ()

FN ()



Data Availability




Acknowledgments


References















































RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia



α4

1113874 1113875 +3β4

1113888 1113889≻c (4)



1113944n

i1ωi (5)


















ni1XδW



Infrastructure

API integration


JSON

JSON

JSON JSON

Extraction pipeline




Concept extraction






4 Discussion








score




True 0594 0271 0373False 0594 0271 0373









5 Conclusion







0

20

40

60

80


FP ()

FN ()



Data Availability




Acknowledgments


References















































RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia








ni1XδW



Infrastructure

API integration


JSON

JSON

JSON JSON

Extraction pipeline




Concept extraction






4 Discussion








score




True 0594 0271 0373False 0594 0271 0373









5 Conclusion







0

20

40

60

80


FP ()

FN ()



Data Availability




Acknowledgments


References















































RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia



4 Discussion








score




True 0594 0271 0373False 0594 0271 0373









5 Conclusion







0

20

40

60

80


FP ()

FN ()



Data Availability




Acknowledgments


References















































RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia








5 Conclusion







0

20

40

60

80


FP ()

FN ()



Data Availability




Acknowledgments


References















































RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia


Data Availability




Acknowledgments


References















































RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia





















RoboticsJournal of




VLSI Design



Shock and Vibration







Journal of



Volume 2018



Volume 2018


Journal of




SensorsJournal of



RotatingMachinery





Propagation






Hindawi


Advances in

Multimedia
