Annual reports and feedback from UMLS licensees

Kin Wah Fung MD, MSc, MAKin Wah Fung MD, MSc, MA

The UMLS TeamThe UMLS Team

National Library of MedicineNational Library of Medicine

Workshop on the Future of the UMLS Semantic Network

2

BackgroundBackground

One of the requirements of the UMLS license is to submit One of the requirements of the UMLS license is to submit an annual report of UMLS usage within 30 days of the end an annual report of UMLS usage within 30 days of the end of each yearof each year

Up to last year, a free-form email (with suggested section Up to last year, a free-form email (with suggested section headings) was the main way to collect the annual reports. headings) was the main way to collect the annual reports. Disadvantages:Disadvantages: low response rate (usually less than 100 received each year)low response rate (usually less than 100 received each year) difficult to analyzedifficult to analyze

From 2005, online form to collect annual reports. From 2005, online form to collect annual reports. Advantages:Advantages: more structured questionsmore structured questions easier to analyzeeasier to analyze targetted email reminders possible => higher response ratetargetted email reminders possible => higher response rate

3

Format of online annual reportFormat of online annual report

Logon page – identity authenticated with UMLS license Logon page – identity authenticated with UMLS license numbernumber

Main report page:Main report page: user info: affiliation, activity type, computing environmentuser info: affiliation, activity type, computing environment usage of UMLSusage of UMLS feedback on user support and documentationfeedback on user support and documentation

Optional sections for feedback on:Optional sections for feedback on: MetathesaurusMetathesaurus Semantic NetworkSemantic Network Lexical toolsLexical tools UMLS Knowledge Source ServerUMLS Knowledge Source Server

4

Results (up to March 31, 2005)Results (up to March 31, 2005)

Total responsesTotal responses 1,0051,005

Valid responsesValid responses 988988

Have used the UMLSHave used the UMLS 495495

Have used the Semantic Have used the Semantic NetworkNetwork

210210

Provided feedback on the Provided feedback on the Semantic NetworkSemantic Network

7676

5

Results (up to March 31, 2005)Results (up to March 31, 2005)

Total responsesTotal responses 1,0051,005

Valid responsesValid responses 988988

Have used the UMLSHave used the UMLS 495495

Have used the Semantic Have used the Semantic NetworkNetwork

210210

Provided feedback on the Provided feedback on the Semantic NetworkSemantic Network

7676

6

AffiliationAffiliation

AcademicAcademic 54 (71%)54 (71%)

CommercialCommercial 12 (16%)12 (16%)

Self-employedSelf-employed 4 (5%)4 (5%)

Government – non NLMGovernment – non NLM 3 (4%)3 (4%)

Government – NLMGovernment – NLM 2 (3%)2 (3%)

Not-for-profitNot-for-profit 1 (1%)1 (1%)

7

Main activity of organization/companyMain activity of organization/company

ResearchResearch 27 (36%)27 (36%)

EducationEducation 21 (28%)21 (28%)

Software developmentSoftware development 11 (14%)11 (14%)

Health care providerHealth care provider 10 (13%)10 (13%)

Information serviceInformation service 2 (3%)2 (3%)

Health insuranceHealth insurance 1 (1%)1 (1%)

OtherOther 3 (4%)3 (4%)

MissingMissing 1 (1%)1 (1%)

8

How long have you been using the UMLS?How long have you been using the UMLS?

Range 2 months – 15 yearsRange 2 months – 15 years Average: 4.3 yearsAverage: 4.3 years Median 2.3 yearsMedian 2.3 years Significantly longer than the rest of respondents - Significantly longer than the rest of respondents -

average 2.6 years, median 1 year average 2.6 years, median 1 year (Independent-samples T test, 2- tailed p = 0.003)(Independent-samples T test, 2- tailed p = 0.003)

9

How are you using the UMLS?How are you using the UMLS?

Terminology researchTerminology research 4949

Map between terminologiesMap between terminologies 3939

Indexing and information retrievalIndexing and information retrieval 3636

Support natural language processingSupport natural language processing 3434

Discover concepts in biomedical textsDiscover concepts in biomedical texts 3232

Acquire biomedical knowledgeAcquire biomedical knowledge 3232

Design, construct and maintain your terminologyDesign, construct and maintain your terminology 2929

Support terminology serviceSupport terminology service 1515

Access specific terminologiesAccess specific terminologies 1515

Publish and distribute your terminologyPublish and distribute your terminology 66

OtherOther 22

10

What kind of information do you use the What kind of information do you use the UMLS to process?UMLS to process?

Clinical informationClinical information Non-clinical informationNon-clinical information

Findings & diagnosisFindings & diagnosis 4646 Genomics & proteomicsGenomics & proteomics 2323

ProceduresProcedures 3131 Bibliographic informationBibliographic information 2121

Laboratory reportsLaboratory reports 2727 Consumer health informationConsumer health information 1010

DrugsDrugs 2626 BillingBilling 33

Clinical guidelinesClinical guidelines 1414 OtherOther 33

Clinical outcomeClinical outcome 1010

Nursing informationNursing information 88

OtherOther 88

TotalTotal 170170 6060

11

Usage of the Semantic NetworkUsage of the Semantic Network

How have you used the Semantic Network?How have you used the Semantic Network? in a research setting – 69in a research setting – 69 in a production environment – 7in a production environment – 7

Ways in which Semantic Network is used:Ways in which Semantic Network is used: in natural language processingin natural language processing

word sense disambiguationword sense disambiguation tagging of natural language expressionstagging of natural language expressions natural language generationnatural language generation

ontological uses ontological uses finding concept distancefinding concept distance reference for building biomedical ontologiesreference for building biomedical ontologies

other usesother uses suggesting relationships between UMLS conceptssuggesting relationships between UMLS concepts restricting search of terms in the Metathesaurusrestricting search of terms in the Metathesaurus research on structure of Semantic Networkresearch on structure of Semantic Network

12

Components of Semantic Network usedComponents of Semantic Network used

Do you use the Semantic Network on its own or in Do you use the Semantic Network on its own or in conjunction with the Metathesaurus concepts?conjunction with the Metathesaurus concepts? on its own - 15on its own - 15 in conjunction - 58in conjunction - 58

Semantic typesSemantic types 64 (84%)64 (84%)

Semantic groupsSemantic groups 48 (63%)48 (63%)

Semantic relationships (e.g. location_of, Semantic relationships (e.g. location_of, causes)causes)

43 (57%)43 (57%)

Semantic relations (e.g. Tissue|Semantic relations (e.g. Tissue|location_of|Neoplastic Process)location_of|Neoplastic Process)

39 (51%)39 (51%)

13

Have you extended or modified the Semantic Have you extended or modified the Semantic Network?Network?

Semantic Semantic TypeType

Semantic Semantic RelationshipRelationship

Semantic Semantic RelationRelation

Area of applicationArea of application

11 ++ ++ creating terminologycreating terminology

22 ++ supporting terminology serversupporting terminology server

33 ++ ++ building knowledge basesbuilding knowledge bases

44 ++ ++ natural language generationnatural language generation

55 ++ ++ ++ creating terminologycreating terminology

66 -- natural language processing of clinical natural language processing of clinical documentsdocuments

77 -- natural language processing of clinical and natural language processing of clinical and biomedical documentsbiomedical documents

88 -- +/-+/- concept distance estimationconcept distance estimation

99 +/-+/- +/-+/- +/-+/- clinical information systemsclinical information systems

14

What improvements would you suggest?What improvements would you suggest?

more semantic types (4) more semantic types (4) better description of semantic types and how they are assigned (2) better description of semantic types and how they are assigned (2) multiple hierarchiesmultiple hierarchies more semantic relationsmore semantic relations more uniform granularitymore uniform granularity extension of relations to concept levelextension of relations to concept level evolve towards a formal ontologyevolve towards a formal ontology

15

One possible way to prioritize changes?One possible way to prioritize changes?

16

One possible way to prioritize changes?One possible way to prioritize changes?

Higher demandHigher demand Lower demandLower demand

17

One possible way to prioritize changes?One possible way to prioritize changes?

Higher demandHigher demand Lower demandLower demand

Less work Less work involved involved

More work More work involvedinvolved

18

One possible way to prioritize changes?One possible way to prioritize changes?

Higher demandHigher demand Lower demandLower demand

Less work Less work involved involved

- better description of semantic - better description of semantic types and how they are types and how they are assigned (2)assigned (2)

- multiple hierarchies- multiple hierarchies

- more semantic relations- more semantic relations

More work More work involvedinvolved

- more semantic types (4)- more semantic types (4) - more uniform granularity- more uniform granularity

- extension of relations to concept - extension of relations to concept levellevel

- evolve towards a formal - evolve towards a formal ontologyontology

19

One possible way to prioritize changes?One possible way to prioritize changes?

Higher demandHigher demand Lower demandLower demand

Less work Less work involved involved

- better description of semantic - better description of semantic types and how they are types and how they are assigned (2)assigned (2)

- multiple hierarchies- multiple hierarchies

- more semantic relations- more semantic relations

More work More work involvedinvolved

- more semantic types (4)- more semantic types (4) - more uniform granularity- more uniform granularity

- extension of relations to concept - extension of relations to concept levellevel

- evolve towards a formal - evolve towards a formal ontologyontology

20

Thank you!