ontology alignment: evaluation and healthcare...

Ontology alignment:evaluation and

healthcare applications

Ernesto Jimenez-Ruiz

HealthInsight workshop, Oslo, May 20, 2016

1 / 32

Outline

Preliminaries

Ontology Alignment

Healthcare Applications

2 / 32

The presentation in a nutshell

Introduction to ontologies

Ontology alignment

Ontology alignment in practiceSemantic annotation and access of clinical lettersSemantic enrichment of UK BioBank Cardiac cine-MRI ScansShared hypothesis testing in the biomedical domainPistoia alliance mapping project (disease-phenotype domain)Lung Cancer Assistant (LCA)Optique: Scalable End-user Access to Big Data

3 / 32

Ontology alignment

3 / 32

Ontology alignment

3 / 32

What is an ontology?

Introduces vocabularyrelevant to a domain

Anatomy

Specifies meaning(semantics) of terms

Heart is a muscular organthat is part of thecirculatory system

Formalised using suitablelogic

Heart SUBCLASSOFMuscularOrgan AND(isPartOf SOMECirculatorySystem)

Borrowed from Ian’s slides: Ontologies and the Semantic Web:The Story So Far. Zhejiang University, April 2010

4 / 32

Anatomy

4 / 32

Anatomy

4 / 32

OWL: Web Ontology Language

World Wide Web Consortium (W3C) standard

The most widely used ontology modelling language

e.g. FMA, NCI, SNOMED CT

Formal underpinning of OWL is based on formal logic

Supported by tools and infrastructure

OWL example axiomsJuvenileArthritis v JuvenileDiseasePolyArthritis ≡ Arthritis u > 5 affects.JointDisease u Joint v ⊥JuvenileIdiopathicArthritis @ “Juvenile Rheumatoid Arthritis”John:Pattient u ∃suffersFrom.JuvenileIdiopathicArthritis

5 / 32

OWL: Web Ontology Language

World Wide Web Consortium (W3C) standard

The most widely used ontology modelling language

e.g. FMA, NCI, SNOMED CT

Formal underpinning of OWL is based on formal logic

Supported by tools and infrastructure

OWL example axiomsJuvenileArthritis v JuvenileDiseasePolyArthritis ≡ Arthritis u > 5 affects.JointDisease u Joint v ⊥JuvenileIdiopathicArthritis @ “Juvenile Rheumatoid Arthritis”John:Pattient u ∃suffersFrom.JuvenileIdiopathicArthritis

5 / 32

What ontologies are good for?

Help identify and resolve disagreements in the domain

Ontology alignment will play a key role

Independence of logical/physical schema

Formulation of queries closer to domain experts

Incomplete and semi-structured data

Integration of heterogeneous sources

6 / 32

What ontologies are good for?

Help identify and resolve disagreements in the domain

Independence of logical/physical schema

Formulation of queries closer to domain experts

Incomplete and semi-structured data

Integration of heterogeneous sources

6 / 32

Outline

Preliminaries

Ontology Alignment

7 / 32

Ontology alignment: motivation

An application domain can be modelled with different pointsof view and purposes

Ontologies with different naming and modellingconventions exist for the same domain

Aligning these ontologies will enable interoperability betweenontology-based information systems

Reusing vocabulary from domain ontologies is a good practicein ontology engineering

8 / 32

Ontology alignment: definition

Ontology alignments or mappings M are sets of tuples〈e1, e2, n, ρ〉

e1, e2 are entities in the input ontologies O1 and O2

n a confidence value between 0 and 1ρ is the semantic relationship between e1 and e2 (e.g.subsumption, equivalence or disjointness)

Formalized as OWL 2 axiomsWhere the semantic relationship ρ is one of {≡,v,w,⊥}Confidence values n are represented as axiom annotationsNo extra semantics

OWL 2 example mappingsO1:Joint ≡ O2:JointO3:Joint structure ≡ O2:JointO3:Joint structure ≡ O2:Articulation

9 / 32

Ontology alignment: systems

Given two input ontologies O1 and O2 generate an alignmentM as output.

Large number of available ontology alignment systems

Ontology Alignment Evaluation Initiative (OAEI)

LogMap . . .performs a simple matching but efficient (based on sophisticatedlexical and structural indexes),can efficiently match semantically rich ontologies containinghundreds of thousands of classes (FMA, NCI and SNOMED),incorporates reasoning and repair capabilitiesis one of the top systems in the OAEI evaluation campaign, andis the only system participating in all OAEI 2015 tracks.

Large-scale Interactive Ontology Matching: Algorithms andImplementation. ECAI 2012

LogMap: Logic-based and Scalable Ontology Matching. ISWC’11

10 / 32

Ontology alignment: systems

Given two input ontologies O1 and O2 generate an alignmentM as output.

Large number of available ontology alignment systems

Ontology Alignment Evaluation Initiative (OAEI)

LogMap . . .performs a simple matching but efficient (based on sophisticatedlexical and structural indexes),can efficiently match semantically rich ontologies containinghundreds of thousands of classes (FMA, NCI and SNOMED),incorporates reasoning and repair capabilitiesis one of the top systems in the OAEI evaluation campaign, andis the only system participating in all OAEI 2015 tracks.

Large-scale Interactive Ontology Matching: Algorithms andImplementation. ECAI 2012

LogMap: Logic-based and Scalable Ontology Matching. ISWC’11

10 / 32

Ontology alignment: resources

UMLS (Unified Medical Language System) Metathesaurus

Integrates more than one hundred thesauri and ontologiesContains more than 6 million entities

BioPortalContains more than 500 ontologiesRepresent a network of ontologiesMore than 10 million mappings are availableAlso includes user-submitted alignments

11 / 32

Ontology alignment: resources

UMLS (Unified Medical Language System) Metathesaurus

Integrates more than one hundred thesauri and ontologiesContains more than 6 million entities

BioPortalContains more than 500 ontologiesRepresent a network of ontologiesMore than 10 million mappings are availableAlso includes user-submitted alignments

11 / 32

Ontology alignment: limitations

Alignment systems, BioPortal and UMLS techniques. . .

are mostly based on lexical algorithms,although they also include user-submitted alignments(BioPortal) and expert assessment (UMLS),may lead to undesired cross-references,may lead to logical errors when considering the semantics of thesources being integrated.

12 / 32

Alignment systems, BioPortal and UMLS techniques. . .are mostly based on lexical algorithms,although they also include user-submitted alignments(BioPortal) and expert assessment (UMLS),may lead to undesired cross-references,may lead to logical errors when considering the semantics of thesources being integrated.

12 / 32

Alignment systems, BioPortal and UMLS techniques. . .are mostly based on lexical algorithms,although they also include user-submitted alignments(BioPortal) and expert assessment (UMLS),may lead to undesired cross-references,may lead to logical errors when considering the semantics of thesources being integrated.

12 / 32

Logic-based Assessment of UMLS

Assessment of the integration of FMA, NCI and SNOMED CTontologies within UMLS

UMLS alignmentsOntologies # Logical errors

FMA-NCI 3,024 655

FMA-SNOMED 9,072 6,179

SNOMED-NCI 19,622 20,944

Ernesto Jimenez-Ruiz, et al. Logic-based assessment of thecompatibility of UMLS ontology sources. J. Biomedical Semantics2(S-1): S2 (2011)

Ernesto Jimenez-Ruiz, Bernardo Cuenca Grau. LogMap:Logic-based and Scalable Ontology Matching. In the 10thInternational Semantic Web Conference (ISWC 2011).

13 / 32

Logic-based Assessment of UMLS

Assessment of the integration of FMA, NCI and SNOMED CTontologies within UMLS

UMLS alignmentsOntologies # Logical errors

FMA-NCI 3,024 655

FMA-SNOMED 9,072 6,179

SNOMED-NCI 19,622 20,944

Ernesto Jimenez-Ruiz, et al. Logic-based assessment of thecompatibility of UMLS ontology sources. J. Biomedical Semantics2(S-1): S2 (2011)

Ernesto Jimenez-Ruiz, Bernardo Cuenca Grau. LogMap:Logic-based and Scalable Ontology Matching. In the 10thInternational Semantic Web Conference (ISWC 2011).

13 / 32

Logic-based Assessment of BioPortal

Identification of error-free sets of alignments

BioPortal alignmentsOntologies # Logical errors

BDO-NCIT 1,636 34,341

CCONT-NCIT 2,097 50,304

EFO-NCIT 2,507 60,347

EP-FMA 78,489 210

EP-NCIT 2,465 14,687

MA-FMA 961 850

OMIM-NCIT 5,178 70,172

SDO-EP 135 44

UBERON-FMA 1,932 4,753

ZFA-EFO 427 913

ZFA-UBERON 724 104

Daniel Faria, Ernesto Jimenez-Ruiz, et al. Towards annotatingpotential incoherences in BioPortal mappings. In the 13thInternational Semantic Web Conference (ISWC 2014).

14 / 32

Logic-based Assessment of BioPortal

Identification of error-free sets of alignments

BioPortal alignmentsOntologies # Logical errors

BDO-NCIT 1,636 34,341

CCONT-NCIT 2,097 50,304

EFO-NCIT 2,507 60,347

EP-FMA 78,489 210

EP-NCIT 2,465 14,687

MA-FMA 961 850

OMIM-NCIT 5,178 70,172

SDO-EP 135 44

UBERON-FMA 1,932 4,753

ZFA-EFO 427 913

ZFA-UBERON 724 104

Daniel Faria, Ernesto Jimenez-Ruiz, et al. Towards annotatingpotential incoherences in BioPortal mappings. In the 13thInternational Semantic Web Conference (ISWC 2014).

14 / 32

Assessment of alignment systems: OAEI

Precision and recall wrt reference alignment or silver/goldstandard |MGS|

Precision (Pre) = |M ∩MGS|/|M|Recall (Rec) = |M ∩MGS|/|MGS|The F-score (F) = (2× Pre× Rec)/(Pre + Rec).

Logical errors ofM wrt O1 and O2.

Computation times are also considered.

Ontology Alignment Evaluation Initiative (OAEI)An annual campaign3 biomedical-themed tracks in 2016: Anatomy, LargeBio,Phenotype-Disease tracks

Ontology Alignment Evaluation Initiative (OAEI):http://www.ontologymatching.org/

15 / 32

Assessment of alignment systems: OAEI

Precision and recall wrt reference alignment or silver/goldstandard |MGS|

Precision (Pre) = |M ∩MGS|/|M|Recall (Rec) = |M ∩MGS|/|MGS|The F-score (F) = (2× Pre× Rec)/(Pre + Rec).

Logical errors ofM wrt O1 and O2.

Computation times are also considered.

Ontology Alignment Evaluation Initiative (OAEI)An annual campaign3 biomedical-themed tracks in 2016: Anatomy, LargeBio,Phenotype-Disease tracks

Ontology Alignment Evaluation Initiative (OAEI):http://www.ontologymatching.org/

15 / 32

Assessment in the OAEI large biomedical track

Ontologies and Reference Alignment

FMA v2.0 (78,989 classes), NCI v.08.05d (66,724 classes) andSNOMED CT v. Jan. 2009 (306,591 classes).Reference alignment based on UMLS

Matching problems

FMA-NCIFMA-SNOMEDSNOMED-NCI

Results

Results are far from perfect when involving SNOMED CT

16 / 32

Voting of computed alignments

Alignments computed in the 2011.5, 2012 and 2013 editions.

FMA-NCI: 18 contributing (independent) systemsFMA-SNOMED: 13 contributing (independent) systemsSNOMED-NCI: 9 contributing (independent) systems

Alignments voted (i.e. computed) by at least one system

Maximum recall

Alignments voted (i.e. computed) by all systems

Maximum precision

17 / 32

Maximum recall

Maximum precision

17 / 32

Maximum recall

Maximum precision

17 / 32

Assessment FMA-NCI: voting

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Number of votes

19,394

3,7092,984 2,779 2,572 2,406 2,168

1,642895

SizePrecision

RecallF-measure

Max recall: 0.96Max precision: 1.0

Vote ≥5: P=0.89, R=0.91

18 / 32

Assessment FMA-SNOMED: voting

1 2 3 4 5 6 7 8 9 10 11 12 13

Number of votes

19,993

2,7052,095

1,688 1,503 1,326 1,091751

SizePrecision

RecallF-measure

Vote ≥3: P=0.89, R=0.76

19 / 32

Assessment SNOMED-NCI: voting

1 2 3 4 5 6 7 8 9

Number of votes

30,252

18,281

15,042

13,404

11,620

10,265

8,9697,843

SizePrecision

RecallF-measure

Vote ≥3: P=0.82, R=0.67

20 / 32

User involvement in ontology alignment

Due to the limits to the quality of automated alignmentalgorithms.

Requires tool support

“Good” interfaces and servicesNumber of questions must be limitedSystems should make the most of user inputsSystems should consider the profile of the user

Currently evaluated in the OAEI

Simulation of users with different error rates

Zlatan Dragisic, Valentina Ivanova, Patrick Lambrix , Daniel Faria,Ernesto Jimenez-Ruiz, and Catia Pesquita. User validation inontology alignment. Submitted to ISWC 2016

21 / 32

Outline

Preliminaries

Ontology Alignment

22 / 32

1. Lung Cancer Assistant (LCA)

An ontology-based system which provides decision supportfor lung cancer treatment

LCA exploits the English Lung Cancer Dataset (LUCADA)

LUCADA ontology represents the semantic layer of the LCA,

Required alignment with SNOMED CTto facilitate interoperability with NHS systemspartially done without too support!

We offered LogMap alignment system to

identify the classes in SNOMED CT related to LUCADAextract a lung cancer-specific module of SNOMED CT

M. Berkan Sesen et al. Lung Cancer Assistant: a hybrid clinicaldecision support application for lung cancer care. Journal of theRoyal Society Interface. 2014

23 / 32

2. EU Project Optique

Scalable End-user Access to Big DataExposing relational data through ontologies

Table Name

AttributeAttributeAttributeAttribute

Table NameAttributeAttributeAttributeAttribute

Table Name

Ontologies

Imported

Metadata

BootstrappedManual

MappingsManual

Semantic Data Access

RelationalData

Bootstrapped

Table Name

Table NameAttributeAttributeAttributeAttribute

Table Name

RDF Data

Export

ApplicationsUsers

E. Jimenez-Ruiz et al. BootOX: Practical Mapping of RDBs toOWL 2. International Semantic Web Conference. 2015

24 / 32

3. UK BioBank Cardiac cine-MRI Scans

Semantic enrichment of free-text annotation of image qualityfor UK BioBank cardiac cine-MRI scansFree text annotations added to a spread-sheetThere is a high lexical and semantic variability in theannotationsOntologies will enhance the processing of the free-textannotationsRequired ontology alignment

to facilitate interoperability with NHS systems (SNOMED CT)and other UK BioBank parts

Ferreira et al. Cardiovascular magnetic resonance artefacts.Journal of Cardiovascular Magnetic Resonance 2013

25 / 32

4. Shared hypothesis testing

Some diseases may be evidenced across multiple biologicalscales (e.g., cellular, molecular, organic, behavioral)

Tests (e.g., cell viability, MRI analysis, gait analysis) mayevidence a factor

Factors are linked among each other via a causal relationship

A set of factors and the causality relationship constitute ahypothesis of the progression of the disease

Different specialists may work on different subparts of thehypothesisSpecialists may rely on different domain ontologies and usedifferent modelling/naming conventions

Use ontology alignment techniques

Asan Agibetov, E. Jimenez-Ruiz et al. Towards Shared HypothesisTesting in the Biomedical Domain. Semantic Web Applications andTools for Life Sciences (SWAT4LS). 2015

EU FP7 MultiScaleHuman project

26 / 32

4. Shared hypothesis testing

Some diseases may be evidenced across multiple biologicalscales (e.g., cellular, molecular, organic, behavioral)

Tests (e.g., cell viability, MRI analysis, gait analysis) mayevidence a factor

Factors are linked among each other via a causal relationship

A set of factors and the causality relationship constitute ahypothesis of the progression of the disease

Different specialists may work on different subparts of thehypothesisSpecialists may rely on different domain ontologies and usedifferent modelling/naming conventions

Use ontology alignment techniques

Asan Agibetov, E. Jimenez-Ruiz et al. Towards Shared HypothesisTesting in the Biomedical Domain. Semantic Web Applications andTools for Life Sciences (SWAT4LS). 2015

EU FP7 MultiScaleHuman project

26 / 32

5. Pistoia alliance ontologies mapping project

Currently looking for suitable tool supportDisease and phenotype domain:

Human Phenotype Ontology (HPO)←→ MammalianPhenotype Ontology (MP)Human Disease Ontology (DOID)←→ Orphanet and RareDiseases Ontology (ORDO)

Motivation: matching human inherited diseases withlaboratory studiesThe Pistoia Alliance Ontologies Mapping project co-organisesthe OAEI phenotype track

Pistoia Alliance Ontologies Mapping project: http:

//www.pistoiaalliance.org/projects/ontologies-mapping/

27 / 32

6. Semantic annotation of clinical letters (i)

The output of a visit is known as clinical letter, which servesto

document the patient’s progresscommunicate findings among specialists

Clinical letters are a key source of knowledge (specially forrare diseases)

The letter is typically tagged manually by a specialistresponsible for reading and annotating the interesting terms.

Common practice in the NHS: Genomics England, OCMR

28 / 32

6. Semantic annotation of clinical letters (i)

The output of a visit is known as clinical letter, which servesto

document the patient’s progresscommunicate findings among specialists

Clinical letters are a key source of knowledge (specially forrare diseases)

The letter is typically tagged manually by a specialistresponsible for reading and annotating the interesting terms.

Common practice in the NHS: Genomics England, OCMR

28 / 32

6. Semantic annotation of clinical letters (ii)

Manual annotation of clinical lettersRequires exhaustive input from a qualified medical professionalTime consumingLeads to the relatively limited use of clinical lettersLoss of potentially important informationSuitable software support is required to assist the clinician

S. Kudama et al. Towards Enabling the Semantic Access ofPhenotypic Information in Clinical Letters. Semantic WebApplications and Tools for Life Sciences (SWAT4LS). 2015

29 / 32

6. Semantic annotation of clinical letters (iii)

Special interest on HPO terms, however. . .

Text annotators may rely on different vocabulariesSome domains may require specific vocabulary as well (headinjuries vocabulary)UMLS vocabulary is richer than HPO vocabulary

Coordinated vocabularies:Exploit (and validate) HPO cross-references to UMLSExploit references of HPO to other (BioPortal) ontologiesUse ontology alignment techniques

30 / 32

6. Semantic annotation of clinical letters (iii)

Special interest on HPO terms, however. . .

Text annotators may rely on different vocabulariesSome domains may require specific vocabulary as well (headinjuries vocabulary)UMLS vocabulary is richer than HPO vocabulary

Coordinated vocabularies:Exploit (and validate) HPO cross-references to UMLSExploit references of HPO to other (BioPortal) ontologiesUse ontology alignment techniques

30 / 32

6. Semantic annotation of clinical letters (iv)

Coordinated vocabularies:

31 / 32

Questions?

Thank you for your attention

[email protected]

Acknowledgements:

EU project Optique and the EPSRC project DBOnto.

32 / 32

ontology alignment: evaluation and healthcare...

Documents