ereefs data brokering layer

eReefs Data Brokering Layer

LAND AND WATER FLAGSHIP | OCEANS AND ATMOSPHERE FLAGSHIP

Jonathan Yu | Research Software EngineerEnvironmental Information Infrastructures / IBMI / WRM / LWF 12 February 2014

Using semantics to support data discovery & access

Outline

1. Background/Use cases - eReefs

2. Data discovery/access challenges

3. Data Provider Node (DPN) concept and semantics

4. Data Brokering Layer (DBL) concept and API

5. Applications

eReefs What is the

health of the reef?

I want to know the Chlorophyll-a

concentration levels in the reef

Where are good diving sites to bring people?

Remote sensing

In-situ sensors(Weather stations

and monitoring sites)

Modelleddata

3 | eReefs Data Brokering Layer | Jonathan Yu

eReefs Data Brokering Layer | Jonathan Yu

Data provider nodes (DPN)

4 |

Feature Service

CoverageService

Map service

Geoserver MapInfo THREDDS 52north

Obsservice

?

API

Multiple combinations of services, interfaces, datasets for each DPN - relies on some cataloguing solutions

https://aurin62.it.csiro.au:8180/geoserver/wescml-sf0/ows?service=WFS&version=1.1.0&request=GetFeature&typeName=wescml-sf0:CombinedMeterReadings&maxFeatures=50&outputFormat=text/xml;+subtype%3Dgml/3.1.1



Complex systems - inefficient, chaos

5 |

Portal X

Portal Y


Current cataloguing solutions and challenges

Strengths• Harvest capability and keyword search• Integration with open data catalogs• Metadata standards (ISO19XXX and

others)

Limitations• Doesn’t quite capture DPN governance• Doesn’t resolve semantic heterogeneity• Doesn’t allow semantics-rich search

and discovery - limited to keywords

6 |


Semantic Heterogeneity

7 |

Enviro Application

#1

Data

DB

Chl_MIM

Enviro Application

#2

Data

DB

mass_conc_chlorophyll_In_sea_water

Enviro Application

#3

Data

DB

mass_conc_chlorophyll_a_In_sea_water


Semantic Heterogeneity… leads to data silos

8 |

Enviro Application

#1

Enviro Application

#2

Enviro Application

#3

Data Data Data

DB DBDB

Chl_MIMmass_conc_chlorophyll_In_sea_water

mass_conc_chlorophyll_a_In_sea_waterX X

Meetings


Harmonised Publish, Discovery, Access and Use

9 |

Relies on community agreed vocabularies

Describe/Publish the data

Query/Use dataEnviro

Application #1

Enviro Application

#2

Enviro Application

#3

Data Data Data

DB DBDB

substanceOrTaxon= http://environment.data.gov.au/def/object/chlorophyll

scaledQuantityKind = http://environment.data.gov.au/def/property/chlorophyll_concentration

Need to communicate more consistentlyRequires shared, precise, agreed semantics

http://environment.data.gov.au/def/object/chlorophyll


http://sissvoc.ereefs.info/sissvoc/ereefs/resource?uri=http://environment.data.gov.au/def/property/chlorophyll_concentration





eReefs Data Brokering Layer (DBL)

10 |

Data Brokering

Layer

End users

eReefsPortal

Remote sensing DPN

DPN Ontology

DomainVocabs

e.g. Water Quality

Hydrodynamic DPN

In-situ monitoring DPNCatalogue

APISCKAN,

pyCSW, ERDDAP

Flexible, ‘pluggable’ DPNs, data services and datasets

Flexible discovery and delivery of data


DPN Semantic Description Example

11 |

dpn:Serviceorg:Organization dpn:Node

dpn:THREDDS

OC THREDDS Service

Ocean Colour DPN

CSIRO Ocean Colour Group

rdfs:subClassOf

rdf:type rdf:typerdf:type

DPN ontologyhttp://purl.org/dpn

DPN descriptioninstance


Data provider nodes (DPN)

12 |

Feature Service

CoverageService

Map service

Geoserver MapInfo THREDDS 52north

Obsservice

DPN Service

Description

Lightweight RDF description




DPN Semantic Description Example

13 |

dpn:Serviceorg:Organization dpn:Node

dpn:THREDDS

OC THREDDS Service

Ocean Colour DPN

CSIRO Ocean Colour Group

rdfs:subClassOf

rdf:type rdf:typerdf:type

THREDDS

NCI deployment

Data Brokering Layer

DPN registered with DBL


Supporting domain vocabularies and def’s

14 |

http://environment.data.gov.au/water/quality/def/op

Observable Property ontology

http://environment.data.gov.au/water/quality/def/op


Example: Domain vocab term

15 |


Example: Domain vocab term

16 |

... wqp:chlorophyll_a_concentration a skos:Concept, op:ScaledQuantityKind, qudt:ChemistryQuantityKind ;

skos:broader wqp:chlorophyll_concentration ; skos:prefLabel "chlorophyll a concentration"@en ;


Searches over vocabularies

17 |

http://sissvoc.ereefs.info/search/?q=chlorophyll&endpoint=http://sissvoc.ereefs.info/sissvoc/ereefs





Data annotated with bindings to vocab URIs

18 |

THREDDS

THREDDS Catalog

Domain Vocabs(Water Quality at

environment.data.gov.au)

Quantities/ Units ontology(QUDT)

substanceOrTaxon= http://environment.data.gov.au

/def/object/chlorophyll

scaledQuantityKind = http://environment.data.gov.au

/def/property/chlorophyll_concentration

unit= http://qudt.org/vocab/unit#Unitless

medium= http://environment.data.gov.au

/def/feature/ocean







http://environment.data.gov.au/





http://qudt.org/vocab/unit#Unitless


http://environment.data.gov.au/def/feature/ocean







DBL Harvesting and End Use

19 |

Data Brokering Layer

THREDDS

THREDDS Catalog

Domain Vocabs(Water Quality at

environment.data.gov.au)

Quantities/ Units ontology(QUDT)

substanceOrTaxon= http://environment.data.gov.au

/def/object/chlorophyll

scaledQuantityKind = http://environment.data.gov.au

/def/property/chlorophyll_concentration

unit= http://qudt.org/vocab/unit#Unitless

medium= http://environment.data.gov.au

/def/feature/ocean

DPN Service

Description

End users

Client application

chlorophyll





















eReefs Visualisation Portal Prototype

http://vizportal.meteor.com

20 |

http://vizportal.meteor.com/

http://vizportal.meteor.com/

eReefs Data Brokering Layer | Jonathan Yu21 |


Benefits

35 |

Data Provider X

Data Brokering

Layer

DPN Ontology

DomainVocabs

e.g. Water Quality

Catalogue APIS

CKAN, pyCSW, ERDDAP

End users

eReefsPortal

Data Provider Y

Flexible, ‘pluggable’ DPNs, data services and datasets

Allows flexible discovery and delivery of data


Benefits

Decoupled system allows for expanding components in isolation:• DPN set• Vocabulary definitions

Enhanced data discovery

Aid operational monitoring of DPN services

Potential to facilitate data assimilation and automated data processing

36 |


Future/Current Work

1. Test whether this the best approach for data services and dataset discovery- Compare/integrate with CKAN, Geonetwork, pyCSW, ERDDAP

2. Explore richer queries using semantics of the DPN metadata, vocabularies and services

3. Use DBL for data assimilation – hydrodynamic models with remote sensing

4. Demonstrate DBL applications beyond eReefs

37 |


Summary

• The ‘Data Provider Node’ concept provides a way of thinking about data ownership, publication, implementation and governance

• Semantic web technologies allow for machine-readable DPN descriptions precisely and richly

• The ‘Data Brokering Layer’ concept uses the domain and DPN semantics to provide:

1. Flexible and ‘pluggable’ DPNs - data services and datasets

2. Allows flexible discovery and use of data

• Prototypes and demos via client applications

• Future work

38 |

LAND AND WATER

Thank you

Land and WaterJonathan YuResearch Software Engineert +61 3 9252 6440e [email protected]

Land and WaterBen Leighton

Software Engineert +61 3 9252 6516e [email protected]

Land and WaterShane Seaton

Software Engineert +61 2 6246 5825e [email protected]

More info:https://wiki.csiro.au/display/ER/Data+Brokering+Layerhttps://wiki.csiro.au/display/ER/Semantic+service+descriptionshttps://wiki.csiro.au/display/ER/Semantic+descriptions+of+datasets

ereefs data brokering layer

Data & Analytics