waterp water enhanced resource planning · 2017-12-08 · waterp water enhanced resource planning...

WatERP

Water Enhanced Resource Planning

“Where water supply meets demand”

GA number: 318603

WP 7: Pilots

7.3.1: Implementation of the Water Data Warehouse (WDW)

V1.0 28/02/2014

www.waterp-fp7.eu

http://www.waterp-fp7.eu/

Ref. 318603 - WatERP D7.3.1_Implementation_of_WDW_v1.0 page 2 of 59

Document Information

Project Number 318603 Acronym WatERP

Full title Water Enhanced Resource Planning “Where water supply meets demand”

Project URL http://www.waterp-fp7.eu

Project officer Grazyna Wojcieszko

Deliverable Number 7.3.1 Title Implementation of the Water Data Warehouse (WDW)

Work Package Number 7 Title Pilots

Date of delivery Contractual 17 Actual 17

Nature Prototype Report X Dissemination Other

Dissemination

Level Public X Consortium

Responsible Author Michael Quenzer Email [email protected]

Partner DISY Phone (+49) 721 1 6006-243

Abstract

(for dissemination)

Implementation of the Water Data Warehouse (WDW): This Task will implement the central

database of the platform in the pilot areas. The main activity in this task is to put together the

methods, reference data models and tools developed in WP3 applied to the specific elements

that integrate the pilot areas.

Key words Water Data Warehouse (WDW), Pilot Integration Manager (PIM), WaterML2.0, SOS, software

integration, pilot implementation

mailto:[email protected]


Glossary of acronyms

ACA Agencia Catalana del Agua (trans.

Catalan Water Agency)

CUAHSI Consortium of the Advanced of

Hydrological Sciences Inc.

DMS Demand Management System

DRL Data Retrieval Language

DSS Decision Support System

DMZ Demilitarized Zone

HIS Hydrologic Information System

Hydro DWG Hydrology Domain Working Group

OWL Web Ontology Language

MAS Multi Agent System

O&M Observations and Measurements

ODM Observation Data Model

OGC Open Geospatial Consortium

OMP Open Management Platform

OWL Web Ontology Language

PIM Pilot Integration Manager

REST Representational State Transfer

SOS Sensor Observation Service

SensorML Sensor Markup Language

SWE Sensor Web Enablement

SWKA Stadtwerke Karlsruhe

WaterML Water Markup Language

WDTF Water Data Transfer Format

WDW Water Data Warehouse

XML Extensible Markup Language


Executive Summary

This document describes the activities to implement the WDW on the pilot site. It describes what sensor

observation data is currently stored on the pilot site and how this data can be accessed. For each pilot it

describes the structure of the sensor observation data and which strategies can be applied to map the

data to the ontology.

Next it describes the steps to integrate the pilot infrastructure into the WDW and what additional tools

have to be implemented to populate both, the ontology and the observation results.

To fully understand this document, the following deliverables have to be read.

Number Title Description

D1.3 Generic Ontology for water

supply distribution chain

This deliverable summarizes the ontology including the scope, purpose and

implementation language to be used.

D1.4.1 Inference and Simulation

Engine Conceptual Design

This deliverable depicts the architecture of the Decision Support System

including a behavioural definition of the inference and simulation engine.

D2.1 External System Integration

requirement

Comprehensive review of generic water supply - distribution chain was

undertaken in order to understand general requirement for system

integration and interoperability to be performed within the WatERP project

development.

D3.1 WDW Conceptual Design

Report summarizing architectural design of the water data warehouse used

to integrate all data relevant for the WatERP project and the interfaces used

for data integration.

D3.2 WDW 1st Prototype Description of the implementation, installation and usage of the Water Data

Warehouse.


Table of contents

1. INTRODUCTION ............................................................................................................................. 10

1.1 WATERP OVERVIEW.................................................................................................................................. 10

1.2 STANDARDS FOR INFORMATION INTERCHANGE .............................................................................................. 11

1.3 SCOPE OF THE FIRST INTEGRATION PHASE .................................................................................................... 11

2. INITIAL CONCEPTS ....................................................................................................................... 12

2.1 PILOT SITE TASKS ...................................................................................................................................... 13

2.1.1 Data Access ....................................................................................................................................... 13

2.1.2 Data Mapping ..................................................................................................................................... 14

2.1.3 Import data ......................................................................................................................................... 16

2.2 BIND PILOT SOS DATA TO WDW ................................................................................................................ 16

2.3 GENERAL MODULE DESIGN CONSIDERATIONS ................................................................................................ 19

3. ACA SITE INTEGRATION .............................................................................................................. 21

3.1 WATERONEFLOW ...................................................................................................................................... 21

3.1.1 History of WaterOneFlow ................................................................................................................... 21

3.1.2 Structure of WaterOneFlow ................................................................................................................ 22

3.1.3 Web Service Interface ........................................................................................................................ 22

3.1.3.1 getSites...................................................................................................................................... 22

3.1.3.2 getSitesXML .............................................................................................................................. 26

3.1.3.3 getSiteInfo ................................................................................................................................. 27

3.1.3.4 getSiteInfoObject ....................................................................................................................... 29

3.1.3.5 getValues................................................................................................................................... 32

3.1.3.6 getValuesObject ........................................................................................................................ 35

3.1.3.7 getVariableInfo .......................................................................................................................... 40

3.1.3.8 getVariableInfoObject ................................................................................................................ 41

3.2 MAPPING TO WATERML2 ........................................................................................................................... 43

3.3 ACA PILOT INTEGRATION MANAGER ........................................................................................................... 46

3.4 NEXT STEPS .............................................................................................................................................. 48

4. SWKA SITE INTEGRATION ........................................................................................................... 49

4.1 EXISTING DATA INFRASTRUCTURE ................................................................................................................ 49


4.2 MAPPING TO WATERML2 ........................................................................................................................... 50

4.3 SWKA PILOT INTEGRATION MANAGER ........................................................................................................ 51

4.3.1 Mapping to WaterML2 ........................................................................................................................ 52

4.3.2 Load Sensor Metadata ....................................................................................................................... 53

4.3.3 Configure New Sensor ....................................................................................................................... 53

4.3.4 Trigger Import/InsertObservation ........................................................................................................ 55

4.4 NEXT STEPS ............................................................................................................................................. 56

5. CONCLUSIONS AND FUTURE WORK ......................................................................................... 57

6. APPENDIX I: OBSERVATION DATA MODEL (ODM) OF THE CUAHSI HIS .............................. 58

7. APPENDIX II: WATERONEFLOW SERVICE ................................................................................ 59


Table of figures

FIGURE 1 "WATERP ARCHITECTURE FROM THE INTEGRATION POINT OF VIEW" .................................................................. 11

FIGURE 2 "INTEGRATION OF EXISTING PILOT INFRASTRUCTURE" ....................................................................................... 12

FIGURE 3 "INTEGRATION OF AN EXISTING SOS INFRASTRUCTURE" ................................................................................... 13

FIGURE 4 "MAPPING EXISTING OBSERVATION DATA" ........................................................................................................ 14

FIGURE 5 "PILOT SITE ACCESS BY THE WDW AND FURTHER PROCESSING" ...................................................................... 17

FIGURE 6 "GENERAL MODULE DESIGN" .......................................................................................................................... 20

FIGURE 7 "SIMPLIFIED WATERONEFLOW STRUCTURE" ................................................................................................... 22

FIGURE 8 "QUERIED ENTITIES FOR GETSITES/GETSITESXML" ......................................................................................... 23

FIGURE 9 "QUERIED ENTITIES FOR GETSITEINFO/GETSITEINFOOBJECT" ........................................................................... 27

FIGURE 10 "QUERIED ENTITIES FOR GETVALUES/GETVALUESOBJECT" ............................................................................. 33

FIGURE 11 "QUERIED ENTITIES FOR GETVARIABLEINFO/GETVARIABLEINFOOBJECT" .......................................................... 40

FIGURE 12 "SUMMARY OF THE OGC OBSERVATIONS AND MEASUREMENTS (O&M) STANDARD" ......................................... 44

FIGURE 13 "PROCESS FLOW DURING SENSOR POLLING" .................................................................................................. 48

FIGURE 14 "POSSIBLE INSTALLATION SCENARIO FOR ACA" ............................................................................................. 49

FIGURE 15 “DATABASE TABLE FROM PILOT SITE SWKA” ................................................................................................. 50

FIGURE 16 "SWKA LOGICAL MODEL" ............................................................................................................................ 51

FIGURE 17 "FLOW CHART OF CLIENT APPLICATION" ........................................................................................................ 52

FIGURE 18 “CONFIGURATION SETTINGS FOR PIM”......................................................................................................... 53

FIGURE 19 "POSSIBLE INSTALLATION SCENARIO FOR SWKA" .......................................................................................... 56

FIGURE 20 "OBSERVATION DATA MODEL CUAHSI HIS" ................................................................................................ 58

FIGURE 21 "WATERONEFLOW SERVICE DIAGRAM" ......................................................................................................... 59


Table of tables

TABLE 1 "PARAMETERS FOR REGISTRATION OF A SENSOR IN THE WDW" .......................................................................... 18

TABLE 2 "GETSITES REQUEST PARAMETERS" ................................................................................................................. 23

TABLE 3 "GETSITESXML REQUEST PARAMETERS" .......................................................................................................... 26

TABLE 4 "GETSITEINFO REQUEST PARAMETERS" ............................................................................................................ 28

TABLE 5 "GETSITEINFOOBJECT REQUEST PARAMETERS" ................................................................................................. 29

TABLE 6 "GETVALUES REQUEST PARAMETERS" .............................................................................................................. 33

TABLE 7 "GETVALUESOBJECT REQUEST PARAMETERS" ................................................................................................... 36

TABLE 8 "GETVARIABLEINFO REQUEST PARAMETERS" ..................................................................................................... 40

TABLE 9 "GETVARIABLEINFOOBJECT REQUEST PARAMETERS" ......................................................................................... 42

TABLE 10 "MAIN ENTITIES IN CUAHSI AND OGC" ......................................................................................................... 45

TABLE 11 "COLUMNS OF SENSOR TABLE IN ACA PILOT INTEGRATION MANAGER" ............................................................... 46


Table of listings

LISTING 1 "REGISTER SENSOR IN SOS SERVER" ........................................................................................................... 15

LISTING 2 "INSERT AN OBSERVATION INTO THE SOS SERVER" .......................................................................................... 16

LISTING 3 "REGISTER PILOT SENSOR IN WDW" .............................................................................................................. 18

LISTING 4 “GETSITES REQUEST EXAMPLE” ..................................................................................................................... 23

LISTING 5 "GETSITES RESPONSE EXAMPLE" ................................................................................................................... 25

LISTING 6 "GETSITESXML REQUEST EXAMPLE" .............................................................................................................. 26

LISTING 7 "GETSITESXML RESPONSE EXAMPLE" ............................................................................................................ 27

LISTING 8 "GETSITEINFO REQUEST EXAMPLE" ................................................................................................................ 28

LISTING 9 "GETSITEINFO RESPONSE EXAMPLE" .............................................................................................................. 29

LISTING 10 "GETSITEINFOOBJECT REQUEST EXAMPLE" ................................................................................................... 30

LISTING 11 "GETSITEINFOOBJECT RESPONSE EXAMPLE" ................................................................................................. 32

LISTING 12 "GETVALUES REQUEST EXAMPLE" ................................................................................................................ 33

LISTING 13 "GETVALUES RESPONSE EXAMPLE" .............................................................................................................. 35

LISTING 14 "GETVALUESOBJECT REQUEST EXAMPLE" ..................................................................................................... 36

LISTING 15 "GETVALUESOBJECT RESPONSE EXAMPLE" ................................................................................................... 40

LISTING 16 "GETVARIABLEINFO REQUEST EXAMPLE" ....................................................................................................... 41

LISTING 17 "GETVARIABLEINFO RESPONSE EXAMPLE" ..................................................................................................... 41

LISTING 18 "GETVARIABLEINFOOBJECT REQUEST EXAMPLE" ........................................................................................... 42

LISTING 19 “GETVARIABLEINFOOBJECT RESPONSE EXAMPLE” ......................................................................................... 43

LISTING 20 "OUTPUT IN WATERML2 FORMAT" ............................................................................................................... 53

LISTING 21 "REQUEST TO REGISTER A SENSOR" ............................................................................................................. 54

LISTING 22 “RESPONSE FROM SOS FOR SENSOR REGISTRATION” .................................................................................... 54

LISTING 23 "REQUEST TO SOS FOR INSERTOBSERVATION" ............................................................................................ 55

LISTING 24 “RESPONSE FROM SOS” ............................................................................................................................. 56


1. Introduction

This document describes the activities to implement the Water Data Warehouse (WDW) on the pilots’

site. It shows what actions have been taken and which tools had to be implemented in order to integrate

the existing data structures. Some text passages can also be found in other documents but are added

to this document for the sake of self-containment. The technical descriptions of tools that were

implemented for integration of the existing data infrastructure are described in more depth in D3.3-

“WDW 2nd

Prototype” which is scheduled for delivery on month 18.

This document will first give a brief overview over the WatERP systems and will explain how pilots are

to be integrated from the architectural point of view. It will describe what general actions are required on

the pilot site and what fundamental issues have to be taken into consideration.

Next, the document gives a detailed description of each pilot’s data infrastructure and what strategy has

been implemented to tie the client to the WDW platform. The decisions are elucidated that have been

drawn so far.

Finally, the document will summarize the state that has been reached so far and explain the future

steps that still have to be taken.

1.1 WatERP overview

As also described in D3.2-“WDW 1st Prototype” the WDW is the central component of the WatERP

software and data infrastructure. Its main function is to work as a reliable and durable data basis for all

other components, providing both data needed for analyses or other functions as well as automated

routines to incorporate new data sources and datasets at any given time. The content is available for

the analysis clients via the Multi Agent System (MAS). The WDW provides both ontology information

and the observation results.

Figure 1 gives an overview over the different layers of the WatERP architecture and, especially, shows

how the layers interact. A more detailed description of the WatERP architecture can be found in D2.3-

“Open Interface Specification” chapter 3-“WatERP architecture”.


Figure 1 "WatERP architecture from the integration point of view"

The integration of the pilot data is realized via an SOS server that contains both ontology metadata and

observation results.

1.2 Standards for information interchange

As was decided in the D2.1-“External systems” chapter 5, all exchange of observation data is based on

WaterML2 time series. This standard provides an XML based combination of ontology and observation

results. For the pilot integration this means that the sensor data has to be transformed into WaterML2

and stored into an SOS server that is accessed from the WDW to import the time series.

This way the WDW has only to deal with a single data format when it accesses the pilot infrastructure.

Every transformation and mapping has to be accomplished by the Pilot Integration Manager (PIM)

which is implemented depending on the pilots’ needs.

1.3 Scope of the first integration phase

For PIM and WDW this first phase is a proof-of-concept as it is the first vertical integration. There are

several tasks that have to be accomplished. One is to identify and access the data containing the

WDW

SOS server

SOS client

On

tolo

gy

MAS

DSS

OGC WPS server

DMS

OGC WPS server

Hydrological forecast

OGC WPS server

SWKA Pilot

OGC SOS Server

ACA Pilot

OGC SOS Server

WatERP Framework

Pilot Data integration

Building Block Integration

OMP

WaterML2

WaterML2, OWL, drl WaterML2, OWL WaterML2


observation results. Next is to map the pilot data to the set of ontology metadata that is required by the

SOS/WaterML standard. These sets of metadata consist of feature of interest, sensor id,

observed property and offering (which directly relates to observed property). Other

information which has to be provided is the location of the feature of interest because it is required

by the SOS protocol and because it is mandatory to enable geospatial reasoning.

This deliverable covers an analysis of the data structures of each pilot. It describes the technical and

conceptional considerations about the way the integration can be achieved. Next, it describes the tools

that had to be implemented to access and convert the data in order to make it available within the SOS

server. Last, it shows by processing a subset of the pilots’ data that the different parts of PIM and WDW

correctly interact with each other and that the pilot data is in the end available for the MAS, and thereby

for the analysis clients.

2. Initial Concepts

One of the most important conception decisions that have been drawn in D3.1-“WDW Conceptual

Design” was that there is a strict separation between the pilot site and the WDW. It was decided that the

pilot data should be transformed into WaterML2 and stored in an SOS server. The architectural benefit

is that, thereby, there is no direct dependency from the WDW to the pilot site. As a consequence, the

WDW and all the other work packages would then be independent of the pilots’ actual data

infrastructure, but would only depend on the content provided by the SOS server. Figure 2 describes

how the pilot integration module serves as an adapter that transforms the existing sensor data into

WaterML2 which is then published via an SOS server.

Figure 2 "Integration of existing pilot infrastructure"

In case of an organization which would directly feed the SOS server from their sensor-network

infrastructure – as described in OGC Sensor Web Enablement (SWE)1 – there would not be a need for

1 Sensor Web Enablement: http://www.opengeospatial.org/ogc/markets-technologies/swe


any kind of integration tool as the WDW could directly access the SOS server that collects the

observation results. Figure 3 shows there is no need for any additional module because the sensor

network directly populates the SOS server that also serves as an interface to the WDW.

Figure 3 "Integration of an existing SOS infrastructure"

This way the WDW design would fit perfectly into an infrastructure that follows the SOS standard

published by OGC. Since neither the ACA nor the SWKA pilot meet these requirements, a separate

integration module had to be implemented. The following chapter describes the different steps that have

to be performed in order to integrate an existing observation-data infrastructure into an SOS-based

environment.

2.1 Pilot site tasks

Integration of an existing observation-data infrastructure has to be done in three different steps, namely

(1) the data access, (2) the data mapping, and (3) the data import.

2.1.1 Data Access

First, it has to be determined where and in which format the data is being kept. A procedure to access

these data has to be decided for the identified data sources. At this point, we had to take into considera-

tion three goals. Access time is one critical matter as we need to provide the observation results as

close to real-time as possible. For each pilot it will be discussed how close we can come to this goal.

Another goal is that in case there is a way to access client data over a standard interface, the

integration module should use it in order to improve the reusability of the implementation for future

pilots. Last, the integration should not compromise the stability of the existing infrastructure.This point

will also be discussed for each pilot.


2.1.2 Data Mapping

The next step after considering how to access the data is mapping the data to the ontology which is

required by the SOS and WaterML2 protocol.

Figure 4 "Mapping existing observation data"

As Figure 4 shows, the time series extracted from the pilot site have to be related to the elements that

the SOS standard requires:

Feature of interest which is the object which is being observed. Besides other attributes

it also contains the location which is essential for the geospatial operations.

Sensor which serves as an identifier

Observed property and Offering are both related to the sensor and describe what

property the sensor observes and what data the sensor offers. This also includes the

measured unit that the time series provides.

This task strongly depends on the existing information available on the pilot site. Again, in an OGC

SWE infrastructure this task is not needed since it is already done. After the mapping has been

decided, the sensor can be registered in the SOS server. Listing 1 shows the XML document with which

a new sensor is being registered.

<?xml version="1.0" encoding="UTF-8"?> <RegisterSensor service="SOS" version="1.0.0"

xmlns="http://www.opengis.net/sos/1.0" xmlns:swe="http://www.opengis.net/swe/1.0.1"

xmlns:ows="http://www.opengeospatial.net/ows" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc"

xmlns:om="http://www.opengis.net/om/1.0" xmlns:sml="http://www.opengis.net/sensorML/1.0.1"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/sos/1.0

http://schemas.opengis.net/sos/1.0.0/sosRegisterSensor.xsd

http://www.opengis.net/om/1.0 http://schemas.opengis.net/om/1.0.0/extensions/observationSpecialization_override.xsd"> <SensorDescription>

<sml:SensorML version="1.0.1">

<sml:member> <sml:System xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<sml:identification>

<sml:IdentifierList> <sml:identifier>

<sml:Term definition="urn:ogc:def:identifier:OGC:uniqueID">


<sml:value>urn:ogc:object:feature:Sensor:DEMO:sensor-1

</sml:value> </sml:Term>

</sml:identifier>

</sml:IdentifierList> </sml:identification>

<sml:capabilities>

<swe:SimpleDataRecord> <swe:field name="FeatureOfInterestID">

<swe:Text definition="FeatureOfInterest identifier">

<swe:value>SAMPLE-FOI</swe:value> </swe:Text>

</swe:field>

</swe:SimpleDataRecord>

</sml:capabilities>

<sml:inputs>

<sml:InputList> <sml:input name="waterpressure">

<swe:ObservableProperty definition="urn:ogc:def:phenomenon:waterpressure" />

</sml:input> </sml:InputList>

</sml:inputs>

<sml:outputs> <sml:OutputList>

<sml:output name=" waterpressure ">

<swe:Quantity definition="urn:ogc:def:phenomenon:waterpressure"> <gml:metaDataProperty>

<offering>

<id>WATERPRESSURE</id> <name>WATERPRESSURE</name>

</offering>

</gml:metaDataProperty>

<swe:uom code="bar" />

</swe:Quantity>

</sml:output> </sml:OutputList>

</sml:outputs>

<sml:components> <sml:ComponentList>

</sml:ComponentList>

</sml:components> </sml:System>

</sml:member>

</sml:SensorML> </SensorDescription>

<ObservationTemplate>

<om:Measurement> <om:samplingTime />

<om:procedure />

<om:observedProperty /> <om:featureOfInterest>

<sa:SamplingPoint gml:id="X:SAMPLE-FOI"> <gml:name>Demo-FOI</gml:name>

<sa:sampledFeature xlink:href="urn:ogc:def:nil:OGC:unknown" />

<sa:position> <gml:Point>

<gml:pos srsName="EPSG:4258">40.66040061445847 -5.420709255985781</gml:pos>

</gml:Point> </sa:position>

</sa:SamplingPoint>

</om:featureOfInterest> <om:result uom="bar">1</om:result>

</om:Measurement>

</ObservationTemplate> </RegisterSensor>

Listing 1 "Register Sensor in SOS Server"


2.1.3 Import data

The last step is to implement a tool which

- accesses the observation results from the identified sources,

- transforms them into SOS/WaterML2 format based on the mapping strategy considered in the

previous step, and

- stores them in the SOS server which is the gateway to the WDW. Here the loading of the data

should be incremental to avoid that all data has to be reprocessed every time new content is

added.

Listing 2 shows how an observation can be inserted into the SOS server:

<?xml version="1.0" encoding="UTF-8"?> <sos:InsertObservation service="SOS" version="1.0.0"

xmlns:sos="http://www.opengis.net/sos/1.0" xmlns:swe="http://www.opengis.net/swe/1.0.1"

xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:InsertObservation="http://Integration.WatERP.eu/InsertObservation"

xmlns:om="http://www.opengis.net/om/1.0" xmlns:ows="http://www.opengis.net/ows/1.1"

xmlns:gml="http://www.opengis.net/gml" xmlns:sa="http://www.opengis.net/sampling/1.0" xmlns:ogc="http://www.opengis.net/ogc">

<sos:AssignedSensorId>urn:ogc:object:feature:Sensor:DEMO:sensor-1 </sos:AssignedSensorId>

<om:Measurement>

<om:samplingTime>

<gml:TimeInstant>

<gml:timePosition>2013-02-06T09:30:00.000+01:00</gml:timePosition>

</gml:TimeInstant> </om:samplingTime>

<om:procedure xlink:href="urn:ogc:object:feature:Sensor:DEMO:sensor-1" />

<om:observedProperty xlink:href="urn:ogc:def:phenomenon:waterpressure" /> <om:featureOfInterest>

<sa:SamplingPoint gml:id="X:SAMPLE-FOI">

<gml:name>Demo-FOI </gml:name> <sa:sampledFeature xlink:href="urn:ogc:def:nil:OGC:unknown" />

<sa:position>

<gml:Point> <gml:pos srsName="EPSG:4258">40.66040061445847 -5.420709255985781

</gml:pos>

</gml:Point> </sa:position>

</sa:SamplingPoint>

</om:featureOfInterest>

<om:result uom="bar">2.3</om:result>

</om:Measurement>

</sos:InsertObservation>

Listing 2 "Insert an observation into the SOS server"

2.2 Bind pilot SOS data to WDW

Figure 5 gives an overview of how the WDW accesses the pilot site and how the WaterML2 documents

are processed. The basic steps of this process are:

1. The PIM registers the pilots’ sensors in the SOS server and adds the observations.

2. The sensors registered in the SOS server are also registered in the WDW.


3. The WDW polls the SOS server for new observation results and stores the observations in the

WDW SOS server.

4. The SOS data is also used to populate the entities in the triple store.

Figure 5 "Pilot Site Access by the WDW and further processing"

After registering the pilots’ sensors in the SOS server and adding the observation time series, the

sensors have to be registered within the WDW. For this registration the information shown in Table 1 is

required:


Information Description

Server URL The URL of the SOS server where the sensor is registered on the pilot site.

Example: http://localhost:4711/52nSOSv3.5.0_EE/sos

Observed

property Observed property as used in the mapping

Offering Offering as used in the mapping

Sensor id Sensor id as used in the mapping

Feature of

interest Feature of interest as used in the mapping

Poll interval The interval in minutes, in which the SOS server is polled for new data

Table 1 "Parameters for registration of a sensor in the WDW"

To perform the actual registration the WDW provides a management REST service that accepts XML.

The service allows adding sensors and sensor processing to the WDW.

Listing 3 is an example for an addSensor call to the WDW server:

<?xml version="1.0" encoding="UTF-8"?> <addSensorRequest xmlns="http://waterp/wdw/ManagementService">

<observedProperty>urn:ogc:def:phenomenon:waterpressure</observedProperty>

<offering>WATERPRESSURE</offering> <sensorId>urn:ogc:object:feature:Sensor:DEMO:sensor-1</sensorId>

<pollIntervalMinutes>60</pollIntervalMinutes>

<featureOfInterest>Demo-FOI</featureOfInterest> <serverUrl>http://pilot.waterp.eu/sosserver/sos</serverUrl>

</addSensorRequest>

Listing 3 "Register pilot sensor in WDW"

After the registration of the sensor the WDW will poll the pilot SOS server for new observation results

for the sensor. Additionally, the WDW will add the sensor to the WDW SOS infrastructure in order to

offer the observation time series in the WDW interface layer. See D3.2-“WDW 1st Prototype” and D3.3-

“WDW 2nd

Prototype” for a more detailed description of the management service and the WDW internal

processing.

As described in D1.4.1-“Extension of taxonomy and ontology to the pilots” chapter 2.2-“Initial Large-

scale population strategy”, the information contained in the WaterML2 XML documents will later be

incorporated into the WaterERP ontology which will then be stored into the WDW triple store.


2.3 General module design considerations

The general module design is described in D3.3-“WDW 2nd

Prototype” (due in month 18, March 2014) in

more detail but as this document should be self-contained and D3.3 will be available after D7.3.1 the

basic design of the pilot integration modules will be briefly explained here.

Chapter 2.1 listed the different tasks that have been implemented during the pilot integration. The steps

differ regarding if they are context specific:

Data Access:

When the investigation of the pilots started it became clear that the infrastructure to be integrated into

the WDW, is highly differing. At SWKA, the PIM must deal with a rather simple database table whereas

at ACA, things were rather unclear in the beginning – but turned out later to be a WaterOneFlow service

call. For the design of the PIM it became apparent that it would have to allow a highly pilot specific

implementation of the data access. For other pilots the integration might require:

Reading the data from a single or multiple data bases as in SWKA.

File parsing.

Calling services as with ACA. Here format and protocol can highly differ.

Screen scraping or embedding legacy code into the integration module.

A mix of the access mechanisms above.

Consequently, the general integration module design must enable the data-access layer to be highly

flexible. There is a possibility that the implementation for one pilot can be used for other pilots as well in

case the pilots use the same infrastructure to make their observation results available.

Data Mapping:

As with the data access there are many possible ways how the pilots’ data can be converted into

WaterML2 documents:

The data that is required to populate the WaterML document might already be available via

database or a service call but has to be transformed.

If not all required data is available in the pilot infrastructure it has to be provided by the

integration module.

As described in chapter 3.2 for ACA the main information for the data mapping is provided by the

existing data infrastructure. In contrast for SWKA the PIM must provide most of the mapping

information. The implementation depends on the information the data access supplies. In consequence,

for the general PIM design the same level of flexibility is required for the data mapping as is for the data

access.


Import Data

Here the mapping information and the time series have to be transformed into WaterML2 and sent to

the SOS server as the external interface to the WDW for registration and storing of the observations.

For both, ACA and SWKA, the result of data access and mapping consists of the same set of

information. Therefore in both scenarios the generation of WaterML2 is identical as it is based on the

same mapped data. Also the the interactions with the SOS server are identical.

In consequence for the WatERP project context the data import is considered to not require any pilot

specific implementation. To enable the interaction of data import with data access and data mapping,

which are highly pilot dependent, classes are implemented that contain the set of information about

mapping time series that is required for the WaterML generation. The combination of data access and

data mapping has to provide this transfer objects and pass them to the data import layer.

If for a future integration activities with other pilots the set of informations to be added to WaterML2

changes or becomes client specific, this decision will have to be reconsidered.

Figure 6 "General module design"

Figure 6 shows the general module design. Data access and data mapping are abstract but have to

populate the time series and mapping information using a predefined structure. The data export which


consumes the data is independent on the actual pilot context as the pilot specifics have no influence on

the export processing.

3. ACA site integration

D1.4.1-“Extension of taxonomy and ontology to the pilots” chapter 2.1.1 explains the ACA ontology, the

logical model and required queries to the triple store. D4.1-“Inference and Simulation Engine

Conceptual Design” describes in chapter 3.1 the scenario for ACA. It deduces the logical model

(chapter 3.1.1) that describes the situation and the entities and variables that must be available as the

model input (chapter 3.1.2). Because the sensor model created inside the pilot SOS server will serve as

the base on which the ontology will be generated, the integration of ACA sensor-data infrastructure

must provide the data on which all other models can be established.

This chapter describes the considerations and activities to integrate the ACA sensor data into the

WDW. It first depicts the current infrastructure that the PIM has to integrate. Next, it analyzes how ACA

data can be mapped to WaterML2 and explains the current level of implementation. Finally, it lists the

next steps in order to integrate ACA into the WDW.

3.1 WaterOneFlow

For some time it was rather unclear how the observation data from ACA could be integrated into the

WDW. In autumn 2013, ACA installed a WaterOneFlow server as a central instance to obtain sensor-

observation results. Currently, the server provides a considerable number of sensor-observation results.

In January 2014, the server provided alone 12,883 sites. As stated in chapter 2.1.1 there are three

issues to be taken into consideration when deciding about how to access the pilots’ data: Performance,

reusability and stability. As WaterOneFlow was designed and installed by ACA as a central point to

access observation results and is long and often used in the hydrological community it meets the

requirements of reusability and stability. The prototype will show if the performance will suffice the

demand. In case critical performance problems appear alternative approaches will have to be

considered together with ACA.

This chapter will describe the history and the data structure of WaterOneFlow as well as the web

service interface by which the data from the ACA can be accessed.

3.1.1 History of WaterOneFlow

WaterOneFlow was developed by the Consortium of the Advanced of Hydrological Sciences Inc.

(CUAHSI) for use in U.S. as a standard mechanism for the transfer of hydrologic data. WaterOneFlow

web service uses WaterML1 for encoding hydrological observations. WaterML2 is significantly different

from WaterML1 as WaterML2 is based on OGC standards Sensor Observation Service (SOS),

Observations and Measurements (O&M) and Sensor Markup Language (SensorML). Moreover,


WaterML2 has been defined by the Hydrology Domain Working Group (Hydro DWG), a workgroup

within OGC. The group had the task to define an exchange standard for hydrological data based on

WaterML1 (CUAHSI) and Water Data Transfer Format (WDTF, Australia). ACA pilot data is currently in

the form of WaterML1 and published on the CUAHSI web service WaterOneFlow.

3.1.2 Structure of WaterOneFlow

The structure of the data that is transferred over WaterOneFlow is very similar to the Observation Data

Model (ODM) of the CUAHSI Hydrologic Information System (HIS)2. A full diagram of the ODM is

shown in section 6 (Appendix I).

Figure 7 shows the simplified structure of WaterOneFlow describing only the main entities. In section

3.1.3 each available service method is described. The chapter will also explain for every method what

entities are returned.

Figure 7 "Simplified WaterOneFlow structure"

3.1.3 Web Service Interface

A summarizing diagram of all web service methods is shown in Appendix II: WaterOneFlow Service. In

this section all methods will be explained using the ACA installation. The security concept states that for

a protected installation the client has to create a security token before the real business call can be

performed. The following list will only describe the business methods, not the additional authorization

calls. For each existing method it will describe the available parameters, show a sample request and

the resulting response document received from the ACA server. In case the response contains a large

series of repeating structures which wouldn’t provide any additional insights these repeating elements

are replaced by ‘…’.

3.1.3.1 getSites

The getSites method provides a description of the sites.

2 http://his.cuahsi.org/odmdatabases.html


Figure 8 "Queried entities for getSites/getSitesXML"

Table 2 contains the list of parameters that can be passed to the getSites call.

Parameter Description

site List of site identifiers in the format ‘NetworkName:SiteCode'.

authToken Previously created secutity token.

Table 2 "getSites request parameters"

Both parameters are optional. If no site is specified, all existing sites will be returned. Listing 4 contains

a getSites request example. Two sites are passed.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:q0="http://ws.waterOneFlow.aca.gencat.cat" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <q0:GetSites> <q0:site>S:430141-002</q0:site> <q0:site>D:AUTOD1</q0:site> <q0:authToken /> </q0:GetSites> </soapenv:Body> </soapenv:Envelope>

Listing 4 “getSites request example”

Listing 5 shows the result of the sample request. In summary getSites provides only the information

directly connected to the site such as name and location (as shown in Figure 8). For more detailed

information getSiteInfo has to be called.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <GetSitesResponse xmlns="http://ws.waterOneFlow.aca.gencat.cat"> <GetSitesReturn> <error xsi:nil="true" /> <queryInfo> <creationTime>2014-02-22T10:52:20.831Z</creationTime> <criteria> <locationParam xsi:nil="true" /> <methodCalled>GetSites</methodCalled> <parameter> <parameter>


<name>site</name> <value>S:430141-002</value> </parameter> <parameter> <name>site</name> <value>D:AUTOD1</value> </parameter> </parameter> <timeParam xsi:nil="true" /> <variableParam xsi:nil="true" /> </criteria> <extension xsi:nil="true" /> <note /> <queryURL xsi:nil="true" /> </queryInfo> <site> <site> <extension /> <seriesCatalog /> <siteInfo> <altname xsi:nil="true" /> <elevationM xsi:nil="true" /> <error xsi:nil="true" /> <extension xsi:nil="true" /> <geoLocation> <geogLocation> <srs>EPSG:4258</srs> </geogLocation> <localSiteXY> <localSiteXY> <note /> <projectionInformation>ED50 / UTM zone 29N </projectionInformation> <x>295364.0</x> <y>4503877.0</y> <z xsi:nil="true" /> </localSiteXY> </localSiteXY> </geoLocation> <metadataTime xsi:nil="true" /> <note /> <oid xsi:nil="true" /> <siteCode> <siteCode> <agencyCode xsi:nil="true" /> <agencyName xsi:nil="true" /> <network>S</network> <siteID xsi:nil="true" /> <value>430141-002</value> </siteCode> </siteCode> <siteName>QLSup(A) - E.Q.1 Amposta M.D. Riu (CHE) (RIADE) </siteName> <siteProperty> <siteProperty> <name>County</name> <type xsi:nil="true" /> <uri xsi:nil="true" /> <value>Tarragona</value> </siteProperty> <siteProperty> <name>State</name> <type xsi:nil="true" /> <uri xsi:nil="true" /> <value>N/D</value> </siteProperty> </siteProperty> <siteType /> <timeZoneInfo xsi:nil="true" />


<verticalDatum>REDNAP</verticalDatum> </siteInfo> </site> <site> <extension /> <seriesCatalog /> <siteInfo> <altname xsi:nil="true" /> <elevationM xsi:nil="true" /> <error xsi:nil="true" /> <extension xsi:nil="true" /> <geoLocation> <geogLocation> <srs>EPSG:4258</srs> </geogLocation> <localSiteXY> <localSiteXY> <note /> <projectionInformation>ED50 / UTM zone 29N </projectionInformation> <x>328794.0</x> <y>4555677.0</y> <z xsi:nil="true" /> </localSiteXY> </localSiteXY> </geoLocation> <metadataTime xsi:nil="true" /> <note /> <oid xsi:nil="true" /> <siteCode> <siteCode> <agencyCode xsi:nil="true" /> <agencyName xsi:nil="true" /> <network>D</network> <siteID xsi:nil="true" /> <value>AUTOD1</value> </siteCode> </siteCode> <siteName>Consum - Riudecanyes (captació regants de Riudecanyes) </siteName> <siteProperty> <siteProperty> <name>County</name> <type xsi:nil="true" /> <uri xsi:nil="true" /> <value>Tarragona</value> </siteProperty> <siteProperty> <name>State</name> <type xsi:nil="true" /> <uri xsi:nil="true" /> <value>N/D</value> </siteProperty> </siteProperty> <siteType /> <timeZoneInfo xsi:nil="true" /> <verticalDatum>REDNAP</verticalDatum> </siteInfo> </site> </site> </GetSitesReturn> </GetSitesResponse> </soapenv:Body> </soapenv:Envelope>

Listing 5 "getSites response example"


3.1.3.2 getSitesXML

The method getSitesXML works very similar to getSites, but it separates the SOAP document

from the site information by embedding the sites XML as a string in the SOAP document.

Table 3 lists the parameters that can be passed with the getSitesXML request.


site List of site identifiers in the format ‘NetworkName:SiteCode'.


Table 3 "getSitesXML request parameters"

Listing 6 shows a sample request of type getSitesXML and Listing 7 exposes the resulting response.

It shows that the sitesResponse is embedded as a string inside the GetSitesXmlReturn.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:q0="http://ws.waterOneFlow.aca.gencat.cat" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <q0:GetSitesXml> <q0:site>S:430141-002</q0:site> <q0:site>D:AUTOD1</q0:site> <q0:authToken /> </q0:GetSitesXml> </soapenv:Body> </soapenv:Envelope>

Listing 6 "getSitesXML request example"

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <GetSitesXmlResponse xmlns="http://ws.waterOneFlow.aca.gencat.cat"> <GetSitesXmlReturn><?xml version="1.0" encoding="UTF-8" standalone="yes"?> <sitesResponse xmlns="http://www.cuahsi.org/waterML/1.1/"> <queryInfo> <creationTime>2014-02-22T10:49:43.396+00:00</creationTime> <criteria MethodCalled="GetSites"> <parameter value="S:430141-002" name="site"/> <parameter value="D:AUTOD1" name="site"/> </criteria> </queryInfo> <site> <siteInfo> <siteName>QLSup(A) - E.Q.1 Amposta M.D. Riu (CHE) (RIADE)</siteName> <siteCode network="S">430141-002</siteCode> <geoLocation> <geogLocation xsi:type="LatLonPointType" srs="EPSG:4258" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <latitude>40.66040061445847</latitude> <longitude>-5.420709255985781</longitude> </geogLocation> <localSiteXY projectionInformation="ED50 / UTM zone 29N"> <X>295364.0</X> <Y>4503877.0</Y> </localSiteXY> </geoLocation>


<verticalDatum>REDNAP</verticalDatum> <siteProperty name="County">Tarragona</siteProperty> <siteProperty name="State">N/D</siteProperty> </siteInfo> </site> <site> <siteInfo> <siteName>Consum - Riudecanyes (captació regants de Riudecanyes)</siteName> <siteCode network="D">AUTOD1</siteCode> <geoLocation> <geogLocation xsi:type="LatLonPointType" srs="EPSG:4258" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <latitude>41.13435372948381</latitude> <longitude>-5.0397950513708825</longitude> </geogLocation> <localSiteXY projectionInformation="ED50 / UTM zone 29N"> <X>328794.0</X> <Y>4555677.0</Y> </localSiteXY> </geoLocation> <verticalDatum>REDNAP</verticalDatum> <siteProperty name="County">Tarragona</siteProperty> <siteProperty name="State">N/D</siteProperty> </siteInfo> </site> </sitesResponse> </GetSitesXmlReturn> </GetSitesXmlResponse> </soapenv:Body> </soapenv:Envelope>

Listing 7 "getSitesXML response example"

3.1.3.3 getSiteInfo

Given a site number, this method returns the site's metadata.

Figure 9 "Queried entities for getSiteInfo/getSiteInfoObject"

As Figure 9 describes, all entities are queried. From the data values not the single observations are

returned but the minimum and maximum timestamp for the queried sensor. Table 4 shows the possible

request parameters.


site Site identifier in the format ‘NetworkName:SiteCode'.



Table 4 "getSiteInfo request parameters"

Listing 8 shows a sample request of type GetSiteInfo passing the site identifier of the queried site.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:q0="http://ws.waterOneFlow.aca.gencat.cat" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <q0:GetSiteInfo> <q0:site>S:430141-002</q0:site> <q0:authToken /> </q0:GetSiteInfo> </soapenv:Body> </soapenv:Envelope>

Listing 8 "getSiteInfo request example"

Listing 9 shows the result of the getSiteInfo request. Besides the basic site information which is also

provided by the getSites/getSitesXML requests, the response also contains the series that exist

for the site. For each series it lists the variable, the datasource, the quality and the time interval for

which data is available. Like with getSitesXML the sitesResponse is separated from the SOAP

XML structure by embedding it as a string inside the GetSiteInfoReturn tag.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <GetSiteInfoResponse xmlns="http://ws.waterOneFlow.aca.gencat.cat"> <GetSiteInfoReturn><?xml version="1.0" encoding="UTF-8" standalone="yes"?> <sitesResponse xmlns="http://www.cuahsi.org/waterML/1.1/"> <queryInfo> <creationTime>2014-02-22T09:27:12.342+00:00</creationTime> <criteria MethodCalled="GetSiteInfo"> <parameter value="S:430141-002" name="site"/> </criteria> </queryInfo> <site> <siteInfo> <siteName>QLSup(A) - E.Q.1 Amposta M.D. Riu (CHE) (RIADE)</siteName> <siteCode network="S">430141-002</siteCode> <geoLocation> <geogLocation xsi:type="LatLonPointType" srs="EPSG:4258" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <latitude>40.66040061445847</latitude> <longitude>-5.420709255985781</longitude> </geogLocation> <localSiteXY projectionInformation="ED50 / UTM zone 29N"> <X>295364.0</X> <Y>4503877.0</Y> </localSiteXY> </geoLocation> <verticalDatum>REDNAP</verticalDatum> <siteProperty name="County">Tarragona</siteProperty> <siteProperty name="State">N/D</siteProperty> </siteInfo> <seriesCatalog menuGroupName="Variables fenomenològiques de SIX"> <series> <variable> <variableCode default="true" vocabulary="S">3965408</variableCode> <variableName>Terbolesa</variableName> <valueType>Automàtic</valueType> <generalCategory>Fisicoquímics</generalCategory> <sampleMedium>Generals</sampleMedium> <unit>


<unitName>NTU</unitName> <unitType>NTU</unitType> <unitAbbreviation>NTU</unitAbbreviation> <unitCode>0001</unitCode> </unit> <noDataValue>-9999.0</noDataValue> <timeScale> <unit> <unitName>minute</unitName> <unitType>Time</unitType> <unitAbbreviation>min</unitAbbreviation> <unitCode>1000</unitCode> </unit> <timeSupport>0.0</timeSupport> </timeScale> <speciation></speciation> </variable> <variableTimeInterval xsi:type="TimeIntervalType" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <beginDateTime>2013-02-06T08:00:00+00:00</beginDateTime> <endDateTime>2014-02-18T20:00:00+00:00</endDateTime> <beginDateTimeUTC>2013-02-06T08:00:00+00:00</beginDateTimeUTC> <endDateTimeUTC>2014-02-18T20:00:00+00:00</endDateTimeUTC> </variableTimeInterval> <source sourceID="1"> <organization>Agència Catalana de l'Aigua</organization> <sourceDescription>Variables fenomenològiques de SIX</sourceDescription> <citation></citation> </source> <qualityControlLevel qualityControlLevelID="0"> <qualityControlLevelCode>0</qualityControlLevelCode> <definition>Raw data</definition> </qualityControlLevel> </series> <series> ... </series> </seriesCatalog> </site> </sitesResponse> </GetSiteInfoReturn> </GetSiteInfoResponse> </soapenv:Body> </soapenv:Envelope>

Listing 9 "getSiteInfo response example"

3.1.3.4 getSiteInfoObject

The getSiteInfoObject is very much like the getSiteInfo except that the response is directly

integrated in the SOAP body. Table 5 lists the parameters that can be passed in the request. Listing 10

shows an example request and Listing 11 shows the corresponding response. The information provided

is very much like in getSiteInfo and the explanation matches also here.


site Single site identifier in the format ‘NetworkName:SiteCode'.


Table 5 "getSiteInfoObject request parameters"


<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:q0="http://ws.waterOneFlow.aca.gencat.cat" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <q0:GetSiteInfoObject> <q0:site>S:430141-002</q0:site> <q0:authToken /> </q0:GetSiteInfoObject> </soapenv:Body> </soapenv:Envelope>

Listing 10 "getSiteInfoObject request example"

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <GetSiteInfoObjectResponse xmlns="http://ws.waterOneFlow.aca.gencat.cat"> <GetSiteInfoObjectReturn> <error xsi:nil="true" /> <queryInfo> <creationTime>2014-02-22T10:56:39.985Z</creationTime> <criteria> <locationParam xsi:nil="true" /> <methodCalled>GetSiteInfo</methodCalled> <parameter> <parameter> <name>site</name> <value>S:430141-002</value> </parameter> </parameter> <timeParam xsi:nil="true" /> <variableParam xsi:nil="true" /> </criteria> <extension xsi:nil="true" /> <note /> <queryURL xsi:nil="true" /> </queryInfo> <site> <site> <extension /> <seriesCatalog> <seriesCatalog> <extension xsi:nil="true" /> <menuGroupName>Variables fenomenològiques de SIX</menuGroupName> <note /> <series> <series> <dataType xsi:nil="true" /> <generalCategory xsi:nil="true" /> <method xsi:nil="true" /> <qualityControlLevel> <definition>Raw data</definition> <explanation xsi:nil="true" /> <qualityControlLevelCode>0</qualityControlLevelCode> <qualityControlLevelID>0</qualityControlLevelID> </qualityControlLevel> <sampleMedium xsi:nil="true" /> <seriesProperty /> <source> <citation /> <contactInformation /> <metadata xsi:nil="true" /> <organization>Agència Catalana de l'Aigua</organization> <sourceCode xsi:nil="true" /> <sourceDescription>Variables fenomenològiques de SIX</sourceDescription> <sourceID>1</sourceID> <sourceLink /> </source>


<valueCount xsi:nil="true" /> <valueType xsi:nil="true" /> <variable> <categories xsi:nil="true" /> <dataType xsi:nil="true" /> <error xsi:nil="true" /> <extension xsi:nil="true" /> <generalCategory>Fisicoquímics</generalCategory> <metadataTime xsi:nil="true" /> <noDataValue>-9999.0</noDataValue> <note /> <oid xsi:nil="true" /> <options xsi:nil="true" /> <related xsi:nil="true" /> <sampleMedium>Generals</sampleMedium> <speciation /> <timeScale> <isRegular>false</isRegular> <timeSpacing xsi:nil="true" /> <timeSupport>0.0</timeSupport> <unit> <unitAbbreviation>min</unitAbbreviation> <unitCode>1000</unitCode> <unitDescription xsi:nil="true" /> <unitID xsi:nil="true" /> <unitName>minute</unitName> <unitType>Time</unitType> </unit> </timeScale> <unit> <unitAbbreviation>NTU</unitAbbreviation> <unitCode>0001</unitCode> <unitDescription xsi:nil="true" /> <unitID xsi:nil="true" /> <unitName>NTU</unitName> <unitType>NTU</unitType> </unit> <valueType>Automàtic</valueType> <variableCode> <variableCode> <network xsi:nil="true" /> <value>3965408</value> <variableID xsi:nil="true" /> <vocabulary>S</vocabulary> </variableCode> </variableCode> <variableDescription xsi:nil="true" /> <variableName>Terbolesa</variableName> <variableProperty /> </variable> <variableTimeInterval> <beginDateTime>2013-02-06T08:00:00.000Z</beginDateTime> <beginDateTimeUTC>2013-02-06T08:00:00.000Z</beginDateTimeUTC> <endDateTime>2014-02-18T20:00:00.000Z</endDateTime> <endDateTimeUTC>2014-02-18T20:00:00.000Z</endDateTimeUTC> </variableTimeInterval> </series> <series> ... </series> </series> <seriesCatalogProperty /> <serviceWsdl xsi:nil="true" /> </seriesCatalog> </seriesCatalog> <siteInfo> <altname xsi:nil="true" /> <elevationM xsi:nil="true" /> <error xsi:nil="true" />


<extension xsi:nil="true" /> <geoLocation> <geogLocation> <srs>EPSG:4258</srs> </geogLocation> <localSiteXY> <localSiteXY> <note /> <projectionInformation>ED50 / UTM zone 29N </projectionInformation> <x>295364.0</x> <y>4503877.0</y> <z xsi:nil="true" /> </localSiteXY> </localSiteXY> </geoLocation> <metadataTime xsi:nil="true" /> <note /> <oid xsi:nil="true" /> <siteCode> <siteCode> <agencyCode xsi:nil="true" /> <agencyName xsi:nil="true" /> <network>S</network> <siteID xsi:nil="true" /> <value>430141-002</value> </siteCode> </siteCode> <siteName>QLSup(A) - E.Q.1 Amposta M.D. Riu (CHE) (RIADE) </siteName> <siteProperty> <siteProperty> <name>County</name> <type xsi:nil="true" /> <uri xsi:nil="true" /> <value>Tarragona</value> </siteProperty> <siteProperty> <name>State</name> <type xsi:nil="true" /> <uri xsi:nil="true" /> <value>N/D</value> </siteProperty> </siteProperty> <siteType /> <timeZoneInfo xsi:nil="true" /> <verticalDatum>REDNAP</verticalDatum> </siteInfo> </site> </site> </GetSiteInfoObjectReturn> </GetSiteInfoObjectResponse> </soapenv:Body> </soapenv:Envelope>

Listing 11 "getSiteInfoObject response example"

3.1.3.5 getValues

The getValues request queries the time series for a specific site and variable.


Figure 10 "Queried entities for getValues/getValuesObject"

As described in Figure 10 getValues and getValuesObject query all existing entities. Table 6

describes the parameters that can be passed with the getValues request and Listing 12 shows an

example request.


location Site identifier in the format ‘NetworkName:SiteCode'.

variable Variable identifier in the format ‘NetworkName:VariableCode’

beginDate Lower limit of the requested time interval in the format ‘YYYY-MM-DD’

endDate Upper limit of the requested time interval in the format ‘YYYY-MM-DD’

authToken Previously created secutity token

Table 6 "getValues request parameters"

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:q0="http://ws.waterOneFlow.aca.gencat.cat" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <q0:GetValues> <q0:location>S:430141-002</q0:location> <q0:variable>S:3965408</q0:variable> <q0:startDate>2013-02-06</q0:startDate> <q0:endDate>2013-02-07</q0:endDate> <q0:authToken /> </q0:GetValues> </soapenv:Body> </soapenv:Envelope>

Listing 12 "getValues request example"

Besides the time series the getValues response contains a description of the site and the variable

which have been queried. For each value it also provides the identifier for source, quality control

and procedure. The source, quality control and feature entities referenced by the values are

also embedded inside the response but are linked with the value tags to avoid redundant information,

and thereby reduce the document size.


Like with getSitesXML and getSiteInfo the timeSeriesResponse is separated from the SOAP

body and as a string embedded in the GetValuesReturn tag.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <GetValuesResponse xmlns="http://ws.waterOneFlow.aca.gencat.cat"> <GetValuesReturn><?xml version="1.0" encoding="UTF-8" standalone="yes"?> <timeSeriesResponse xmlns="http://www.cuahsi.org/waterML/1.1/"> <queryInfo> <creationTime>2014-02-22T12:17:39.500+00:00</creationTime> <criteria MethodCalled="GetValues"> <parameter value="S:430141-002" name="site"/> <parameter value="S:3965408" name="variable"/> <parameter value="2013-02-06" name="startDate"/> <parameter value="2013-02-07" name="endDate"/> </criteria> </queryInfo> <timeSeries> <sourceInfo xsi:type="SiteInfoType" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <siteName>QLSup(A) - E.Q.1 Amposta M.D. Riu (CHE) (RIADE)</siteName> <siteCode network="STR">430141-002</siteCode> <geoLocation> <geogLocation xsi:type="LatLonPointType" srs="EPSG:4258"> <latitude>40.66040061445847</latitude> <longitude>-5.420709255985781</longitude> </geogLocation> <localSiteXY projectionInformation="ED50 / UTM zone 29N"> <X>295364.0</X> <Y>4503877.0</Y> </localSiteXY> </geoLocation> <verticalDatum>REDNAP</verticalDatum> <note title="County">Tarragona</note> <note title="State">N/D</note> </sourceInfo> <variable> <variableCode default="true" vocabulary="STR">3965408</variableCode> <variableName>Terbolesa</variableName> <unit> <unitName>NTU</unitName> <unitType>NTU</unitType> <unitAbbreviation>NTU</unitAbbreviation> <unitCode>0001</unitCode> </unit> <noDataValue>-9999.0</noDataValue> <timeScale> <unit> <unitName>minute</unitName> <unitType>Time</unitType> <unitAbbreviation>min</unitAbbreviation> <unitCode>1000</unitCode> </unit> <timeSupport>0.0</timeSupport> </timeScale> <speciation></speciation> </variable> <values> <value qualityControlLevelCode="0" sourceCode="1" methodCode="0" dateTimeUTC="2013-02-06T15:00:00+00:00" timeOffset="+01:00" dateTime="2013-02-06T15:00:00+00:00" censorCode="nc">4</value> <value qualityControlLevelCode="0" sourceCode="1" methodCode="0" dateTimeUTC="2013-02-06T15:15:00+00:00" timeOffset="+01:00" dateTime="2013-02-06T15:15:00+00:00" censorCode="nc">4</value> ... <value qualityControlLevelCode="0" sourceCode="1" methodCode="0" dateTimeUTC="2013-02-


06T10:15:00+00:00" timeOffset="+01:00" dateTime="2013-02-06T10:15:00+00:00" censorCode="nc">4</value> <qualityControlLevel qualityControlLevelID="0"> <qualityControlLevelCode>0</qualityControlLevelCode> <definition>Raw data</definition> <explanation></explanation> </qualityControlLevel> <method methodID="0"> <methodCode>0</methodCode> <methodDescription>Not defined</methodDescription> </method> <source sourceID="1"> <sourceCode>1</sourceCode> <organization>Agència Catalana de l'Aigua</organization> <sourceDescription>Variables fenomenològiques de SIX</sourceDescription> <contactInformation> <contactName></contactName> <typeOfContact></typeOfContact> <email></email> <phone></phone> <address xsi:type="xs:string" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xs="http://www.w3.org/2001/XMLSchema"></address> </contactInformation> <sourceLink></sourceLink> <citation></citation> </source> <censorCode> <censorCode>nc</censorCode> <censorCodeDescription>not censored</censorCodeDescription> </censorCode> </values> </timeSeries> </timeSeriesResponse> </GetValuesReturn> </GetValuesResponse> </soapenv:Body> </soapenv:Envelope>

Listing 13 "getValues response example"

3.1.3.6 getValuesObject

The getValuesObject request is similar to the getValues request except that the time series are

directly embedded into the SOAP body. The document structures differ but regarding the content the

description of getValues applies also to getValuesObject.

Table 7 contains the list of available parameters for the getValuesObject request. Listing 14 shows

an example request and Listing 15 contains the response.


location Site identifier in the format ‘NetworkName:SiteCode'.

variable Variable identifier in the format ‘NetworkName:VariableCode’.

beginDate Lower limit of the requested time interval in the format ‘YYYY-MM-DD’.

endDate Upper limit of the requested time interval in the format ‘YYYY-MM-DD’.



Table 7 "getValuesObject request parameters"

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:q0="http://ws.waterOneFlow.aca.gencat.cat" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <q0:GetValuesObject> <q0:location>S:430141-002</q0:location> <q0:variable>S:3965408</q0:variable> <q0:startDate>2013-02-06</q0:startDate> <q0:endDate>2013-02-07</q0:endDate> <q0:authToken /> </q0:GetValuesObject> </soapenv:Body> </soapenv:Envelope>

Listing 14 "getValuesObject request example"

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <GetValuesObjectResponse xmlns="http://ws.waterOneFlow.aca.gencat.cat"> <GetValuesObjectReturn> <error xsi:nil="true"/> <queryInfo> <creationTime>2014-02-22T12:24:34.953Z</creationTime> <criteria> <locationParam xsi:nil="true"/> <methodCalled>GetValues</methodCalled> <parameter> <parameter> <name>site</name> <value>S:430141-002</value> </parameter> <parameter> <name>variable</name> <value>S:3965408</value> </parameter> <parameter> <name>startDate</name> <value>2013-02-06</value> </parameter> <parameter> <name>endDate</name> <value>2013-02-07</value> </parameter> </parameter> <timeParam xsi:nil="true"/> <variableParam xsi:nil="true"/> </criteria> <extension xsi:nil="true"/> <note/> <queryURL xsi:nil="true"/> </queryInfo> <timeSeries> <timeSeries> <name xsi:nil="true"/> <sourceInfo xmlns:ns1="http://_1.waterml.cuahsi.org" xsi:type="ns1:SiteInfoType"> <altname xsi:nil="true"/> <elevationM xsi:nil="true"/> <error xsi:nil="true"/> <extension xsi:nil="true"/> <geoLocation> <geogLocation> <srs>EPSG:4258</srs> </geogLocation>


<localSiteXY> <localSiteXY> <note/> <projectionInformation>ED50 / UTM zone 29N</projectionInformation> <x>295364.0</x> <y>4503877.0</y> <z xsi:nil="true"/> </localSiteXY> </localSiteXY> </geoLocation> <metadataTime xsi:nil="true"/> <note> <note> <href xsi:nil="true"/> <show xsi:nil="true"/> <title>County</title> <type xsi:nil="true"/> <value>Tarragona</value> </note> <note> <href xsi:nil="true"/> <show xsi:nil="true"/> <title>State</title> <type xsi:nil="true"/> <value>N/D</value> </note> </note> <oid xsi:nil="true"/> <siteCode> <siteCode> <agencyCode xsi:nil="true"/> <agencyName xsi:nil="true"/> <network>STR</network> <siteID xsi:nil="true"/> <value>430141-002</value> </siteCode> </siteCode> <siteName>QLSup(A) - E.Q.1 Amposta M.D. Riu (CHE) (RIADE)</siteName> <siteProperty/> <siteType/> <timeZoneInfo xsi:nil="true"/> <verticalDatum>REDNAP</verticalDatum> </sourceInfo> <values> <values> <censorCode> <censorCode> <censorCode>nc</censorCode> <censorCodeDescription>not censored</censorCodeDescription> <censorCodeID xsi:nil="true"/> </censorCode> </censorCode> <method> <method> <methodCode>0</methodCode> <methodDescription>Not defined</methodDescription> <methodID>0</methodID> <methodLink xsi:nil="true"/> </method> </method> <offset/> <qualifier/> <qualityControlLevel> <qualityControlLevel> <definition>Raw data</definition> <explanation/> <qualityControlLevelCode>0</qualityControlLevelCode> <qualityControlLevelID>0</qualityControlLevelID> </qualityControlLevel>


</qualityControlLevel> <sample/> <source> <source> <citation/> <contactInformation> <contactInformation> <address> <address xsi:type="xsd:string"/> </address> <contactName/> <email> <email/> </email> <phone> <phone/> </phone> <typeOfContact/> </contactInformation> </contactInformation> <metadata xsi:nil="true"/> <organization>Agència Catalana de l'Aigua</organization> <sourceCode>1</sourceCode> <sourceDescription>Variables fenomenològiques de SIX</sourceDescription> <sourceID>1</sourceID> <sourceLink> <sourceLink/> </sourceLink> </source> </source> <units xsi:nil="true"/> <value> <value> <accuracyStdDev xsi:nil="true"/> <censorCode>nc</censorCode> <codedVocabularyTerm xsi:nil="true"/> <dateTime>2013-02-06T15:00:00.000Z</dateTime> <dateTimeUTC>2013-02-06T15:00:00.000Z</dateTimeUTC> <labSampleCode xsi:nil="true"/> <metadataTime xsi:nil="true"/> <methodCode>0</methodCode> <methodID xsi:nil="true"/> <offsetTypeCode xsi:nil="true"/> <offsetTypeID xsi:nil="true"/> <offsetValue xsi:nil="true"/> <oid xsi:nil="true"/> <qualifiers/> <qualityControlLevelCode>0</qualityControlLevelCode> <sampleID xsi:nil="true"/> <sourceCode>1</sourceCode> <sourceID xsi:nil="true"/> <timeOffset>+01:00</timeOffset> <value>4</value> </value> <value> <accuracyStdDev xsi:nil="true"/> <censorCode>nc</censorCode> <codedVocabularyTerm xsi:nil="true"/> <dateTime>2013-02-06T15:15:00.000Z</dateTime> <dateTimeUTC>2013-02-06T15:15:00.000Z</dateTimeUTC> <labSampleCode xsi:nil="true"/> <metadataTime xsi:nil="true"/> <methodCode>0</methodCode> <methodID xsi:nil="true"/> <offsetTypeCode xsi:nil="true"/> <offsetTypeID xsi:nil="true"/> <offsetValue xsi:nil="true"/> <oid xsi:nil="true"/> <qualifiers/>


<qualityControlLevelCode>0</qualityControlLevelCode> <sampleID xsi:nil="true"/> <sourceCode>1</sourceCode> <sourceID xsi:nil="true"/> <timeOffset>+01:00</timeOffset> <value>4</value> </value> ... <value> <accuracyStdDev xsi:nil="true"/> <censorCode>nc</censorCode> <codedVocabularyTerm xsi:nil="true"/> <dateTime>2013-02-06T10:15:00.000Z</dateTime> <dateTimeUTC>2013-02-06T10:15:00.000Z</dateTimeUTC> <labSampleCode xsi:nil="true"/> <metadataTime xsi:nil="true"/> <methodCode>0</methodCode> <methodID xsi:nil="true"/> <offsetTypeCode xsi:nil="true"/> <offsetTypeID xsi:nil="true"/> <offsetValue xsi:nil="true"/> <oid xsi:nil="true"/> <qualifiers/> <qualityControlLevelCode>0</qualityControlLevelCode> <sampleID xsi:nil="true"/> <sourceCode>1</sourceCode> <sourceID xsi:nil="true"/> <timeOffset>+01:00</timeOffset> <value>4</value> </value> </value> </values> </values> <variable> <categories xsi:nil="true"/> <dataType xsi:nil="true"/> <error xsi:nil="true"/> <extension xsi:nil="true"/> <generalCategory xsi:nil="true"/> <metadataTime xsi:nil="true"/> <noDataValue>-9999.0</noDataValue> <note/> <oid xsi:nil="true"/> <options xsi:nil="true"/> <related xsi:nil="true"/> <sampleMedium xsi:nil="true"/> <speciation/> <timeScale> <isRegular>false</isRegular> <timeSpacing xsi:nil="true"/> <timeSupport>0.0</timeSupport> <unit> <unitAbbreviation>min</unitAbbreviation> <unitCode>1000</unitCode> <unitDescription xsi:nil="true"/> <unitID xsi:nil="true"/> <unitName>minute</unitName> <unitType>Time</unitType> </unit> </timeScale> <unit> <unitAbbreviation>NTU</unitAbbreviation> <unitCode>0001</unitCode> <unitDescription xsi:nil="true"/> <unitID xsi:nil="true"/> <unitName>NTU</unitName> <unitType>NTU</unitType> </unit> <valueType xsi:nil="true"/>


<variableCode> <variableCode> <network xsi:nil="true"/> <value>3965408</value> <variableID xsi:nil="true"/> <vocabulary>STR</vocabulary> </variableCode> </variableCode> <variableDescription xsi:nil="true"/> <variableName>Terbolesa</variableName> <variableProperty/> </variable> </timeSeries> </timeSeries> </GetValuesObjectReturn> </GetValuesObjectResponse> </soapenv:Body> </soapenv:Envelope>

Listing 15 "getValuesObject response example"

3.1.3.7 getVariableInfo

The method getVariableInfo provides a description of one specific variable or all existing variables,

depending on the parameters passed.

Figure 11 "Queried entities for getVariableInfo/getVariableInfoObject"

As Figure 11 shows, only the variables are being queried here. Table 8 lists the available parameters

that can be passed to the request.


variable Variable identifier in the format ‘NetworkName:VariableCode’. If the value is omitted

all variables will be returned.

authToken Previously created secutity token

Table 8 "getVariableInfo request parameters"

Listing 16 shows a getVariableInfo request example and Listing 17 shows the resulting response.

The main information which is returned is the name of the variable and the unit of the values provided.

Like with setSitesXML, getSiteInfo and getValues, this method strictly separates the

variablesResponse from the SOAP body by embedding it as a string.


<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:q0="http://ws.waterOneFlow.aca.gencat.cat" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <q0:GetVariableInfo> <q0:variable>S:3965408</q0:variable> <q0:authToken /> </q0:GetVariableInfo> </soapenv:Body> </soapenv:Envelope>

Listing 16 "getVariableInfo request example"

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <GetVariableInfoResponse xmlns="http://ws.waterOneFlow.aca.gencat.cat"> <GetVariableInfoReturn><?xml version="1.0" encoding="UTF-8" standalone="yes"?> <variablesResponse xmlns="http://www.cuahsi.org/waterML/1.1/"> <queryInfo> <creationTime>2014-02-22T12:35:16.513+00:00</creationTime> <criteria MethodCalled="GetVariableInfo"> <parameter value="S:3965408" name="variable"/> </criteria> </queryInfo> <variables> <variable> <variableCode default="true" vocabulary="S">3965408</variableCode> <variableName>Terbolesa</variableName> <unit> <unitName>NTU</unitName> <unitType>NTU</unitType> <unitAbbreviation>NTU</unitAbbreviation> <unitCode>0001</unitCode> </unit> <noDataValue>-9999.0</noDataValue> <timeScale> <unit> <unitName>minute</unitName> <unitType>Time</unitType> <unitAbbreviation>min</unitAbbreviation> <unitCode>1000</unitCode> </unit> <timeSupport>0.0</timeSupport> </timeScale> <speciation></speciation> </variable> </variables> </variablesResponse> </GetVariableInfoReturn> </GetVariableInfoResponse> </soapenv:Body> </soapenv:Envelope>

Listing 17 "getVariableInfo response example"

3.1.3.8 getVariableInfoObject

Except for some differences in the response structure, the getVariableInfoObject works exactly in

the same way as the getVariableInfo method. It can also return the information of one specific

variable or all variables. The only difference with respect to getVariableInfo is that the response is

directly embedded inside the SOAP body. Table 9 describes the available parameters, Listing 18

contains an example request, and Listing 19 shows the response received from the server.



variable Variable identifier in the format ‘NetworkName:VariableCode’. If the value

is omitted all variables will be returned.


Table 9 "getVariableInfoObject request parameters"

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:q0="http://ws.waterOneFlow.aca.gencat.cat" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <q0:GetVariableInfoObject> <q0:variable>S:3965408</q0:variable> <q0:authToken /> </q0:GetVariableInfoObject> </soapenv:Body> </soapenv:Envelope>

Listing 18 "getVariableInfoObject request example"

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <GetVariableInfoObjectResponse xmlns="http://ws.waterOneFlow.aca.gencat.cat"> <GetVariableInfoObjectReturn> <error xsi:nil="true" /> <queryInfo> <creationTime>2014-02-22T12:37:35.854Z</creationTime> <criteria> <locationParam xsi:nil="true" /> <methodCalled>GetVariableInfo</methodCalled> <parameter> <parameter> <name>variable</name> <value>S:3965408</value> </parameter> </parameter> <timeParam xsi:nil="true" /> <variableParam xsi:nil="true" /> </criteria> <extension xsi:nil="true" /> <note /> <queryURL xsi:nil="true" /> </queryInfo> <variables> <variable> <variable> <categories xsi:nil="true" /> <dataType xsi:nil="true" /> <error xsi:nil="true" /> <extension xsi:nil="true" /> <generalCategory xsi:nil="true" /> <metadataTime xsi:nil="true" /> <noDataValue>-9999.0</noDataValue> <note /> <oid xsi:nil="true" /> <options xsi:nil="true" /> <related xsi:nil="true" /> <sampleMedium xsi:nil="true" /> <speciation /> <timeScale> <isRegular>false</isRegular> <timeSpacing xsi:nil="true" />


<timeSupport>0.0</timeSupport> <unit> <unitAbbreviation>min</unitAbbreviation> <unitCode>1000</unitCode> <unitDescription xsi:nil="true" /> <unitID xsi:nil="true" /> <unitName>minute</unitName> <unitType>Time</unitType> </unit> </timeScale> <unit> <unitAbbreviation>NTU</unitAbbreviation> <unitCode>0001</unitCode> <unitDescription xsi:nil="true" /> <unitID xsi:nil="true" /> <unitName>NTU</unitName> <unitType>NTU</unitType> </unit> <valueType xsi:nil="true" /> <variableCode> <variableCode> <network xsi:nil="true" /> <value>3965408</value> <variableID xsi:nil="true" /> <vocabulary>S</vocabulary> </variableCode> </variableCode> <variableDescription xsi:nil="true" /> <variableName>Terbolesa</variableName> <variableProperty /> </variable> </variable> </variables> </GetVariableInfoObjectReturn> </GetVariableInfoObjectResponse> </soapenv:Body> </soapenv:Envelope>

Listing 19 “getVariableInfoObject response example”

3.2 Mapping to WaterML2

As D2.3-“Open Interface Specification” chapter 3.1.1.2 describes, the ACA data provided by the

WaterOneFlow server has to be mapped to SOS/WaterML2. This chapter will list the information

relevant to generate the basics of the ontology and consider how this information can be extracted from

WaterOneFlow. The consideration will be based on the real responses from the ACA installation

described in chapter 3.1.3.

Figure 12 provides an overview over the OGC O&M standard while Appendix I section 6 gives an

overview over the CUAHSI ODM.


Figure 12 "Summary of the OGC Observations and Measurements (O&M) standard"3

Table 10 shows the most significant differences between the entities of WaterML1 (CUAHSI) and

WaterML2 (OGC) focused on the data that is required to generate the ontology. For each element it

shows an example to clarify the similarities or differences.

WaterML2 WaterML1

Feature of interest identifier from feature of interest

inside OM_Observation

Format is free (gml:identifier tag or

wml2:monitoringPoint/@gml:id)

Site identifier from source site

Example: S:3965408

Observed property from OM_Observation

Example: ‘urn:ogc:def:phenomenon:waterpressure’

Variable name from source variable

Example:

<variableName>Terbolesa</variableName>

Offering from the sensor description output list Variable name from source variable

3 http://external.opengeospatial.org/twiki_public/HydrologyDWG/USGSStandardsIssues


metadata

Example: ‘WATERPRESSURE’

Example:

<variableName>Terbolesa</variableName>

Sensor from procedure description linked with

OM_Observation

Format is free (wml2:processReference/@xlink:href)

There is no information in the ACA

WaterOneFlow responses what might serve as

the basis for creation of the sensor.

Location from feature of interest monitoring point

Example:

<gml:pos srsName=

"http://www.opengis.net/def/crs/EPSG/0/4258">

40.660 -5.4207

</gml:pos>

GeoLocation from source site

Example:

<geogLocation

xsi:type="LatLonPointType"

srs="EPSG:4258”>

<latitude>40.660</latitude>

<longitude>-5.4207<longitude>

</geogLocation>

Unit from the observation result default point metadata

Example:

<wml2:uom code="bar"/>

Unit name from source variable

Example:

<unitName>NTU</unitName>

Table 10 "Main entities in CUAHSI and OGC"

The transformation of feature of interest, location and unit is not problematic. The notation

of the SRS (Spatial Reference System) has to be changed but the general information is available.

Mapping of the sensor identifier is also possible as there is no special notation provided that the

sensor can be identified. As in WaterOneFlow the sensor is modelled as the combination of a site and a

variable, the two identifiers can be used to generate the unique sensor identifier for WaterML2.

The mapping of observed property and offering is more difficult because the observed

property requires a special format and should use predefined standard URIs. The same applies for

the offering. As an example, Table 10 "Main entities in CUAHSI and OGC" displays

‘urn:ogc:def:phenomenon:waterpressure’ for observed property and ‘WATERPRESSURE’ for

offering.

In consequence, the conversion from WaterOneFlow to SOS requires a mapping from the variable

names used by ACA to the observed properties and offerings used in WaterML2.


3.3 ACA Pilot Integration Manager

This chapter describes the current state of implementation of the ACA pilot integration manager (PIM).

D3.3-“WDW 2nd

Prototype” will contain a full documentation of the system. This document will give a

short technical overview focussing on the level of integration of ACA data. The application is

implemented as a web application which polls the ACA server in configured intervals. The sensors’

meaning pairs of site identifier and variable identifier are stored inside a database table. The full content

of the table is listed in Table 11.

Column Description

ID Identifier

SITE_CODE Site identifier

VARIABLE_CODE Variable identifier

START_POLLING Start date from which on the time series should be converted to

SOS/WaterML2

HIGHEST_TIMESTAMP Highest timestamp of the observation results that are already converted to

SOS/WaterML2

DEFINED_IN_SOS ‘false’ indicates that the sensor isn’t yet defined in the SOS server. Once the

application creates the SOS sensor the column will be changed to true.

Table 11 "Columns of sensor table in ACA pilot integration manager"

The application creates the ACA sensor before the first observation result is to be inserted. In order to

avoid polling all the historical data when adding a new sensor, the timestamp where the polling starts

can be configured.

To reduce the risk of destabilizing the WaterOneFlow server, the ACA server has a maximum time

interval that can be queried by one request. If this limit is exceeded the server responds with an error

code to avoid running out of memory. For this reason (and also not to cause instabilities in the PIM

itself), a maximum query interval is defined in the configuration of the PIM. First, for each polled sensor

the available time range available is determined by calling getSiteInfoObject. This data is merged

with the additional information as START_POLLING and HIGHEST_TIMESTAMP stored in the sensor

table. With this data PIM determines the time interval it has to query on the ACA server. After that the

time interval is separated into subintervals as to meet the restrictions of the server about maximum time

intervals allowed for querying. The observation results are obtained by calling getValues. Next, the

response is analysed and converted to WaterML2. The current implementation realizes a very simple

conversion:


The sensor identifier is generated by combining site and variable identifier of

WaterOneFlow. The pattern is <siteCodeNetwork>-<siteCode>/<variableCodeVocabulary>-

<variableCode>. For example for site S:430141-002 and variable S:3965408 the sensor

identifier would be S-430141-002/S-3965408.

The observed properties are created by applying the pattern

urn:x-ogc:def:phenomenon:OGC:<variableName>. As for phemomenons, there exists already a

standard list of predefined URNs4. When mapping ACA data these predefined URNs should be

used if possible. Therefore the mapping should map the variable identifier or the variable

names to the phenomenon URNs to be used for WaterML2 generation.

For offering and offeringName the variable name from WaterOneFlow is used. This mapping

is to be reconsidered as it might be altered according to the changed mapping of the observed

properties. Like with the observed property a mapping from the ACA variable name or identifier

to the offering can be added to solve the problem.

The feature of interest is created based on the pattern <siteCodeNetwork>-<siteCode>.

The name of the feature of interest is taken from the site name in WaterOneFlow.

Unit and location are transferred from their counterparts in WaterOneFlow. For location

SRS the prefix urn:ogc:def:crs:EPSG:: is added to meet the restrictions of the SOS protocol.

Figure 13 shows the processing flow when a sensor is polled.

4

https://www.seegrid.csiro.au/subversion/xmml/OGC/branches/SWE_gml2/sweCommon/current/examples/pheno

mena.xml


Figure 13 "Process flow during sensor polling"

3.4 Next steps

For the technical implementation the next and most critical step is to adjust the conversion from

WaterOneFlow to WaterML2. For each variable there must be a mapping to the matching offering

and observed property.

Next, the existing site and variable combinations have to be extracted and stored into the sensor

table. The polling start time which defines the extent of historical data available in the WDW, must be

set according to the requirements of the analysis. After installation of the ACA pilot integration module

and the pilot SOS server, the system will start to poll and fill the SOS database.

Then ACA will be prepared to be integrated into the WDW for further analysis. As the WaterOneFlow is

already available externally, a first integration in WatERP is possible without any installations on the

pilot site. Figure 14 shows a possible installation scenario.


Figure 14 "Possible installation scenario for ACA"

4. SWKA site integration

In this section the activities taken so far to integrate SWKA pilot sensor data into the WDW are

described. First, an overview of the existing data infrastructure for SWKA is given, then the data,

mapping to ontology and the SWKA Integration Manager are explained.

4.1 Existing data infrastructure

The data infrastructure is highly different to the one at ACA. SWKA runs a data warehouse which can

be considered as a black box. All the sensor network results are fed into this system. To make the data

available for WatERP, the observation results have to be exported. In this export, the time interval and

the step size of the time series have to be specified. Furthermore, the exported data has already

passed some steps of preparation before it enters the WDW.

Up to now, the work to analyze and process the data from the SWKA pilot site has been based on an

export provided in the format of an Oracle database dump file5 which had been created by SWKA

containing a snapshot of the observation results. The file was imported in an Oracle6 database on the

development site to implement the data access. From the example table depicted in Figure 15, it can be

noted that the sensor data, with other data fields such as quality and rate, is a series of time and value

pairs. In this table, PP_ID is the sensor ID. Indeed, each sensor has different readings at different

times. Thus, the first level of integration is to convert the data into WaterML2, so that the data can be

stored into an SOS server. To accomplish this task, the entities identified in WP4 for SWKA that have to

be part of the WaterML2 document, are first mapped as described in next section.

5 Data Pump Export http://docs.oracle.com/cd/B28359_01/server.111/b28319/dp_export.htm#i1006388

6Oracle Database (”http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html”)


Figure 15 “Database table from pilot site SWKA”

4.2 Mapping to WaterML2

D4.1-‘Inference and Simulation Engine Conceptual Design’ chapter 3.2 explains the logical model as

well as the data this model has to be based on.

As displayed in Figure 16 the required features of interest that were identified in D4.1 are:

Karlsruhe City: Urban demand in Karlsruhe city. This is the area of which the DMS is going to

provide the forecasted demand for, and where the DSS will provide recommendations for the

pumps management to satisfy demand with the minimum possible energy cost.

HW Pumping: Hardtwald Water Work. Ground water pumped from well-fields, and distributed

through four pumps with constant displacement. It supplies water to Karlsruhe North and to the

Main Reservoir.

MW Pumping: Mörscher Wald Water Work. Ground water pumped from well-fields, and

distributed through four pumps with constant displacement. It supplies water to Karlsruhe South

and to the Main Reservoir.

RW Pumping: Rheinwald Water Work. Ground water pumped from well-fields, and distributed

through four pumps with constant displacement. It supplies water to Karlsruhe South and the

Main Reservoir.

DW Pumping: Durlacher Water Work. Ground water pumped from three well-fields. It provides

water directly to the system (Karlsruhe City) and to the Main Reservoir.

HB Luss - Main Reservoir: Storage Tank Luss or Main Reservoir, it supplies fresh water to

Karlsruhe City. It is used as an emergency water reserve, to support demand peaks


Figure 16 "SWKA logical model"

The most detailed description can be found in table 11 of D4.1. It lists all variables for each feature of

interest and serves as the central information for registering the sensors on the SOS server.

SWKA will provide the mapping of the lines in this table to the PP_IDs in the data warehouse, as well as

the list of locations for each feature of interest within the next deliverable. Then the registration of the

sensors and the loading of the obvservation results will be fully implemented.

4.3 SWKA Pilot Integration Manager

This section is focused on defining the whole process of converting the data into WaterML 2, adding

sensor and sensor observation data from SWKA database onto SOS server by PIM (Pilot Integration

Manager). Currently, the PIM has a very simple implementation in order to prove that for a single

sensor reading, mapping and storing the data is working for SWKA. In the next iteration, the design


must be extended to allow that all required sensor results can be mirrored into the pilot SOS server.

Chapter 4.4 explains the next steps in more detail.

4.3.1 Mapping to WaterML2

The PIM consists of a wrapper module for each step of the mapping process. A flow chart diagram is

shown in Figure 17 to describe these implementation steps. A database wrapper gets data from the

database, an XML schema defines the look of the XML document, and a transformer wrapper

transforms this into a WaterML2 document with the help of an XSLT wrapper.

Figure 17 "Flow chart of client application"

The output of this application is an XML document in WaterML2 format. It contains a root element

MeasurementTimeSeries with several time-series observations inside a Point element as shown in

Listing 20. At the moment, only few readings of a given sensor id are taken from the database to

check the output. Future work will involve reading data for all sensor ids and building a trigger to

execute the export when new sensor data comes in. To store the observations, the WaterML2 data is

sent to the SOS Server.


Listing 20 "Output in WaterML2 format"

4.3.2 Load Sensor Metadata

In this step, the sensor metadata and the configuration settings are passed to the WatERP framework

as an XML file whose format is based on SML. The configuration settings are shown in the Figure 18.

Figure 18 “Configuration Settings for PIM”

4.3.3 Configure New Sensor

In the SWKA scenario there exist two types of sensors: in-situ and dynamic. Both kind of sensors have

been configured and subscribed into an SOS server. The SOS server provides an API for managing

deployed sensors and retrieving sensor information related to their measurement (e.g., observations,

measurement, etc.) and configuration (e.g., type of sensor, id, etc.). Based on this stored information,

the SOS server offers some operations to gather sensor information (measurement information and

configuration): i) GetObservation, ii) DescribeSensor, and iii) GetCapabilites.

The current work related to sensor integration inside the WatERP solution has been focused on adding

SWKA sensors into WDW server. In order to perform this task, the first step consists on incorporating

sensors into the SOS server. The process to include a new sensor inside the SOS server includes the

registration of the sensor and the configuration of the sensor’s observations. This process is supported

by two transactional operations: RegisterSensor and InsertObservation.


Using these operations, the PIM registers the sensor and inserts the sensor observation from the

database onto the SOS server. For this reason, the client application is fitted with two more wrapper

modules, one with the purpose of registering a sensor and another one to insert the sensor observation.

The standard used to register a sensor is Sensor Model Language (SML). Furthermore, the

observations are inserted following the OGC Observation and Management Specification (O&M). These

standards propose to encode the information of the observation to be inserted into XML. Additionally,

each RegisterSensor operation has two mandatory attributes which are service and version.

The request for this operation is contained in the SensorDescription element. The response to a

RegisterSensor request contains an AssignedSensorID which is the identifier assigned by the

SOS to designate the new sensor. The identifier is of type anyURI and must be either a URN or a URL.

The request of register a sensor and the response from SOS is shown in Listing 21 and Listing 22,

respectively.

Listing 21 "Request to register a sensor"

Listing 22 “Response from SOS for sensor registration”


At the moment, the RegisterSensor wrapper is registering sensors using hardcoded data. Some of

these hardcoded data are stored in configuration properties files and some other are stored in the class

variables. This wrapper generates an XML file in the format of SML (SML-XML file) which is sent to

SOS server in order to register the sensor. In the same way, there is a partial implementation of the

InsertObservation operation. If a sensor was previously registered, the re-registration request

receives an error message from SOS in the response, stating that “The sensor is already registered at

this SOS”.

4.3.4 Trigger Import/InsertObservation

Once the sensor has been registered with SOS, the PIM can begin inserting observations for that

sensor. The AddSensorObservation wrapper in the client application gets data from database

tables, creates an XML document using XML schema and transforms this document into O&M standard

using XSLT transformations. As described in 4.3.2, this first PIM version uses information passed as

configuration settings instead of triple store data to describe a sensor in the SOS server such as

feature of interest, observed property, offering, sensor ID and unit. The time

series and measured value are retrieved from the database. Each InsertObservation request

includes the AssignedSensorID returned from the RegisterSensor operation and has mandatory

attributes of service and version. The request of InsertObservation and response from SOS is

shown in Listing 23 and Listing 24.

Listing 23 "Request to SOS for InsertObservation"

It is important to note that the response from SOS contains AssignedObservationID for the new

observation entered. If the same observation is entered twice, an error message is received in the

response body stating that the observation is already present in the SOS database.


Listing 24 “Response from SOS”

4.4 Next Steps

As mentioned before, the current version of the SWKA PIM implementation is a proof-of-concept. In

order to supply the WDW with all data required to set-up the ontology and provide the data, the system

must be able to process more sensors at a time as well as skip observation results that have already

been mirrored into the SOS server. To do so, the system will require a persistence that will be similar to

the sensor table of the ACA PIM. As for ACA the ontology can be derived from the WaterOneFlow

model whereas for SWKA it can’t, the SWKA sensor table must also provide the necessary mapping

data.

The strategy adopted for the SWKA PIM is very different to the ACA’s, where it is reasonable to set up

a poller that periodically queries the WaterOneFlow server. In SWKA the situation is different as new

data will only be available once a new export has been triggered in the data warehouse. Therefore, an

application which can be explicitly started after the export has finished succesfully, is preferable to a

system which polls in fixed time periods.

It is also pending to define the setup of the installation on the pilot site. Figure 19 shows a possible

installation scenario according to the architecture described.

The reasons to place the SOS server inside the DMZ are:

By design the SOS server is used as the link between the pilot site and the WDW, thus placing

it in the DMZ would reflect this design.

The SOS server does not communicate directly with other systems except the SOS database

which offers HTTP services to both the internal SWKA network and the WatERP platform. Only

standard HTTP protocol (e.g. port 80) access is required when configuring the firewalls.

Figure 19 "Possible installation scenario for SWKA"


Of course, this installation scenario represents only an early sketch. The final setup will be decided

together with SWKA in the next deliverable D7.3.2.

5. Conclusions and future work

This section summarizes all the activities performed so far and describes the next steps.

The existing data infrastructure has been analysed and the sources for observation result data have

been identified for both clients (ACA and SWKA). Also for each context the access to the pilot data has

been implemented. The implementation of the data access layer is almost complete for both pilots.

Regarding the mapping of the pilots’ data to the ontology, the knowledge acquisition task has reached a

point where the next steps are clear. For ACA, most of the information is already available, but an

additional mapping is required. For SWKA, the missing information has been requested which is

necessary to complete the mapping. As a conclusion, most of the raised uncertainties have been

clarified.

The generation of the WaterML2 documents has been implemented as a prototype. The final definition

of the WaterML2 contents that will serve as the base to generate the ontology is currently in progress.

The implementation of the SOS client for registering new sensors and inserting observations has been

completed.

Both scenarios, ACA and SWKA, have been installed on the development site and tested. To prove if

the access to the WDW works properly, the sensors have also been registered in the WDW and the

data available in the WDW interface layer have been verified.

The results were accomplished based on the work of the other work packages, especially WP1, WP2,

WP3 and WP4. The decisions were drawn in close consultation with other work packages as well as

with the pilots.

For the future work, the WaterML2 content has to be finally specified in order to provide the base of the

population of the ontology.

To finish the integration of the ACA pilot, the mapping of the WaterML1 data to WaterML2 must be

finished. Here, especially observed property and offering are to be adjusted. Afterwards, the list

of required sensors has to be defined and added to the sensor list that controls the polling.

For the SWKA pilot, the mapping of the sensor PP_IDs to WaterML2 must be finished. Further, the PIM

implementation must be extended to support polling of all required sensors as described in chapter 4.4.

Finally, the decision of how to setup the integration on the SWKA pilot site has to be drawn.


6. Appendix I: Observation Data Model (ODM) of the CUAHSI

HIS

Figure 20 "Observation Data Model CUAHSI HIS"

When describing the ACA pilot integration, chapter 3 explains also the structure of the WaterML1

entities which are very similar to the Observation Data Model implemented by CUAHSI HIS. Figure 20

gives a complete overview over the ODM.


7. Appendix II: WaterOneFlow Service

Figure 21 "WaterOneFlow Service diagram"

Figure 21 lists all methods implemented by the WaterOneFlow web service. Chapter 3.1.3 gives a more

detailed description of the methods and the parameters, displaying also examples with the data offered

by ACA.

waterp water enhanced resource planning · 2017-12-08 · waterp water enhanced resource planning...

Documents