d5.3 eo services and tools - databio · d5.3 – eo services and tools h2020 contract no. 732064...

D5.3 – EO Services and Tools H2020 Contract No. 732064 Final – v1.0, 15/6/2018

Dissemination level: PU -Public

Project Acronym: DataBio

Grant Agreement number: 732064 (H2020-ICT-2016-1 – Innovation Action)

Project Full Title: Data-Driven Bioeconomy

Project Coordinator: INTRASOFT International

DELIVERABLE

D5.3 – EO Services and Tools

Dissemination level PU -Public

Type of Document Report

Contractual date of delivery 31/05/2018

Deliverable Leader Fraunhofer

Status - version, date Final – v1.0, 15/6/2018

WP / Task responsible WP 5

Keywords: EO and Geospatial Data, Big Data Technology, IOT

services and tools, DataBio Platform, DataBio Pilots,

DataBio Toolset



Executive Summary The deliverable D5.3 – EO Services and Tools provides a user-oriented perspective on the

technical deliveries of the DataBio project in the scope of handling big Earth observation (EO)

and geospatial data. The deliverable is targeted at people who are familiar with EO and

geospatial applications and seek innovative solutions for the upcoming challenges of storing,

managing, processing, analysing and visualizing vast amount of EO and geospatial data.

The document reports on the services and tools that will be delivered by DataBio partners for

handling big EO and geospatial data. These services and tools are generated by adapting,

extending and implementing the software components provided as basic technology (see

[REF-05] and [REF-07]). This process of generating tailored services has been triggered by the

requirements specifications of various pilot applications in the DataBio project coming from

the agricultural, fishery and forestry domains. These pilot applications formulate their specific

needs and adapted services and tools are generated based on existing components in order

to meet these requirements. These services and tools are implemented in the scope of

individual pilots but might be of interest for other DataBio pilots or for external users of the

DataBio platform as well. They can be used to offer directly functionalities in agriculture,

forestry and fishery applications or can be used, as part of the DataBio toolset, to build such

applications.

The document provides an overview about the types of services and tools along with a short

description about their purpose and functionality. More detailed information is given through

a narrative example of how the service or tool is used in a typical application. Further,

information on the ability of this piece of software for handling big data is provided. Finally,

the external accessibility of the technology is documented, along with contact points and

information on IPR and licenses.

Relation with Other DataBio Platform Deliverables The DataBio project includes three piloting work packages (WP1-3) and two related platform

work packages (WP4 handling IoT data and WP5 processing Earth Observation and geospatial

data) that support the pilots (Figure 1). The DataBio platform provides Big Data capabilities to

the pilots by forming pipelines of software components. At the beginning each pipeline, there

are source data from agriculture, forestry and fishery. These data are then processed using

different analytical software components and the resulting data or information are presented

at the end of the pipeline.

The platform developed in DataBio is described in the Deliverables D4.1, D4.2, D4.3 (WP4)

and D5.1, D5.2 D5.3 (WP5) (Figure 1). Deliverables D4.1-3 define the Milestone M7 Service

ready for Trial 1, whereas Deliverables D5.1-3 define the Milestone M9 EO Services ready for

integration. The platform services and pipelines have been in trials since April 2018 (M16).



Figure 1: Work packages and their roles in DataBio

More specifically, the public deliverable D4.1 Platforms and Interfaces describes the software

components to be utilized by the pilots. Most components are already in use in the first pilot

trials. In addition, this deliverable reports the outcome of a matchmaking process, in which

the pilots selected which components to deploy in their pilots.

Deliverable D4.2 Services for Tests builds on D4.1 and provides an overview of the component

pipelines as identified at month 16 (M16) of the project. It also provides guidelines for

successful implementation and deployment of the pipelines.

Deliverable D4.3 Data Sets, Formats and Models is due at the end of August 2018. While the

two earlier reports deal with software modules, this report will focus on the data sets and

streams employed in DataBio. Data formats, standards and models enabling easy findability,

access, interoperability, and reusability of data (FAIR principle) will be dealt with. Thus, we

will address in this deliverable topics beyond the coverage of single pilots.

Deliverable D5.1 EO Component Specification includes an analysis of the EO dataset and

component related requirements provided by the pilots. It was published in end of 2017 and

contains an overview of best practices of EO access and initial component and dataset

requirements based on the DataBio pilot needs.

Deliverable D5.2 EO Component and Interfaces describes, building on D5.1, the Earth

Observations component pipelines similarly as D4.2 does for IoT components. It also includes

examples of data experimentations with the pipelines.

Deliverable D5.3 EO Services and Tools builds on 5.1 and 5.2 and describes how the technical

components from DataBio can be scaled-up to services and tools that are installed as Software

as a Service (SaaS) or on-premise. It further provides the information how and under which

conditions these services and tools can be externally accessed.



Deliverable Leader: Eva Klien (Fraunhofer)

Contributors:

Ivo Senner (Fraunhofer)

Yves Coene (SPACEBEL)

Miguel Ángel Esbrí (ATOS)

Marco Corsi (e-Geos)

Marco Folegani (MEEO)

Adrian Stoica (TerraS)

Michal Kepka (UWB)

Karel Jedlicka (UWB)

Tomáš Řezník (Lespro)

Karel Charvat (Lespro)

Jesus Maria Estrada Villegas (Tragsa / Tragsatec)

Erwin Goor (VITO)

Renne Tergujeff (VTT)

Reviewers:

Jesus Maria Estrada Villegas (Tragsa / Tragsatec)

Tomas Mildorf (UWB)

Marco Folegani (MEEO)

Approved by: Athanasios Poulakidas (INTRASOFT)

Document History

Version Date Contributor(s) Description

0.1 8/3/2018 Eva Klien Proposed TOC

0.2 Eva Klien Questionnaire distributed and document

structure adapted accordingly

0.3 Eva Klien, all

contributors

Integration and editing of partner

contributions

0.4 Eva Klien Executive summary + References

0.5 30/5/2018 Eva Klien Final draft for Review

0.5.1 4/6/2018

Tomas Mildorf, Yves

Coene, Erwin Goor,

Renne Tergujeff,

Tomas Reznik

Comments integrated into the document

0.6 8/6/2018

Karel Jedlicka, Tomas

Reznik, Jesus Maria

Estrada Villegas,

Renne Tergujeff,

Marco Folegani, Marc

Gilles, Eva Klien

Review comments and additional material

integrated; definition of tools updated;

final consolidation; submitted for final

quality control

1.0 15/6/2018 Athanasios Poulakidas Final version for submission



Table of Contents EXECUTIVE SUMMARY ..................................................................................................................................... 2

RELATION WITH OTHER DATABIO PLATFORM DELIVERABLES ........................................................................................... 2

TABLE OF CONTENTS ........................................................................................................................................ 5

TABLE OF FIGURES ........................................................................................................................................... 7

LIST OF TABLES ................................................................................................................................................ 7

DEFINITIONS, ACRONYMS AND ABBREVIATIONS ............................................................................................. 9

INTRODUCTION .................................................................................................................................... 14

1.1 PROJECT SUMMARY ..................................................................................................................................... 14 1.2 DOCUMENT SCOPE ...................................................................................................................................... 16 1.3 DOCUMENT STRUCTURE ............................................................................................................................... 17

EO AND GEOSPATIAL SERVICES AND TOOLS DELIVERED FOR DATABIO ................................................ 18

DETAILS FOR AN INDIVIDUAL ASSESSMENT ON THE USEFULNESS OF EACH SERVICE OR TOOL ............. 24

3.1 TOOL FOR DATA MANAGEMENT AND PROCESSING (SPATIO-TEMPORAL SENSOR, IMAGE, SIMULATION AND STATISTICS

DATA) 24 3.1.1 Narrative ....................................................................................................................................... 24 3.1.2 Technology for handling big data ................................................................................................. 25 3.1.3 Documentation ............................................................................................................................. 25

3.2 SERVICE AND TOOL ON DATA DOWNLOAD AND ACCESS (EO IMAGES) ................................................................... 26 3.2.1 Narrative ....................................................................................................................................... 26 3.2.2 Technology for handling big data ................................................................................................. 27 3.2.3 Documentation ............................................................................................................................. 28

3.3 TOOL FOR IMAGES AND ORTHOPHOTOS PROCESSING .......................................................................................... 31 3.3.1 Narrative ....................................................................................................................................... 32 3.3.2 Technology for handling big data ................................................................................................. 32 3.3.3 Documentation ............................................................................................................................. 33

3.4 TOOL TO EXPLOIT DERIVED DATA/INFORMATION FROM EO DATA SOURCES (DASHBOARD) ......................................... 34 3.4.1 Narrative ....................................................................................................................................... 34 3.4.2 Technology for handling big data ................................................................................................. 34 3.4.3 Documentation ............................................................................................................................. 34

3.5 BACKEND SERVICES FOR PROCESSING ON SENTINEL-2 AND METEO DATA ................................................................. 35 3.5.1 Narrative ....................................................................................................................................... 35 3.5.2 Technology for handling big data ................................................................................................. 35 3.5.3 Documentation ............................................................................................................................. 36

3.6 SERVICE FOR THE FUSION OF SENTINEL-1 AND SENTINEL-2 DATA OVER A RESTRICTED AOI AND DATE RANGE ................. 36 3.6.1 Narrative ....................................................................................................................................... 36 3.6.2 Technology for handling big data ................................................................................................. 37 3.6.3 Documentation ............................................................................................................................. 37

3.7 TOOL FOR DATA PROCESSING (COPERNICUS DATA) ............................................................................................. 38 3.7.1 Narrative ....................................................................................................................................... 38 3.7.2 Technology for handling big data ................................................................................................. 38 3.7.3 Documentation ............................................................................................................................. 39

3.8 TOOL FOR DATA ACCESS AND VISUALIZATION (EO AND CLIMATE DATA) .................................................................. 39 3.8.1 Narrative ....................................................................................................................................... 40 3.8.2 Technology for handling big data ................................................................................................. 40



3.8.3 Documentation ............................................................................................................................. 41 3.9 MOSAIC CLOUD FREE BACKGROUND SERVICE ................................................................................................... 44

3.9.1 Narrative ....................................................................................................................................... 44 3.9.2 Technology for handling big data ................................................................................................. 44 3.9.3 Documentation ............................................................................................................................. 45

3.10 EO CROP MONITORING SERVICE ............................................................................................................... 45 3.10.1 Narrative .................................................................................................................................. 45 3.10.2 Technology for handling big data ............................................................................................ 46 3.10.3 Documentation ........................................................................................................................ 46

3.11 SENTINEL 2 CLOUDS, SHADOWS AND SNOW MASK TOOL ............................................................................... 47 3.11.1 Narrative .................................................................................................................................. 47 3.11.2 Technology for handling big data ............................................................................................ 48 3.11.3 Documentation ........................................................................................................................ 48

3.12 ONLINE SERVICE PLATFORM FOR PROCESSING SATELLITE DATA FOR FORESTRY...................................................... 48 3.12.1 Narrative .................................................................................................................................. 49 3.12.2 Technology for handling big data ............................................................................................ 50 3.12.3 Documentation ........................................................................................................................ 51

3.13 TOOLS FOR PRODUCING FORESTRY INFORMATION FROM EO DATA .................................................................... 51 3.13.1 Narrative .................................................................................................................................. 52 3.13.2 Technology for handling big data ............................................................................................ 52 3.13.3 Documentation ........................................................................................................................ 52

3.14 TOOL FOR DATA MANAGEMENT AND VISUALIZATION (2D / 3D) ...................................................................... 53 3.14.1 Narrative .................................................................................................................................. 53 3.14.2 Technology for handling big data ............................................................................................ 53 3.14.3 Documentation ........................................................................................................................ 54

3.15 SERVICE FOR INTERACTIVE ANALYSIS AND AGGREGATION ............................................................................... 55 3.15.1 Narrative .................................................................................................................................. 55 3.15.2 Technology for handling big data ............................................................................................ 55 3.15.3 Documentation ........................................................................................................................ 56

3.16 TOOL FOR VISUALIZATION OF 2D, 3D AND 4D DATA OF A HIGH VOLUME ........................................................... 56 3.16.1 Narrative .................................................................................................................................. 57 3.16.2 Technology for handling big data ............................................................................................ 60 3.16.3 Documentation ........................................................................................................................ 61

3.17 SERVICE AND TOOL FOR DATA MANAGEMENT (SENSOR DATA) ......................................................................... 61 3.17.1 Narrative .................................................................................................................................. 61 3.17.2 Technology for handling big data ............................................................................................ 62 3.17.3 Documentation ........................................................................................................................ 62

3.18 TOOL FOR METADATA MANAGEMENT ((OPEN)MICKA) ................................................................................. 63 3.18.1 Narrative .................................................................................................................................. 63 3.18.2 Technology for handling big data ............................................................................................ 63 3.18.3 Documentation ........................................................................................................................ 66

CONCLUSION ........................................................................................................................................ 68

REFERENCES ......................................................................................................................................... 69



Table of Figures FIGURE 1: WORK PACKAGES AND THEIR ROLES IN DATABIO ................................................................................................. 3 FIGURE 2: DATA MANAGER COLLECTIONS CONFIGURATION FILE (EXTRACT) .......................................................................... 27 FIGURE 3: INGESTION ENGINE REQUESTS CONFIGURATION FILE (EXTRACT) ............................................................................ 27 FIGURE 4: IMAGE ENHANCEMENT FRAMEWORK OF THE TOOL FOR IMAGES AND ORTHOPHOTOS PROCESSING .............................. 32 FIGURE 5: STATISTICS ABOUT THE PRODUCTS ACCESSED VIA EODATASERVICE ......................................................................... 41 FIGURE 6: STATISTICS ABOUT CLIMATHON PRODUCTS ACCESSED VIA EODATASERVICE + JUPITER ............................................... 41 FIGURE 7: WEB INTERFACE FOR THE MULTI-SENSORS EVOLUTION ANALYSIS .......................................................................... 43 FIGURE 8: SCREENSHOT OF THE JUPYTER NOTEBOOKS – PYTHON CONSOLE ........................................................................... 43 FIGURE 9: ENDPOINT ACCESS VIA WCS QUERYING ........................................................................................................... 44 FIGURE 10: OVERALL CONCEPT OF THE FORESTRY THEMATIC EXPLOITATION PLATFORM .......................................................... 49 FIGURE 11: SCREENSHOT OF THE FORESTRY TEP USER INTERFACE ....................................................................................... 50 FIGURE 12: SAMPLE OUTPUT FROM THE PROBABILITY ANALYSIS CHAIN: GROWING STOCK VOLUME BY TREE SPECIES ...................... 52 FIGURE 13: SCREENSHOT OF A 2D VISUALISATION OF LARGE VECTOR DATA STORED IN GEOROCKET ........................................... 54 FIGURE 14: 3D OPEN LAND USE VISUALISATION ............................................................................................................. 58 FIGURE 15: ROSTĚNICE FARM VISUALISATION ................................................................................................................. 59 FIGURE 16: LINKED DATA INTEGRATION TO 3D VIRTUAL ENVIRONMENT ............................................................................... 60 FIGURE 17: SCHEMATIC OVERVIEW OF THE DATABIO METADATA, LINKED DATA AND GRAPH DATA PIPELINE THAT RE-USES

(META)DATA FROM THE COPERNICUS OPEN ACCESS HUB TO DISPLAY METADATA DIRECTLY IN A MAP VIEWER .................... 65 FIGURE 18: SCREENSHOT OF THE CURRENT VERSION (DATED TO 18 MAY 2018) OF THE PROGRESS IN THE DATABIO METADATA,

LINKED DATA AND GRAPH DATA PIPELINE............................................................................................................. 65

List of Tables TABLE 1: THE DATABIO CONSORTIUM PARTNERS ............................................................................................................. 14 TABLE 2: LIST OF EO AND GEOSPATIAL SERVICES AND TOOLS DELIVERED FOR THE DATABIO-PLATFORM AND PILOTS .................... 18 TABLE 3: SUMMARY OF TOOL FOR DATA MANAGEMENT AND PROCESSING (SPATIO-TEMPORAL SENSOR, IMAGE, SIMULATION AND

STATISTICS DATA) ............................................................................................................................................. 25 TABLE 4: SUMMARY OF SERVICE AND TOOL ON DATA DOWNLOAD AND ACCESS (EO IMAGES) – FEDEO GATEWAY COMPONENT ... 28 TABLE 5: SUMMARY OF SERVICE AND TOOL ON DATA DOWNLOAD AND ACCESS (EO IMAGES) – FEDEO CATALOG COMPONENT .... 29 TABLE 6: SUMMARY OF SERVICE AND TOOL ON DATA DOWNLOAD AND ACCESS (EO IMAGES) – DATA MANAGER COMPONENT ..... 30 TABLE 7: SUMMARY OF SERVICE AND TOOL ON DATA DOWNLOAD AND ACCESS (EO IMAGES) – INGESTION ENGINE COMPONENT .. 31 TABLE 8: SUMMARY OF TOOL FOR IMAGES AND ORTHOPHOTOS PROCESSING ........................................................................ 33 TABLE 9: SUMMARY OF TOOL TO EXPLOIT DERIVED DATA/INFORMATION FROM EO DATA SOURCES (DASHBOARD) ....................... 34 TABLE 10: SUMMARY OF BACKEND SERVICES FOR PROCESSING ON SENTINEL-2 AND METEO DATA ............................................. 36 TABLE 11: SUMMARY OF FUSED SENTINEL-1 AND SENTINEL-2 DATA OVER A RESTRICTED AOI AND DATE RANGE .......................... 37 TABLE 12: SUMMARY OF TOOL FOR DATA PROCESSING (COPERNICUS DATA) ......................................................................... 39 TABLE 13: SUMMARY OF MULTI SENSORS EVOLUTION ANALYSIS - MEA ............................................................................... 41 TABLE 14: SUMMARY OF MOSAIC CLOUD FREE BACKGROUND SERVICE ............................................................................... 45 TABLE 15: SUMMARY OF EO CROP MONITORING SERVICE ................................................................................................ 46 TABLE 16: SUMMARY OF SENTINEL 2 CLOUDS, SHADOWS AND SNOW MASK TOOL................................................................ 48 TABLE 17: SUMMARY OF ONLINE SERVICE PLATFORM FOR PROCESSING SATELLITE DATA FOR FORESTRY ...................................... 51 TABLE 18: SUMMARY OF TOOLS FOR PRODUCING FORESTRY INFORMATION FROM EO DATA .................................................... 52 TABLE 19: SUMMARY OF TOOL FOR DATA MANAGEMENT AND VISUALIZATION (2D / 3D) ....................................................... 54 TABLE 20: SUMMARY OF SERVICE FOR INTERACTIVE ANALYSIS AND AGGREGATION ................................................................ 56 TABLE 21: SUMMARY OF TOOL FOR VISUALIZATION OF 2D, 3D AND 4D DATA OF A HIGH VOLUME ............................................ 61



TABLE 22: SUMMARY OF SERVICE AND TOOL FOR DATA MANAGEMENT (SENSOR DATA) .......................................................... 62 TABLE 23: SUMMARY OF TOOL FOR METADATA MANAGEMENT ......................................................................................... 66



Definitions, Acronyms and Abbreviations Acronym/

Abbreviation Title

AOI Area of Interest

API Application Programming Interface

BDVA Big Data Value Association

CAP Common Agriculture Policy

CEOS Committee on Earth Observing Satellites

CMEMS Copernicus Marine Environment Monitoring Service

EO Earth Observation

ESA European Space Agency

ECSS European Collaboration on Space Standardisation

GEO Group on Earth Observation

HMA Heterogeneous Missions Accessibility

JSON JavaScript Object Notation

INSPIRE Infrastructure for Spatial Information in Europe

IoT Internet of Things

ISO International Organisation for Standardisation

LPIS Land Parcel Identification System

MODIS Moderate Imaging Spectroradiometer

NASA National Aeronautics and Space Administration

OGC Open Geospatial Consortium

O&M Observations and Measurements

PPP Public-Private Partnership

W3C World Wide Web Consortium

WGISS Working Group on Information Systems and Services

WCPS Web Coverage Processing Service

WCS Web Coverage Service

WMS Web Map Service

WMTS Web Map Tile Service

WP Work Package

WPS Web Processing Service

XML eXtensible Markup Language



Term Definition

AQUA

Aqua is a major international Earth Science satellite mission centred at

NASA. Launched on May 4, 2002, the satellite has six different Earth-

observing instruments on board and is named for the large amount of

information it collects about water in the Earth system. Aqua gathers

this information from its stream of approximately 89 Gigabytes of data

a day. The water variables being measured include almost all elements

of the water cycle and involve water in its liquid, solid, and vapour forms.

Additional variables being measured include radiative energy fluxes,

aerosols, vegetation cover on the land, phytoplankton and dissolved

organic matter in the oceans, and air, land, and water temperatures

Bounding Box Bounding box is an expression of the maximum extents of a 2-

dimensional object (e.g. point, line, and polygon) or set of objects within

its (or their) 2-D (x, y) coordinate system, in other words min(x), max(x),

min(y), max(y).

CODE-DE The Copernicus Data and Exploitation Platform – Deutschland (CODE-

DE) is the German entry point to the Sentinel Satellite Systems, their

data products and the products of the Copernicus Services.

Commercial

Mission

The products from high resolution and very high resolution commercial

missions are purchased on the market. The term “commercial” is used

to denote both optical and radar missions.

Copernicus

Open Access

Hub

The Copernicus Open Access Hub (previously known as Sentinels

Scientific Data Hub) provides complete, free and open access to

Sentinel-1, Sentinel-2 and Sentinel-3 user products, starting from the In-

Orbit Commissioning Review (IOCR)

Crop

inadvertences

maps

A control tool used as an assessment of the validity of the declared data

(land use / crop type) included in the LPIS (Land Parcel Identification

System). The inadvertences analysis involves two main components: (a)

a pixel-based analysis, which states whether pixel values correspond or

are different from typical spectral values of the declared crop types; (b)

an object-based analysis, revealing the plots / parcels for which the

declared type of crop appears to be different from the one identified

based on satellite imagery. The plot-based analysis involves a relative

pixel count within the parcel boundary, in order to determine whether

the parcel corresponds to the crop type declared or not.

Dataset Identifiable collection of data [REF-09]

In the EO Community, a dataset is typically called “product”.

Dataset Series Collection of datasets sharing the same product specification [REF-09].

In the EO Community, a dataset series is also called “collection” or

“dataset” (in GSCDA).

Elasticsearch Elasticsearch is a search engine based on Lucene. It provides a



distributed, multitenant-capable full-text search engine with an HTTP

web interface and schema-free JSON documents. Elasticsearch is

developed in Java and is released as open source under the terms of the

Apache License (https://www.elastic.co).

Exploitation

Platform

An Exploitation Platform is a virtual workspace, providing the user

community with access to (i) large volume of data (EO/non-space data),

(ii) algorithm development and integration environment, (iii) processing

software and services (e.g. toolboxes, retrieval baselines, visualization

routines), (iv) computing resources (e.g. hybrid cloud/grid), (v)

collaboration tools (e.g. forums, wiki, knowledge base, open

publications, social networking…), (vi) general operation capabilities

(e.g. user management and access control, accounting).

FedEO FedEO (Federated Earth Observation missions access) provides a unique

entry point to a growing number of scientific catalogues and services for,

but not limited to, EO European and Canadian missions. FedEO is

deployed with ESA (European Space Agency) infrastructure as a gateway

to:

- Provide brokered discovery, access and ordering capability to

European/Canadian EO missions data based on HMA

(Heterogeneous Missions Accessibility) interfaces;

- Implement the OpenSearch OGC (Open Geospatial Consortium) and

other interfaces for an increased number of discoverable and

accessible EO data collections;

- Implement the OpenSearch OGC interfaces for interfacing with CEOS

Community Catalogues and Clients.

GeoTrellis GeoTrellis is a geographic data processing engine for high performance

applications (https://geotrellis.io/).

Kubernetes Kubernetes is an open-source container-orchestration system for

automating deployment, scaling and management of containerized

applications. It was originally designed by Google and is now maintained

by the Cloud Native Computing Foundation. It aims to provide a

"platform for automating deployment, scaling, and operations of

application containers across clusters of hosts". It works with a range of

container tools, including Docker [REF-08]

Landsat The Landsat Program provides repetitive acquisition of high resolution

multispectral data of the Earth’s surface on a global basis. The data from

Landsat spacecraft constitute the longest record of the Earth’s

continental surfaces as seen from space.

SAFE Format The SAFE (Standard Archive Format for Europe) [REF-10] has been

designed to act as a common format for archiving and conveying data

within ESA Earth Observation archiving facilities.

Special attention has been taken to ensure that SAFE conforms to the

https://www.elastic.co/

https://geotrellis.io/



ISO 14721:2003 OAIS (Open Archival Information System) reference

model and related standards such as the emerging CCSDS/ISO XFDU

(XML Formatted Data Units) packaging format.

Sentinel-1 The Copernicus Sentinel-1 earth observation mission developed by ESA

provides continuity of data from ERS and Envisat missions, with further

enhancements in terms of revisit, coverage, timeliness and reliability of

service. The SENTINEL-1 mission comprises a constellation of two polar-

orbiting satellites, operating day and night performing C-band synthetic

aperture radar imaging, enabling them to acquire imagery regardless of

the weather. The two-satellite constellation offers a 6 days revisit time.

A summary of mission objectives are:

● Monitoring sea ice zones and the Arctic environment, and

surveillance of marine environment;

● Monitoring land surface motion risks;

● Mapping of land surfaces: forest, water and soil;

● Mapping in support of humanitarian aid in crisis situations;

● Spatial Resolution: 5m, 20m, 40m.

Source: Wikipedia and Sentinel Online Web site

(https://sentinels.copernicus.eu).


provides continuity to services relying on multi-spectral high-resolution

optical observations over global terrestrial surfaces. Sentinel-2 sustains

the operational supply of data for services such as forest monitoring,

land cover changes detection or natural disasters management.

The Sentinel-2 mission offers an unprecedented combination of the

following capabilities:

● Multi-spectral information with 13 bands in the visible, near

infra-red and short wave infra-red part of the spectrum;

● Systematic global coverage of land surfaces: from 56°South to

84°North, coastal waters and all Mediterranean Sea;

● High revisit: every 5 days at equator under the same viewing

conditions;

● High spatial resolution: 10m, 20m and 60m;

● Wide field of view: 290 km.

Source: Wikipedia and Sentinel Online Web site



main objective is to measure sea-surface topography, sea- and land-

surface temperature and ocean- and land-surface colour.

A pair of Sentinel-3 satellites will enable a short revisit time of less than

two days for OLCI instrument and less than one day for SLSTR at the

equator.

https://sentinels.copernicus.eu/




Mission objectives are:

● Measure sea-surface topography, sea-surface height and

significant wave height;

● Measure ocean and land-surface temperature;

● Measure ocean and land-surface colour

● Monitor sea and land ice topography;

● Sea-water quality and pollution monitoring;

● Inland water monitoring, including rivers and lakes;

● Aid marine weather forecasting with acquired data;

● Climate monitoring and modelling;

● Land-use change monitoring;

● Forest cover mapping;

● Fire detection;

● Weather forecasting;

● Measuring Earth's thermal radiation for atmospheric

applications.

The Sentinel-3A mission has now reached the full operational capacity

and preparations for Sentinel-3B launch is-going (mission status on 6

December 2017).

Sources: Wikipedia and Sentinel Online Web site


Service Services provide functionalities to users that typically do not need to

understand the inner working of the services. These services are

typically accessed via standardized interfaces like APIs (e.g. web services

or library interfaces), interactive UIs, standard data transfer or remote

call protocols. Services refer (often) to end points that are ”black-box”

services activated remotely and executed e.g. in the cloud.

TERRA Terra aims to analyse Earth changes and the consequences. In

December 1999, NASA launched the Terra satellite as the flagship

mission of the Earth Observing System to answer these questions.

Terra carries five instruments that observe Earth’s atmosphere, ocean,

land, snow and ice, and energy budget. Taken together, these

observations provide unique insight into how the Earth system works

and how it is changing. Terra observations reveal humanity’s impact on

the planet and provide crucial data about natural hazards like fire and

volcanoes

Tool Tools are software products, typically installed on premise that, similarly

to Services, also provide ready-to-use functionalities. Often, the same

set of functionalities can be delivered either as Service or as Tool.

Third Party

Mission

ESA uses its multi-mission ground systems to acquire, process, archive

and distribute data from other satellites - so called Third Party Missions.

Source: http://earth.esa.int/missions/thirdpartymission/.


http://earth.esa.int/missions/thirdpartymission/



Introduction 1.1 Project Summary The data intensive target sector selected for the

DataBio project is the Data-Driven Bioeconomy.

DataBio focuses on utilizing Big Data to

contribute to the production of the best possible

raw materials from agriculture, forestry and

fishery/aquaculture for the bioeconomy

industry, in order to output food, energy and

biomaterials, also taking into account various

responsibility and sustainability issues.

DataBio will deploy state-of-the-art big data technologies and existing partners’ infrastructure

and solutions, linked together through the DataBio Platform. These will aggregate Big Data

from the three identified sectors (agriculture, forestry and fishery), intelligently process them

and allow the three sectors to selectively utilize numerous platform components, according

to their requirements. The execution will be through continuous cooperation of end user and

technology provider companies, bioeconomy and technology research institutes, and

stakeholders from the Big Data Value Public-Private Partnership programme.

DataBio is driven by the development, use and evaluation of a large number of pilots in the 3

identified sectors, where also associated partners and additional stakeholders are involved.

The selected pilot concepts will be transformed to pilot implementations utilizing co-

innovative methods and tools. The pilots select and utilize the best suitable market ready or

almost market ready ICT, Big Data and Earth Observation methods, technologies, tools and

services to be integrated to the common DataBio Platform.

Based on the pilot results and the new DataBio Platform, new solutions and new business

opportunities are expected to emerge. DataBio will organize a series of trainings and

hackathons to support its take-up and to enable developers outside the consortium to design

and develop new tools, services and applications based on and for the DataBio Platform.

The DataBio consortium is listed in Table 1. For more information about the project see

www.databio.eu.

Table 1: The DataBio consortium partners

Number Name Short name Country

1 (CO) INTRASOFT INTERNATIONAL SA INTRASOFT Belgium

2 LESPROJEKT SLUZBY SRO LESPRO Czech Republic

3 ZAPADOCESKA UNIVERZITA V PLZNI UWB Czech Republic

4

FRAUNHOFER GESELLSCHAFT ZUR FOERDERUNG DER

ANGEWANDTEN FORSCHUNG E.V. Fraunhofer Germany

5 ATOS SPAIN SA ATOS Spain

http://www.databio.eu/



6 STIFTELSEN SINTEF SINTEF ICT Norway

7 SPACEBEL SA SPACEBEL Belgium

8

VLAAMSE INSTELLING VOOR TECHNOLOGISCH

ONDERZOEK N.V. VITO Belgium

9

INSTYTUT CHEMII BIOORGANICZNEJ POLSKIEJ

AKADEMII NAUK PSNC Poland

10 CIAOTECH Srl CiaoT Italy

11 EMPRESA DE TRANSFORMACION AGRARIA SA TRAGSA Spain

12 INSTITUT FUR ANGEWANDTE INFORMATIK (INFAI) EV INFAI Germany

13 NEUROPUBLIC AE PLIROFORIKIS & EPIKOINONION NP Greece

14

Ústav pro hospodářskou úpravu lesů Brandýs nad

Labem UHUL FMI Czech Republic

15 INNOVATION ENGINEERING SRL InnoE Italy

16 Teknologian tutkimuskeskus VTT Oy VTT Finland

17 SINTEF FISKERI OG HAVBRUK AS

SINTEF

Fishery Norway

18 SUOMEN METSAKESKUS-FINLANDS SKOGSCENTRAL METSAK Finland

19 IBM ISRAEL - SCIENCE AND TECHNOLOGY LTD IBM Israel

20 MHG SYSTEMS OY - MHGS MHGS Finland

21 NB ADVIES BV NB Advies Netherlands

22

CONSIGLIO PER LA RICERCA IN AGRICOLTURA E

L'ANALISI DELL'ECONOMIA AGRARIA CREA Italy

23 FUNDACION AZTI - AZTI FUNDAZIOA AZTI Spain

24 KINGS BAY AS KingsBay Norway

25 EROS AS Eros Norway

26 ERVIK & SAEVIK AS ESAS Norway

27 LIEGRUPPEN FISKERI AS LiegFi Norway

28 E-GEOS SPA e-geos Italy

29 DANMARKS TEKNISKE UNIVERSITET DTU Denmark

30 FEDERUNACOMA SRL UNIPERSONALE Federu Italy

31

CSEM CENTRE SUISSE D'ELECTRONIQUE ET DE

MICROTECHNIQUE SA - RECHERCHE ET

DEVELOPPEMENT CSEM Switzerland

32 UNIVERSITAET ST. GALLEN UStG Switzerland

33 NORGES SILDESALGSLAG SA Sildes Norway

34 EXUS SOFTWARE LTD EXUS

United

Kingdom

35 CYBERNETICA AS CYBER Estonia

36

GAIA EPICHEIREIN ANONYMI ETAIREIA PSIFIAKON

YPIRESION GAIA Greece

37 SOFTEAM Softeam France

38

FUNDACION CITOLIVA, CENTRO DE INNOVACION Y

TECNOLOGIA DEL OLIVAR Y DEL ACEITE CITOLIVA Spain

39 TERRASIGNA SRL TerraS Romania



40

ETHNIKO KENTRO EREVNAS KAI TECHNOLOGIKIS

ANAPTYXIS CERTH Greece

41

METEOROLOGICAL AND ENVIRONMENTAL EARTH

OBSERVATION SRL MEEO Italy

42 ECHEBASTAR FLEET SOCIEDAD LIMITADA ECHEBF Spain

43 NOVAMONT SPA Novam Italy

44 SENOP OY Senop Finland

45

UNIVERSIDAD DEL PAIS VASCO/ EUSKAL HERRIKO

UNIBERTSITATEA EHU/UPV Spain

46

OPEN GEOSPATIAL CONSORTIUM (EUROPE) LIMITED

LBG OGCE

United

Kingdom

47 ZETOR TRACTORS AS ZETOR Czech Republic

48

COOPERATIVA AGRICOLA CESENATE SOCIETA

COOPERATIVA AGRICOLA CAC Italy

1.2 Document Scope The deliverable D5.3 – EO Services and Tools provides a user-oriented perspective on the

technical deliveries of the DataBio project in the scope of handling big Earth Observation (EO)

and geospatial data. The reading will be specifically interesting for people, who are familiar

with EO and geospatial applications and seek innovative solutions for the upcoming

challenges of storing, managing, processing, analysing and visualizing vast amount of EO and

geospatial data.

The document reports on the services and tools that will be delivered by DataBio partners for

handling big EO and geospatial data. These services and tools are generated by adapting,

extending and implementing the software components provided as basic technology (see

[REF-05] and [REF-07]). This process of generating tailored services has been triggered by the

requirements specifications of various pilot applications in the DataBio project coming from

the agricultural, fishery and forestry domains. These pilot applications formulate their specific

needs and adapted services and tools are generated based on existing components in order

to meet these requirements. These services and tools are deployed in the scope of individual

pilots but might be of interest for other DataBio pilots or for external users of the DataBio

platform as well. They can be used to offer directly functionalities in agriculture, forestry and

fishery applications or can be used, as part of the DataBio toolset, to build such applications

The document starts with an overview in Chapter 2 about the types of services and tools

delivered along with a short description about their purpose and functionality.

In case that you find an interesting functionality described in this table, you can directly move

to the respective section in Chapter 3, which provides for each type of service or tool:

• A narrative example of how the service or tool is used in a typical application

• Information on the ability of this piece of software for handling big data



• Documentation about the external accessibility of the technology along with contact

points and information on IPR and licenses.

1.3 Document Structure

This document is comprised of the following chapters:

Chapter 1 presents an introduction to the project and the document.

Chapter 2 provides an overview about the types of services and tools delivered along with a

short description about their purpose and functionality.

Chapter 3 provides details for each service / tool in order to enable external readers to assess

the usefulness for their individual purposes.



EO and Geospatial Services and Tools

delivered for DataBio This chapter provides an overview about the types of services and tools delivered by DataBio

partners along with a short description about their purpose and functionality. The services

and tools are generated by extending, adapting or composing existing software components

in order to fulfil specific pilot requirements. The underlying software components are

described in detail in the DataBio deliverables D5.1 EO Component Specification [REF-05] and

D5.2 EO Component and Interfaces [REF-07]. Both deliverables are public and can be

accessed for further technical details. Further details related to the DataBio Pilot applications

and their requirements are documented in the respective deliverables D1.1: Agriculture Pilot

Definition [REF-01], D2.1: Forestry Pilot Definition [REF-02], and D3.1: Fishery Pilot

Definition [REF-03].

Services provide functionalities to users that typically do not need to understand the inner

working of the services. These services are typically accessed via standardized interfaces like

APIs (e.g. web services or library interfaces), interactive UIs, standard data transfer or remote

call protocols. Services refer (often) to end points that are “black-box” services activated

remotely and executed e.g. in the cloud. Tools are software products, typically installed on

premise that, similarly to Services, also provide ready-to-use functionalities. Often, the same

set of functionalities can be delivered either as Service or as Tool.

For each Type of Service or Tool listed in Table 2, you can directly move to the corresponding

section in Chapter 3 for more detailed information.

Table 2: List of EO and Geospatial Services and Tools delivered for the DataBio-Platform

and Pilots

Section Type of

Service or

Tool

Description Involved

components

Responsible

Partner

3.1 Tool for data

management

and

processing

Rasdaman array-database enables

scalable storage, retrieval and in-situ

processing of extremely large

amounts (petabytes) of multi-

dimensional and spatio-temporal

arrays-based information (such as

such as sensor, image, and statistics

data).

C05.01

Rasdaman

ATOS Spain

AS, Spain

3.2 Service and

tool for data

download and

This Spacebel service enables

systematically downloading, storing

and cataloging EO products from EO

C07.01 –

FedEO

Gateway

Spacebel SA,

Belgium



access (EO

images)

data providers for subsequent local

data analytics.

C07.03 –

FedEO Catalog

C07.04 - Data

Manager

C07.06 –

Ingestion

Engine

3.3 Tool for

Images and

Orthophotos

processing

This tool provides colour correction

and homogenization process of

orthophotos from different areas

and/or dates. This tool increase

orthophotos homogeneity and

improve their subsequent

possibilities of use, both for agrarian

and environmental purposes, using

image analysis automatized

processes.

C11.03 –

Radiometric

Corrections

Empresa de

Transformac

ion Agraria

SA (Tragsa),

Spain

3.4 Tool to exploit

derived

data/informat

ion from EO

data sources

(Dashboard)

A frontend tool will be developed

allowing the users to exploit

data/information derived from

Sentinel-2 and meteo data (e.g.

vegetation parameters). This

exploitation includes viewing

services, time series viewing for user-

tailored polygons, editing and

viewing of field-level in-situ data,

output from models.

C08.02 Proba-

V MEP

Vlaamse

Instelling

voor

Technologis

ch

Onderzoek

N.V. (VITO),

Belgium

3.5 Backend

services for

processing on

Sentinel-2 and

meteo data

The Proba-V MEP provides several

backend services to process Sentinel-

2 data, provide WMTS service for the

derived products and meteo data,

datacube RESTful API for these data,

ElasticSearch for in-situ data.

C08.02 Proba-

V MEP

Vlaamse

Instelling

voor

Technologis

ch

Onderzoek

N.V. (VITO),

Belgium

3.6 Service for the

fusion of

Sentinel-1 and

Sentinel-2

data over a

restricted AoI

Fusion of datasets (e.g. Sentinel-1

and Sentinel-2) is prototyped to

provide temporal more dense time

series than can be realised by using

Sentinel-2 only, because of frequent

clouds.

C08.02 Proba-

V MEP

Vlaamse

Instelling

voor

Technologis

ch

Onderzoek



and date

range

N.V. (VITO),

Belgium

3.7 Tool for data

processing

(Copernicus

data)

This tool will process in a scalable

way massive amount of data and it is

designed to be cross-platform

(multiple cloud) and targeting open

Copernicus data.

C28.01-eEOPS

- e-GEOS EO

processing

service

E-Geos SPA,

Spain

3.8 Tool for data

access and

visualization

(EO and

climate data)

This tool allows the spatial and

temporal sub-setting of geospatial

raster data. It is a fully operative

implementation of the data cube

concept based on OGC WCS 2.0

standard and it supports all the

raster data formats implemented for

EO data and climate data.

C41.01

MEA.WCS

server

C41.02 MEA

GUI.

Meteorologi

cal and

Environmen

tal Earth

Observation

SRL (Meeo),

Italy

3.9 Mosaic Cloud

Free

Background

Service

This service constructs and keeps an

up to date collage (mosaic) of

Sentinel2 and Lansat8 images,

covering a given area of interest

(AOI) with the latest, cloud free

satellite scenes. The fusion and

harmonization between the two

types of sensor data are made only at

RGB level, mainly for eye inspection,

but also for other possible advanced

processing. A similar service is

provided for Sentinel1 data (for both

ascending and descending orbits),

offering a false colour map of the

same AOI.

C39.01 –

Mosaic Cloud

Free

Background

C39.03 - S2

Clouds,

Shadows and

Snow Masks

Terrasigna

SRL,

Romania

3.10 EO Crop

Monitoring

Service

The service assesses the agriculture

parcels from satellite data and

farmers declarations in order to

create a series of products like Crop

masks, Parcels use maps and Crop

inadvertences maps, in support of

the Common Agriculture Policy

(CAP). As input data, it relays on

Copernicus Sentinel 2 data and

farmers declaration of intention with

respect of crops types.

C39.02 - EO

Crop

Monitoring

C39.03 - S2

Clouds,

Shadows and

Snow Masks

Terrasigna

SRL,

Romania



3.11 Sentinel 2

Clouds,

Shadows and

Snow Mask

Tool

The tool produces Sentinel2 Clouds,

Shadows and Snow Masks, based

only on raw data, improving the

results of the genuine quality

assessment band, without any

external data. The process chain is

based on a formula developed by

TERRASIGNA and was intensely

tested on many full Sentinel2 scenes,

in all seasons and in various

geographical situation. Internal

benchmarking shown better

performances than other known

solution.

C39.03 - S2

Clouds,

Shadows and

Snow Masks

Terrasigna

SRL,

Romania

3.12 Online service

platform for

processing

satellite data

for forestry

Forestry TEP (Forestry Thematic

Exploitation Platform) is a novel

online solution for efficient

processing of satellite data for

analysis and monitoring of forests. It

offers global optical and radar

satellite imagery, processing services

and tools, and capability for users to

develop and share their own

processing services.

C16.10 -

Forestry TEP

VTT

Technical

Research

Centre of

Finland Ltd,

Finland

3.13 Tools for

producing

forestry

information

from EO data

Probability is a toolset for estimating

forest variables based on EO data

and reference data. AutoChange is a

tool for forest/land cover change

detection based on two-time instant

images. Envimon is a set of EO data

pre-processing tools for unpacking,

radiometric corrections and

geometric corrections.

C16.07 -

Probability,

C16.08 -

AutoChange,

C16.09 -

Envimon

VTT

Technical

Research

Centre of

Finland Ltd,

Finland

3.14 Tool for Data

management

and

visualization

(2D / 3D)

This tool will manage a massive

amount of geospatial data in a fast,

intuitive and interactive way. It will

provide the means for visually

querying and accessing the data

considering not only the spatial and

temporal extent but the attributes as

well.

C04.02-

GeoRocket +

Groundstation

, C04.04-

SmartVis3D,

C04.03-

GeoToolbox

Fraunhofer

Gesellschaft

zur

Förderung

der

angewandte

n Forschung

e.V.,

Germany

3.15 Service for

Interactive

This service allows to analyse and

aggregate data in an interactive way.

C04.02-

GeoRocket +

Fraunhofer

Gesellschaft



Analysis and

Aggregation

Starting with a simple query (e.g.

spatial extend), it will be possible to

add more constraints, or aggregation

functions to build and refine a query

for analysing geospatial data.

Groundstation

, C04.03-

GeoToolbox

zur

Förderung

der

angewandte

n Forschung

e.V.,

Germany

3.16 Tool for

visualization

of 2D, 3D and

4D data of a

high volume

The tool can handle both vector and

mosaic data. Moreover, it is possible

to process and visualize various types

of georeferenced data structures:

- tables and relationships (entity-

relationship-attribute model,

database)

- tree (hierarchical data, such as

XML or JSON based datasets)

- general graph (linked data, RDF)

Particularly, those data connectors

exist:

- PostGIS database

- Web Map Service for raster and

imagery data

- GeoJSON for vector data

- Resource Description Framework

(RDF) for linked data

- OpenStreetMap live data pump

for vector data from OSM

C02.01-

SensLog,

C19.01-

Proton,

C02.05-

FarmTelemetr

y,

C02.06-Data

model for PA

Lesprojekt –

sluzby s.r.o.,

Czech

Republic

3.17 Service and

tool for Data

management

(Sensor data)

This component is a web-based

sensor data management application

that can be used as tool in

standalone installation or can be

used as service on general instance

for DataBio.

The component is receiving sensor

data from data producers’

components, stores data in the

database and publish stored data by

the system of RESTful web services.

C02.03 -

HSLayers,

C02.05

FarmTelemetr

y, C02.06 Data

model for PA

University of

West

Bohemia

(UWB),

Czech

Republic

3.18 Tool for

Metadata

Management

(Open) Micka is a web application for

management and discovery of

geospatial metadata. The following

features are supported by an (Open)

C02.03-

HSLayers NG

2.1.0

Lesprojekt –

sluzby s.r.o.,

Czech

Republic



Micka: OGC Catalogue service (CSW

2.0.2), Transactions and harvesting,

Metadata editor, Multilingual user

interface, ISO AP 1.0 profile, Feature

catalogue (ISO 19110), Interactive

metadata profiles – management,

WFS/Gazetteer for defining metadata

– extent, GEMET thesaurus built-in

client, INSPIRE registry built-in client,

OpenSearch, INSPIRE ATOM

download service - automatically

creation from metadata.

C02.04-

HSLayers

Mobile 2.1.0

C02.02 sub-

component

Metadata

editor



Details for an individual assessment on the

usefulness of each Service or Tool For each of the services and tools introduced in the table above, we provide the following

information:

1. A narrative example of how the service / tool is used in a typical application.

The narrative example provides a concrete insight into the functionality of the service or tool

independently form any technical details. It supports the readers in the evaluation whether a

service / tool might be useful for their specific requirements. The services and tools can be

used to offer directly functionalities in agriculture, forestry and fishery applications or can be

used, as part of the DataBio toolset, to build such applications

2. Information on the ability for handling big data.

This information will foster a better understanding for the innovative character of the offered

technology and provides sound argumentation why DataBio technology will exceed the

capabilities of current standard software packages.

3. Documentation

Further to the basic understanding of the underlying technology and the ability to assess its

usefulness, it is essential to know about the terms and conditions for using these services and

tools. Thus, for each service and tool, we provide a brief documentation of all information

needed by an interested party to be able to make access and use them.

3.1 Tool for Data Management and Processing (spatio-temporal

sensor, image, simulation and statistics data) Rasdaman array-database enables scalable storage, retrieval and in-situ processing of

extremely large amounts (petabytes) of multi-dimensional and spatio-temporal arrays-based

information (such as such as sensor, image, and statistics data).

3.1.1 Narrative

For the improvement of the profitability of oceanic tuna fisheries through savings in fuel costs

through fish observation and route optimization, data scientists want to know what are the

best fish catch areas in the whole Indian Ocean, where the company vessels operate, and they

want to determine the best routes for reaching them. For that, the data analysts need to store

and analyse in an efficient manner massive amounts of data from the vessels’ sensors (i.e.,

engines, propulsion, route and speed of the vessel, destination), in combination with

SENTINEL-3 and Copernicus Marine Environment Monitoring Service (CMEMS) data (i.e. sea

surface currents, temperature, wind speed, chlorophyll, phytoplankton and other

oceanographic parameters), as well as current weather conditions information and models.

To carry out these analyses, historical data covering the whole Indian Ocean region and

several years backward (as well as current one) has to be stored.



3.1.2 Technology for handling big data

Rasdaman is an array database system which provides flexible, fast, scalable geo services for

multi-dimensional (1-D, 2-D, 3-D, 4-D, and beyond) spatio-temporal sensor, image,

simulation, and statistics data of unlimited volume. Ad-hoc access, extraction, aggregation, as

well as remix and analytics is enabled through an SQL raster query language (RASQL) with

highly effective server-side optimization.

Rasdaman incorporates efficient spatial indexing and adaptive tiling for fast data access. In

this regard, raster objects partitioned into tiles and stored in the file system (aside from a

regular subdivision, any user or system generated partitioning is possible), a

PostgreSQL/SQLite database is used to store metadata about each raster object and its

related tiles.

Its deployment flexibility enables to install it in a simple laptop computer or in a more complex

setup in the cloud, where data is distributed among each node (each node has a Rasdaman

instance acting as a slave, being a designated node as the acting master that distributes the

requests based on the data being queried). The later approximation allows parallelizing the

requests (the request is decomposed into smaller queries and then distributed among the

nodes that contain the data) and providing tile streaming, providing thus unlimited scalability.

This can be further complemented with the use of available GPUs in the system in order to

more efficiently process the raster data.

3.1.3 Documentation

Table 3: Summary of Tool for Data Management and Processing (spatio-temporal sensor,

image, simulation and statistics data)

Tool for Data Management and Processing (spatio-temporal sensor, image, simulation and

statistics data)

IPR Owner rasdaman GmbH (developed by Jacobd University, Germany)

http://www.rasdaman.com/

Contact Miguel Ángel Esbrí

Email [email protected]

Technology PostgresSQL Database >= 9.5

SQLite

Tomcat

gdal

Ubuntu/CentOS

URL http://www.rasdaman.org/

http://www.rasdaman.com/

mailto:[email protected]

http://www.rasdaman.org/



Deployment type Server architecture deployed on premise

Endpoint Not applicable (it is currently deployed at AZTI infrastructure, only

accessible via private VPN)

Documentation http://www.rasdaman.org/wiki/Documentation

IPR Rasdaman has dual license (open-source / commercial) depending on the

features available

Specific system

requirements

JavaRuntimeEnvironment (JRE)

Data types

handeled

spatio-temporal sensor, image, simulation and statistics data

3.2 Service and Tool on Data Download and Access (EO images) This service enables systematically downloading, storing and cataloguing EO products from

EO data providers for subsequent local data analytics.

3.2.1 Narrative

For the implementation of an automated processing chain based on EO images, the service

provider needs to discover new EO products available at EO data providers and systematically

download, store and catalogue these products for subsequent local data analytics. The

downloaded data accumulate over time, thus increasing in volume and velocity. For example,

in case of the Fishery Pilots in DataBio, the software components are installed at the premises

of the Spanish Foundation AZTI, and EO data from Copernicus Open Access Hub and CMEMS

are systematically retrieved for further processing.

The data access is highly configurable for the specific needs of an application. The following

list provides examples of various configurations for a service instantiation in terms of EO data

provider, type of EO product, Area of Interest, cloud coverage, etc.:

• Download and access service for Sentinel-2 L1C data with cloud coverage lower than

25% over Czech and Finnish territories

• Download and access service for SRTM DEM 1 sec HGT (30m) datasets over Czech and

Finnish territories

• Download and access service for Sentinel-2 Level-2A products corresponding to

(29TQH, 30TUN, 30TTN & 30TUM) tiles data with cloud coverage lower than 5%

• Access service to MODIS high resolution: Aqua (2002-present) and Terra (1999-

present)

http://www.rasdaman.org/wiki/Documentation



Figure 2: Data Manager Collections configuration file (extract)

Figure 3: Ingestion Engine Requests configuration file (extract)


The Spacebel Download and Access Services and Tools are composed of the software

components FedEO Gateway, FedEO Catalog, Data Manager and Ingestion Engine. The

components can be deployed on heterogeneous infrastructures (e.g. bare-metal, hypervisor

or cloud) and work is underway to package all components as (Docker) containers that are



easily deployed on a Kubernetes Cluster (e.g. Google, Amazon AWS, Microsoft Azure,

Interoute) by middle of 2018. Handling a vast amount of metadata and data is as easy as

adding new nodes to the Kubernetes cluster. Utilizing the open source framework Apache

SolrCloud and ZooKeeper, allows doing high-performance spatial queries on the FedEO

Catalog and/or FedEO Gateway [REF-11]. The components implement state-of-the-art

interoperable interfaces defined by CEOS and OGC.

Most of the above tools are being used operationally at ESA/ESRIN (FedEO) [REF-11] and/or

DLR (CODE-DE).

The FedEO Gateway component acts as a unique endpoint allowing clients to access metadata

and data from different backend EO catalogues implementing different protocols. It supports

access through OGC Open Search interfaces OGC 10-032r8 (Geo and time extension of

OpenSearch [REF-12]) and OGC 13-026r8 (EO Extension of OpenSearch [REF-13]) and provides

Atom or GeoJSON responses with metadata in OGC 10-157r4 format (i.e. EO Profile

Observations and Measurements [REF-14]). This component access data providers using a

unique standard protocol, thus facilitates adding additional EO data providers with minimal

changes.

The FedEO Catalog component implements an EO catalogue server that allows storing

downloaded or derived EO (satellite) collections (series) and products (datasets) metadata. It

offers an API to populate the catalogue and an API to search the catalogue.

3.2.3 Documentation

Table 4: Summary of Service and Tool on Data Download and Access (EO images) – FedEO

Gateway component

FedEO Gateway

IPR Owner Spacebel S.A.

Contact Yves Coene

Email [email protected], [email protected]

Technology Java application, Apache http client (v4.5), Apache commons, Xalan-Java,

Apache Xerces for Java, Log4J, Apache Tomcat, Geonames, Axis2/Java, Axiom,

JDOM, JSON.simple, Saxon9HE, Jersey

URL http://ceos.org/ourwork/workinggroups/wgiss/access/fedeo/

Deployment

type

Can be made available as software to be deployed elsewhere or can be hosted

as a service.

Endpoint http://fedeo.esa.int/opensearch/readme.html

http://geo.spacebel.be/opensearch/readme.html



http://ceos.org/ourwork/workinggroups/wgiss/access/fedeo/

http://fedeo.esa.int/opensearch/readme.html




Documentation This software was developed for the European Space Agency. Therefore, ECSS-

compliant technical documentation is available including Software

Requirements Specification (SRS), Software Design Document (SDD), Interface

Control Document, Software Configuration File (SCF), Software Validation

Specification (SVS), Software Validation Report, Software Release Document

(SRelD).

• ECSS European Cooperation for Space Standardization,

http://ecss.nl/standard/ecss-e-st-40c-software-general-requirements

• CEOS OpenSearch Best Practice,

http://ceos.org/document_management/Working_Groups/WGISS/Interest_Gr

oups/OpenSearch/CEOS-OPENSEARCH-BP-V1.1.1-Final.pdf

• OGC 10-032r8 – Geo and Time extension of OpenSearch,

http://www.opengeospatial.org/standards/opensearchgeo

• OGC 13-026r8 – EO extension of OpenSearch,

http://docs.opengeospatial.org/is/13-026r8/13-026r8.html

• http://www.opensearch.org/Specifications/OpenSearch/1.1

IPR Commercial license.

Specific system

requirements

JRE

Table 5: Summary of Service and Tool on Data Download and Access (EO images) – FedEO

Catalog component

FedEO Catalog


Contact Yves Coene


Technology Java application, Tomcat, Apache Solr, Apache http client, Apache

commons,

Apache Xerces for Java, Log4J, Xalan-Java, Jaxen, Geonames, JDOM, WSDL 4

Java

URL http://ceos.org/ourwork/workinggroups/wgiss/access/fedeo/

Deployment type Can be made available as software to be deployed elsewhere or can be

hosted as a service.

Endpoint http://fedeo.esa.int/opensearch/readme.html



http://ceos.org/document_management/Working_Groups/WGISS/Interest_Groups/OpenSearch/CEOS-OPENSEARCH-BP-V1.1.1-Final.pdf

http://ceos.org/document_management/Working_Groups/WGISS/Interest_Groups/OpenSearch/CEOS-OPENSEARCH-BP-V1.1.1-Final.pdf



http://www.opensearch.org/Specifications/OpenSearch/1.1



http://ceos.org/ourwork/workinggroups/wgiss/access/fedeo/

http://fedeo.esa.int/opensearch/readme.html




Documentation This software was developed for the European Space Agency. Therefore,

ECSS-compliant technical documentation is available including Software

Requirements Specification (SRS), Software Design Document (SDD),

Interface Control Document, Software Configuration File (SCF), Software

Validation Specification (SVS), Software Validation Report, Software Release

Document (SRelD).

• ECSS European Cooperation for Space Standardization,


IPR Commercial license.

Specific system

requirements

JRE

Table 6: Summary of Service and Tool on Data Download and Access (EO images) – Data

Manager component

Data Manager


Contact Yves Coene


Technology Java application, Glasfish, NFS, HDFS

URL https://proba-v-mep.esa.int/about-mep-proba-v/mep-proba-v-architecture

Deployment type Can be made available as software to be deployed.

Endpoint


ECSS1-compliant technical documentation is available including Software


Software Configuration File (SCF), Software Validation Specification (SVS),

Software Validation Report, Software Release Document (SRelD).

IPR Dual license:

- Commercial license

- Open-source license (upon completion of ESA Project).

1 http://ecss.nl/standard/ecss-e-st-40c-software-general-requirements/




https://proba-v-mep.esa.int/about-mep-proba-v/mep-proba-v-architecture

http://ecss.nl/standard/ecss-e-st-40c-software-general-requirements/



Specific system

requirements

JRE

Table 7: Summary of Service and Tool on Data Download and Access (EO images) – Ingestion Engine component

Ingestion Engine


Contact Yves Coene


Technology Java application

URL

Deployment type Can be made available as software to be deployed.

Endpoint


ECSS2-compliant technical documentation is available including Software


Software Configuration File (SCF), Software Validation Specification (SVS),

Software Validation Report, Software Release Document (SRelD).

IPR Dual license:

- Commercial license

- Open-source license (upon completion of ESA Project).

Specific system

requirements

Requires C07.04 - Data Manager.

JRE

3.3 Tool for Images and Orthophotos processing This tool provides colour correction and homogenization process of orthophotos from

different areas and/or dates. This tool increase orthophotos homogeneity and improve their

subsequent possibilities of use, both for agrarian and environmental purposes, using image

analysis automatized processes.

2 http://ecss.nl/standard/ecss-e-st-40c-software-general-requirements/



http://ecss.nl/standard/ecss-e-st-40c-software-general-requirements/



3.3.1 Narrative

The Spanish National Plan of Aerial Orthophotography (PNOA) aims to obtain digital aerial

orthophotos with resolution of 25 or 50 cm and digital elevation models (DEM) of high

precision throughout the Spanish territory, with an update period of 2 or 3 years, according

to the zones. It is a cooperative project and co-financed between the General State

Administration and the Autonomous Communities.

Due to meteorological conditions, flight dates or camera features can be very different; the

resulting images are disparate in some areas. The Radiometric Component homogenizes the

images and their spectral response.

Figure 4: Image enhancement framework of the tool for Images and Orthophotos processing


From several photogrammetric flights, a rigorous treatment of the data is carried out,

complying with agreed technical specifications among all the participating Public

Administrations and covering all Spanish Surface. This decentralized and cooperative

production approach between the different administrations is in line with the spirit of the

Inspire Directive for the establishment of a Geographical Data Infrastructure in Europe, which



seeks to capture the maximum level of geographical information at a single time and that is

shared openly among the different social agents that need it.

Aerial photography is the basis for the realization of cartography and geographic information

in general, land occupation, urban planning and land management, cadastre, forest

management, hydrography, etc. Using different photogrammetric data, a significant

geometrical and temporal coherence of the cartographic and geographical databases existing

in all administrations is also achieved. The characteristics of the products obtained by this tool

satisfy the needs of all the administrations involved.

The following images transformation algorithms have been implemented to process the aerial

orthophotos:

• BW2: Testing algorithm

• Levels: colour levels adjustment

• Interblocks: processing of big sets of images

• ATRPol: the main algorithm of radiometric correction and spectral unification

All of them are applied to a considerable number of images. Currently, it is applied on a

centralized server, but it has been designed to use parallelized processing on HPC

environments.

3.3.3 Documentation

Table 8: Summary of Tool for Images and Orthophotos processing

Tool for Images and Orthophotos processing

IPR Owner TRAGSA Group (TRAGSA & TRAGSATEC)

Contact Jesús Estrada


Technology Radiometric Correction is based on Polynomial Radiometric Aerotriangulation

algorithm

(https://www.researchgate.net/publication/312047120_AEROTRIANGULACION

_RADIOMETRICA_POLINOMIAL) and composed by different software

components developed on Microsoft .NET platform.

URL www.databio.eu

Deployment type Client tool deployed on premise

Endpoint Not applicable

Documentation A manual about the Radiometric Correction component is available in Spanish.

Please get in contact with Tragsa / Tragsatec, who will provide English

documentation and support on request


https://www.researchgate.net/publication/312047120_AEROTRIANGULACION_RADIOMETRICA_POLINOMIAL

https://www.researchgate.net/publication/312047120_AEROTRIANGULACION_RADIOMETRICA_POLINOMIAL




IPR Radiometric correction is a commercial product

Specific system

requirements

Microsoft Windows Developed on .NET platform.

Data types

handled

Orthophotos

3.4 Tool to exploit derived data/information from EO data sources

(Dashboard) A frontend tool will be developed allowing the users to exploit data/information derived from

Sentinel-2 and meteo data (e.g. vegetation parameters). This exploitation includes viewing

services, time series viewing for user-tailored polygons, editing and viewing of field-level in-

situ data, output from models.

3.4.1 Narrative

The end users, who can be individual farmers want a user-friendly Web-based UI to explore

the information extracted from EO-data (Copernicus data, meteo data) and in-situ/model

data, at field level. The UI shall present the information in a way that this is understandable

by farmers, i.e. not EO-data scientists.


The UI links to data provided by different Web services which are exposed by different big

data technologies at the Proba-V MEP platform, e.g. data cube powered by GeoTrellis and

ElasticSearch. Also, OGC standards for e.g. viewing are used. This approach demonstrates that

scalable big data platforms, such as Proba-V MEP, can expose their data via Web services

which allows clients to extract small pieces of information from these (i.e. several) big data in

synchronous communication i.e. in a few seconds.

3.4.3 Documentation

Table 9: Summary of Tool to exploit derived data/information from EO data sources (Dashboard)

Tool to exploit derived data/information from EO data sources (Dashboard)

IPR Owner VITO NV

Contact Erwin Goor


Technology Angular 5

URL http://databio.vgt.vito.be/


http://databio.vgt.vito.be/



Deployment type Deployed on Proba-V MEP

Endpoint http://databio.vgt.vito.be/

Documentation Not Applicable

IPR Commercial product of VITO

Specific system

requirements

Not Applicable

Data types

handled

Derived data /information from EO data sources

3.5 Backend services for processing on Sentinel-2 and meteo data The Proba-V MEP provides several backend services to process Sentinel-2 data, provide

WMTS service for the derived products and meteo data, datacube RESTful API for these data,

ElasticSearch for in-situ data.

3.5.1 Narrative

For some of the DataBio pilots from the agricultural domain, the different capabilities of the

Proba-V MEP platform hosted by VITO are used to process and analyse Sentinel-2 data, to

make meteorological data from different providers and in-situ data exploitable. A scalable

processing chain is provided on the Proba-V MEP hosted Hadoop/Spark cluster and several

tools (Geotrellis, ElasticSearch) are used to make these data searchable. Also, GeoServer is

used to offer WMTS services on raster data.

To develop models for the pilots, the Proba-V MEP hosted Jupyter Notebook solution is used

to write and test the model written in python or R and execute it on the Proba-V MEP platform

where the data is present.


The Proba-V MEP platform capabilities are brought in as background by VITO and consists of

the following big data support:

• OpenStack private cloud with access to all the data hosted at the VITO datacentre,

where user VMs can be provided

• Access the ESA PAC (Scihub) to download Sentinel-2 data to the platform

• Hadoop/Spark processing cluster

• Jupyter notebooks

• Time series viewer, powered by Geotrellis

• ElasticSearch

• OpenSearch catalogue interface on the data archive

• Geoserver providing WMTS and WCS webservices

http://databio.vgt.vito.be/



The Proba-V MEP platform was specifically developed for serving the land community in

developing and operating applications. It is used in DataBio to provide this service to several

pilots from the agriculture domain.

3.5.3 Documentation

Table 10: Summary of Backend services for processing on Sentinel-2 and meteo data

Backend services for processing on Sentinel-2 and meteo data

IPR Owner VITO NV

Contact Erwin Goor


Technology OpenStack, Jupyter Notebook, Hadoop, Spark, Geotrellis, ElasticSearch,

GeoServer

URL https://proba-v-mep.esa.int/

Deployment type Client-Server architecture deployed on premise

Endpoint https://proba-v-mep.esa.int/documentation/manuals/web-interfaces

Documentation https://proba-v-mep.esa.int/documentation/manuals/developer-guide


Specific system

requirements

Deployed at the VITO data centre on CentOS

Data types

handled

Sentinel 2 and meteorological data

3.6 Service for the fusion of Sentinel-1 and Sentinel-2 data over a

restricted AoI and date range Fusion of datasets (e.g. Sentinel-1 and Sentinel-2) is prototyped to provide temporal more

dense time series than can be realised by using Sentinel-2 only, because of frequent clouds.

3.6.1 Narrative

For several pilots from the agriculture domain the fusion of Sentinel-1 and Sentinel-2 is

absolutely necessary since only using Sentinel-2 is not enough in many places worldwide for

field-based monitoring because of the lack of frequent observations due to clouds.

In DataBio a sample dataset will be processed over a restricted Area of Interest (AoI) and will

be made available on Proba-V MEP in the datacube. So, the activity is limited to a prototype.


https://proba-v-mep.esa.int/

https://proba-v-mep.esa.int/documentation/manuals/web-interfaces

https://proba-v-mep.esa.int/documentation/manuals/developer-guide




The Proba-V MEP platform is used for this activity, because:

• Sentinel-1 and Sentinel-2 data over the pilot areas is available

• A VM is provided so that different researchers can develop the source code on the

platform with access to all the data

• A Machine Learning framework is provided on Proba-V MEP, which is needed for the

Data Fusion.

• The Geotrellis Data cube allow to access pixels for a user-defined polygon very

efficiently

The strength of the Proba-V MEP is that the platform can be used to develop the source code,

perform tests and can be used, thanks to the Hadoop/Spark cluster, for an operational setup

later on.

3.6.3 Documentation

Table 11: Summary of Fused Sentinel-1 and Sentinel-2 data over a restricted AoI and date

range

Fused Sentinel-1 and Sentinel-2 data over a restricted AoI and date range

IPR Owner VITO NV

Contact Erwin Goor


Technology Python script, running on Hadoop/spark cluster provided by Proba-V MEP

URL https://proba-v-mep.esa.int/

Deployment type Deployed on Proba-V MEP

Endpoint The data will be accessible via https://proba-v-

mep.esa.int/documentation/manuals/time-series-query-api

Documentation Not Applicable


Specific system

requirements

Not Applicable

Data types

handled

Sentinel 1 and Sentinel 2


https://proba-v-mep.esa.int/

https://proba-v-mep.esa.int/documentation/manuals/time-series-query-api

https://proba-v-mep.esa.int/documentation/manuals/time-series-query-api



3.7 Tool for data processing (Copernicus data) This tool will process in a scalable way massive amount of data and it is designed to be cross-

platform (multiple cloud) and targeting open Copernicus data.

3.7.1 Narrative

For the implementation of the Common Agricultural Policy (CAP) a data analyst wants to know

the distribution of crops in a given region in order to check if they comply with what was

declared. For this, the data required are normally Sentinel 2 and Landsat data that should be

pre-processed (atmospheric correction, cloud detection, mosaicking, co-registration and

stacking) to extract time-series trends related to parcels.


The e-GEOS EO processing service is an architecture which is used to perform EO image

processing from pre-processing steps to final information extraction. It works with following

interfaces:

• REST (all data/parameters sent to a Job using POST)

• FTP (data transferred using FTP and parameters sent to a Job using POST)

It can distribute jobs on commodity hardware using agent architecture.

The processing includes several steps with different purposes such as:

• Data retrieval: Sentinel, Landsat, MODIS, and other data download based on typical

use cases (e.g. AOI and time range). Support also generic FTP transfer for traditional

commercial EO data provision.

• Data pre-processing:

o Optical data: atmospheric correction, cloud detection, cloud-free mosaic

generation

o SAR data: co-registration, coherence, multi-temporal coherence, map of

interferometry, segmentation

• Feature extraction:

o Optical: classification, feature extraction, band calculation to provide spectral

indexes

o SAR data: ship detection, oil detection, flood detection

Pilots can submit their needs in terms of AOI and time range and then get base layers (e.g.

change detection for forest monitoring, ship detection for fishery monitoring in natural areas)

to integrate with other data into their business case.



3.7.3 Documentation

Table 12: Summary of Tool for data processing (Copernicus data)

Tool for data processing (Copernicus data)

IPR Owner E-GEOS

Contact M.Corsi

Email [email protected] (please include in your email a reference to the

service/tool and the contact person, so your request can be forwarded)

Technology Based on open source products like Geoserver, Jenkins and on internal

developments

URL www.databio.eu

Deployment type ClieSaaS


Documentation N/A

IPR N/A

Specific system

requirements


Data types

handled

Copernicus data

3.8 Tool for data access and visualization (EO and climate data) This tool allows the spatial and temporal subsetting of geospatial raster data. It is a fully

operative implementation of the datacube concept based on OGC WCS 2.0 standard and it

supports all the raster data formats implemented for EO data and climate data.

MEA.WCS server and MEA GUI have been implemented in eodataservice.org web portal

where datacubes of EO and climate data are made available and accessible.

A datacube is a massive multi-dimensional array, also called “raster data” or “gridded data”;

“massive” entails that we talk about sizes significantly beyond the main memory resources of

the server hardware. Data values, all of the same data type, sit at grid points as defined by

the d axes of the d-dimensional datacube. Coordinates along these axes allow addressing data

values unambiguously. A datacube implementation based on OGC WSC standard makes

possible the effective retrieval of multidimensional geospatial datasets.





3.8.1 Narrative

For the implementation of the Farm Weather insurance assessment, climate data such as

precipitation and temperature have been collected over the Netherland to make some

statistical analysis and comparison between the climate conditions and records of crop

damages and losses provided by insurance companies. In that case the proposed technology

has been used to collect different kind of data (precipitation, hail, temperature, NVDI) in

different map projections and data format and make them available and usable for data

analysis. The datasets have not been modified with respect to their original content (i.e.

interpolation) and format (i.e. from HDF to GeoTIFF), but they have been registered into

MEA.WCS server where data access services provided with specific data readers have been

implemented. By means of the WCS service implemented into MEA.WCS server, any kind of

extraction can made out on-the-fly with the benefit of not moving big amount of useless data

on the web but only the subset (time series or area of interest) really needed for the analysis.

Thanks to this technology the duplication of data can be avoided and most part of the data

preparation for analysis can be skipped or reduced.


The Multisensor Evolution Analysis (MEA) technology facilitates the access to full resolution

heterogeneous datasets providing in a one-stop-shop the access services and basic data

mining tools necessary to explore geospatial data. The system is easy to understand

regardless of language barriers by the use of pictogram icons for specific functions.

MEA adopts open source technologies to provide the users advanced access functionalities

(e.g. spatial / temporal / spectral subsetting), and on the fly data processing (e.g. on-the-fly

data interaction to apply cloud mask, extract vegetation indexes, etc.).

This component can be deployed as a virtual machine and provide a geospatial data server

based on Web Coverage Service (WCS). WCS is a standard data-access protocol defined by

the Open Geospatial Consortium (OGC) that defines and enables web-based retrieval of multi-

dimensional geospatial datasets. WCS provides access to the full range of geospatial data

served from a web server and allows for requesting only a subset of the data. A WCS supports

slice and trim operations, where either the data dimension (slice) or the data extent (trim) is

reduced.

MEA technology is continuously maintained and updated by MEEO staff since it is operatively

implemented in several web platforms such as eodatacube - MaaS (http://eodatacube.eu),

INSAR Italy (http://insaritaly.services.meeo.it), EOCHA data portal

(http://eocha.servcies.meeo.it).

TAMP data portal (http://vtpip.zamg.ac.at), earthserver (http://eodataservce.org), EVEREST

(http://ever-est.eu) Heracles (http://heracles-project.eu), WAT-ENER-CAST climate data

platform (http://wec.services.meeo.it) and AUSTRALIAN Geoscience Data Cube

(http://www.datacube.org.au).

http://eodatacube.eu/

http://insaritaly.services.meeo.it/

http://eocha.servcies.meeo.it/

http://vtpip.zamg.ac.at/

http://eodataservce.org/

http://ever-est.eu/

http://heracles-project.eu/

http://wec.services.meeo.it/

http://www.datacube.org.au/



In order to give an example about how MEA is supporting the BIG DATA Challenge, the figures

below show few stats about the access to eodataservice portal during the Climathon event

(Oct 2017) promoted by Climate-kic initiative where MEEO was asked to support the

Copernicus data access and dissemination by making available Copernicus datasets.

Figure 5: Statistics about the products accessed via eodataservice

Figure 6: Statistics about Climathon Products accessed via eodataservice + Jupiter

In overall, more than 345.000 products got accessed in less than 36 hours.

3.8.3 Documentation

Table 13: Summary of Multi sensors evolution analysis - MEA

Multi sensors evolution analysis - MEA

IPR Owner MEEO srl

Contact Marco Folegani

THE USERS PLAYED WITH ALL PRODUCTS

MOST USED WERE

- MODIS LST (45%)

- ERA INTERIM – 2M TEMPERATURE (21%)

- SENTINEL 2 TRUE COLOR (20%)

Products accessed via eodataservice

Sentinel 2A True Color Sentinel 2A Vegetation Index ERA Interim - 2m Temperature

ERA Interim - precipiation ERA Interim - Soil Moisture ERA Interim - Particulate Matter 2.5

ERA Interim - Particulate Matter 10 landsat 8 True Color landsat 8 Vegetation Index

MODIS Land Surface Temperature

- ECMWF ERA INTERIM: 73,008 PRODUCTS

- SENTINEL 2: 57,507 PRODUCS

- OTHER DATA: 214,734 PRODUCTS




Technology

URL http://www.eodataservice.org/

Deployment type Server – Client architecture deployed on cloud resources and on-premises

Endpoint

Documentation http://www.eodataservice.org/

IPR MEA is a commercial tool

Specific system

requirements

Data types

handled

EO and climate raster data

The data can be made accessible via different access systems (after automatic registration to

the system), which are able to accommodate different kind of users:

Web-based data visualization and access platform (https://www.eodataservice.org): this GUI

interface is devoted to all users and takes the advantages of WCS standard for data

visualization. From this interface the participants can visualize the available data, extract time

series on points/areas, extract statistical information and download the data for further

analysis (e.g. via GIS tools/Excel); some usage examples are reported on the MEEO You Tube

Channel: https://www.youtube.com/channel/UCqy-bd6CpXMOlRqfHmBqlZw.

The web interface is provided with a “take a tour” function that can help the user to explore

all the features of the tool (Figure 7).


http://www.eodataservice.org/

http://www.eodataservice.org/

https://www.eodataservice.org/

https://www.youtube.com/channel/UCqy-bd6CpXMOlRqfHmBqlZw



Figure 7: Web interface for the Multi-sensors evolution analysis

Jupyter Notebooks - python console (https://www.eodataservice.org): this web application

allows the code editing with specific libraries for WCS 2.0 commands via browser (Figure 8).

Figure 8: Screenshot of the Jupyter Notebooks – python console

Endpoint access via WCS querying: the direct data access via CLI/REST allows any

programming skilled end user to add specific queries in its own code (Figure 9).

WEB GUI

Pan

Visualizazionconfiguration

Area selection tools

Layers manager

Processing

Ground points

Location search

Reset

Projections

1D Plots

Full catalogue

Time bar

Personal basket

Look-up table

Zoom box

Points manager

https://www.eodataservice.org/



Figure 9: Endpoint access via WCS querying

3.9 Mosaic Cloud Free Background Service

3.9.1 Narrative

The service is deployed on an application server and provides a remote sensing monitoring

service. This service constructs and keeps an up to date a mosaic of Sentinel2 and Lansat8

images, covering a given area of interest (AOI) with the latest, cloud free data from satellite

scenes. The whole process chain is independent and self-content, based on cloud and

shadows mask extraction, histogram matching procedures and, finally, a pixel-based analysis.

A similar service is provided for Sentinel1 data, offering a false coloured map of the same AOI,

in order to better understand the observed objects and to elucidate any kind of problems or

issues observed in the optical mosaic map. All backgrounds are updated automatically, as

soon as a new raw scene is available. The service is used to create a cloud free background

layer to display and analyse the crop monitoring results.


The service is based on daily auto-downloading Sentinel 1, Sentinel 2 and Landsat 8 cloud free

satellite scenes, from live repositories (e.g. Copernicus Open Access Hub, Amazon, and Google

storage). Both optical and radar backgrounds are subject to real-time analysis, as they are

updated automatically, soon after a new raw scene is available.

The service simplifies the processing of massive volumes of data through its efficient and cost-

effective mechanisms and supports large AOIs (as large as a European country or geographic

area - e.g. The Balkans, The Alps etc.).



3.9.3 Documentation

Table 14: Summary of Mosaic Cloud Free Background Service

Mosaic Cloud Free Background Service

IPR Owner TerraS

Contact Adrian Stoica


Technology The service is deployed on an application server and provides a remote

sensing monitoring service. This service constructs and keeps an up to date

mosaic of Sentinel2 and Lansat8 images, covering a given area of interest

(AOI) with the latest, cloud free data from satellite scenes. All backgrounds

are updated automatically, as soon as a new raw scene is available. The

service can run on Linux server, delivering results via WMTS.

URL Will be provided as request.

Deployment type Service hosted by TERRASIGNA

Endpoint Not applicable.

Documentation WMTS service GetCapability function retrieves metadata about the service,

including supported operations and parameters, and a list of the available

layers.

IPR The service is provided free for the DataBio project pilots.

Specific system

requirements

Ubuntu Linux, Python, GNU Octave, GDAL, GraphicsMagick

MapProxy, MapServer, PostgreSQL, PostGIS

Data types

handled

Sentinel 2 and Landsat 8 images

3.10 EO Crop Monitoring Service The service assesses the agriculture parcels from satellite data and farmers declarations in

order to create a series of products like Crop masks, Parcels use maps and Crop inadvertences

maps, in support of the Common Agriculture Policy (CAP). As input data, it relays on

Copernicus Sentinel 2 data and farmers declaration of intention with respect of crops types.

3.10.1 Narrative

The component is able to assess the agriculture parcels from satellite data and farmers

declarations in order to create a series of products like, Crop masks, Parcel use maps and Crop

inadvertences maps, in support of the Common Agriculture Policy (CAP).




The analysis is performed on two levels of detail: pixel-based and plot-based.

The first type of analysis involved estimating the pixel’s crop type and comparing the obtained

values to the declared crops included in the LPIS (Land Parcel Identification System) database.

The plot-based analysis involves identifying the outliers – the parcels over which the declared

crop is potentially different from the one that extracted from the EO models. The service is

developed through machine learning methods and is mainly based on Sentinel data. The pixel-

based results were reported to the entire surface of the plots and the average values were

compared to the spectral values that are specific to the declared crops included in the LPIS

database. The objective of this type of analysis was to provide a qualitative scale, ranging from

high level of disagreement to high level of correspondence to the declared crops included in

the LPIS.

Also, the developed algorithms have the capability to identify different crops present inside a

single farm when the global size of declared surface is exceeding a specific threshold.


The service has been tailored on the specific needs of an end user operating at National level

– the Romanian Agriculture Ministry. The service allows the performing of big data analytics

to various crop indicators on parcel level. It is based on specialized highly automated

algorithms for processing big data, in support to the CAP and relying on multi-temporal series

(e.g. one year of data over entire Romania) of free and open EO data, with focus on

Copernicus Sentinel 2 data.

Image processing, data mining and machine learning techniques are involved. The steps of

the process chain are: A) Image preprocessing (numerical enhancements for S2 scenes,

ingestion for external data – parcel map, declared crops map etc.), B) Extraction of cloud and

shadow mask, C) Individual scene classification, D) Unsupervised machine learning, for

obtaining the map of crops probability at scene level, E) Time series analysis, to build the

overall crops probability map, F) Generating the specific products/maps. The process chain

can be adapted to other AOIs.

The service is developed in a way that allows processing in cloud with process parallelization

and distribution.

3.10.3 Documentation

Table 15: Summary of EO Crop Monitoring Service

EO Crop Monitoring Service

IPR Owner TerraS






Technology The service is based on image processing, data mining and machine learning

techniques. The steps of the process chain are: A) Image preprocessing

(numerical enhancements for S2 scenes, ingestion for external data – parcel

map, declared crops map etc.), B) Extraction of cloud and shadow mask, C)

Individual scene classification, D) Unsupervised machine learning, for

obtaining the map of crops probability at scene level, E) Time series

analysis, to build the overall crops probability map, F) Generating the

specific products (Crop masks, Parcels used maps and Crop inadvertences

maps). The component is running on a Linux server and can deliver results

via WMTS.

URL Not applicable.

Deployment type It is made available as a service hosted by TERRASIGNA for the DataBio C2.1

pilot.


Documentation DataBio documentation [REF-05] and [REF-07]

IPR The IPRs belongs to Terrasigna.

Specific system

requirements


MapProxy, MapServer, PostgreSQL, PostGIS

Data types

handled

Sentinel 2 data

3.11 Sentinel 2 Clouds, Shadows and Snow Mask Tool The tool produces Sentinel2 Clouds, Shadows and Snow Masks, based only on raw data,

improving the results of the genuine quality assessment band, without any external data.

3.11.1 Narrative

The tool is used as a pre-processing step in an application for supporting the Common

Agriculture Policy (CAP).

The process chain is based on a formula developed by TERRASIGNA, and was intensely tested

on many Sentinel2 scenes, in all seasons and in various geographical situation. Internal

benchmarking shown better performances than other known solution (e.g. fMask or the

genuine Sentinel2 algorithm). The input data is a full Sentinel2 scene and the processing steps

are: A) For each band, making a specific, fix level contrast, B) Assembling two particular grey

scale maps, one dedicated to clouds and the other dedicated to shadows, C) Extraction of

cloud mask and shadows through mutual confirmation, using an unsupervised machine

learning technique, D) Snow mask extraction, based on a thresholding procedure applied to

an original snow index map. The results are raster maps (GeoTiff) with 4 label codes: 0 – for

no data, 1 – for uncontaminated/free pixels, 2 – for snow, 3 – for shadows and 4 – for clouds.




The tool uses unsupervised machine learning techniques for the extraction of the cloud and

shadow masks through mutual confirmation.


Table 16: Summary of Sentinel 2 Clouds, Shadows and Snow Mask Tool

Sentinel 2 Clouds, Shadows and Snow Mask Tool

IPR Owner TerraS



Technology The tool produces Sentinel2 Clouds, Shadows and Snow Masks based only

on raw data, without any external data sources, improving the results of the

genuine quality assessment band. The tool can work as a script in Linux

environment.

URL It will be provided on request.

Deployment type The tool can be made available as software to be deployed elsewhere.


Documentation Delivered with the software package.

IPR The tool is provided free for the DataBio project pilots.

Specific system

requirements


Data types

handled

Sentinel 2 data

3.12 Online service platform for processing satellite data for forestry The Forestry Thematic Exploitation Platform (Forestry TEP) [REF-15] is a novel online solution

for efficient processing of satellite data for analysis and monitoring of forests. It offers global

optical and radar satellite imagery, processing services and tools, and capability for users to

develop and share their own processing services.




Figure 10: Overall concept of the Forestry Thematic Exploitation Platform

3.12.1 Narrative

In its forest management work, the Finnish Forest Centre (FFC) needs to detect unwanted shrubs in forest regeneration areas, to be able to remove them timely. These forest stands spread over large areas, and detection through field inspections is inefficient. Using Forestry TEP, this information can be produced efficiently. Optical multispectral imagery from the Sentinel-2 satellite is used to analyse the vegetation and further to quantify the need for fieldwork in the areas of interest. 3

3 http://www.esa.int/Our_Activities/Observing_the_Earth/Space_helps_forests_regenerate

http://www.esa.int/Our_Activities/Observing_the_Earth/Space_helps_forests_regenerate



Figure 11: Screenshot of the Forestry TEP user interface


The design and architecture of Forestry TEP follows the paradigms Software as a Service

(SaaS), Platform as a Service (PaaS) and Infrastructure as a Service (IaaS). The platform is

currently deployed in the EO Cloud environment of CloudFerro, which offers computing

power and storage capacity adequate for efficient processing of massive data amounts. The

underlying environment is based on the open source OpenStack cloud computing software.

Forestry TEP employs a fixed resourcing of over 20 cores, of which four worker virtual

machines (VMs) with 32 GB RAM each, plus dynamic resourcing for more intensive

processing, as well as 4 TB storage. The configuration allows Forestry TEP to scale up its

resources and performance dynamically according to user demand.

Via EO Cloud, Forestry TEP provides direct access to the full global catalogue satellite data

from the Copernicus Sentinel-2 and Sentinel-3 missions, as well as Landsat and Sentinel-1 GRD

level data. The data are originally sourced from the European Space Agency ESA. The size of

the data repository is over 6400 TB and growing by over 30 TB daily. The ability to process the

data online is a major advantage for the users.




Table 17: Summary of Online service platform for processing satellite data for forestry

Online service platform for processing satellite data for forestry (Forestry Thematic Exploitation

Platform)

IPR Owner Open source (CGI IT UK, ESA)

Contact Renne Tergujeff, VTT Technical Research Centre of Finland Ltd


Technology Web-based platform for the forestry domain, providing satellite imagery,

readily available processing services, applications that can be executed via

browser (SNAP, Monteverdi, QGIS), sharing capabilities, and developer

features.

Development of own services is based on Docker technology and Linux shell

scripts, with support for e.g. Python. For the developer, an online

development and execution environment with ready-made templates is

provided - no local setups or deep understanding of Docker needed.

URL https://forestry-tep.eo.esa.int/

Deployment type Online service. Web GUI for processing services and applications.

Endpoint N/A

Documentation https://forestry-tep.eo.esa.int/user-manual

https://forestry-tep.eo.esa.int/sites/default/files/ForestryTEP_RSD18.pdf

IPR Forestry TEP is a commercial service; as of May 2018 it remains free of

charge.

Specific system

requirements

For the user, only a modern web browser is needed.

Data types

handled

Satellite data

3.13 Tools for producing forestry information from EO data Probability, AutoChange and Envimon are tools provided by VTT, useful in producing forestry

information from EO data. Probability allows estimating forest variables, AutoChange is a tool

for forest/land cover change detection, and Envimon is a set of EO data pre-processing tools.



https://forestry-tep.eo.esa.int/

https://forestry-tep.eo.esa.int/user-manual

https://forestry-tep.eo.esa.int/sites/default/files/ForestryTEP_RSD18.pdf



3.13.1 Narrative

Probability allows estimation of forest variables based on EO data and reference data. It has

been applied on five continents, in forest monitoring and for land use and land cover (LULC)

mapping. An example is the production of a LULC map for the Savannekhet province in Laos.

The user provides the software image files in any GDAL support format, and reference data,

e.g. field plot or stand variable data. The output can be e.g. a GeoTIFF file.

Figure 12: Sample output from the Probability analysis chain: growing stock volume by tree species

AutoChange is a tool for forest/land cover change detection based on two-time instant

images. It applies hierarchical clustering to two-temporal data. The tool can be used for

detecting and identifying rapid changes such as cuttings, storm or pest damage.

Envimon is a set of EO data pre-processing tools for unpacking, radiometric corrections (top-

of-atmosphere and bottom-of-atmosphere reflectance) and geometric corrections. It can be

used in both batch and interactive modes. Envimon allows, for example, unpacking Sentinel-

2 images and producing a cloud mask.


Probability, AutoChange and Envimon are all designed to support exploiting the massive

amounts of EO data being generated.


Table 18: Summary of Tools for producing forestry information from EO data

Tools for producing forestry information from EO data (Probability, AutoChange and Envimon)

IPR Owner VTT Technical Research Centre of Finland Ltd

Contact Renne Tergujeff, VTT




Technology Linux/Windows software

URL Online access by agreement via https://forestry-tep.eo.esa.int/

Deployment type Probability, AutoChange and Envimon can either be installed locally, or

access to them provided via Forestry TEP.

Endpoint N/A

Documentation Documentation, guidance and usage examples are available by request. The

original method in AutoChange is described in

http://dx.doi.org/10.1080/014311698215612

IPR Negotiable

Specific system

requirements

Linux/Windows. GDAL required, and for the GUI(s) also Java Runtime

Environment.

3.14 Tool for Data management and visualization (2D / 3D) This tool will manage a massive amount of geospatial data in a fast, intuitive and interactive

way. It will provide the means for visually querying and accessing the data considering not

only the spatial and temporal extent but the attributes as well.

3.14.1 Narrative

The agronomist of a farming cooperation/ or an insurance stakeholder wants to visually

observe several attributes of the farms (e.g. average monthly NDVI, average monthly yield) in

colour coding so as to assess the farming practices and to detect abnormalities in crop

cultivation.


The Fraunhofer Tool for Data Management and Visualization is composed of the software

components GeoRocket and GeoToolbox. GeoRocket is a high-performance data-store for

geospatial files. The architecture is reactive, asynchronous and scalable based on the Open-

Source toolkit Vert.x. The component can be deployed on heterogeneous infrastructures (e.g.

bare-metal, cluster or cloud) with multiple back-ends such as Amazon S3, MongoDB,

distributed file systems (e.g. HDFS or Ceph), or your local hard drive (enabled by default).

Handling a vast amount of data is as easy as adding new nodes to the GeoRocket cluster.

Utilizing the popular OpenSource framework elasticsearch, allows to do high-performance

spatial queries and aggregations based on attributes, layers and tags.

Current solutions allow to visualize only a small amount of vector data, while keeping

interaction in place. A new, innovative tiling approach which is provided by the GeoToolbox

enables users to interactively explore huge datasets with millions of polygons. Using

aggregation and query features provided by elasticsearch allows to execute complex data



https://forestry-tep.eo.esa.int/

http://dx.doi.org/10.1080/014311698215612



analytic tasks (see also 3.15 “Service for Interactive Analysis and Aggregation”). GeoRocket

Pro simplifies the access by using domain specific languages for query and aggregation

building. An intuitive web interface named Groundstation provides access to all these features

without in-depth API knowledge. Cartographic visualization of geospatial files stored in

GeoRocket is realized by integrating the Open-source visualization frameworks MapBox (in

2D) and Cesium (in 3D).

Figure 13: Screenshot of a 2D visualisation of large vector data stored in GeoRocket


Table 19: Summary of Tool for Data management and visualization (2D / 3D)

Tool for Data Management and Visualization

IPR Owner Fraunhofer IGD

Contact Ivo Senner


Technology The Fraunhofer Data Management and Visualization Tool is implemented

based on GeoRocket Pro. GeoRocket Pro is composed of several other

software components ElasticSearch (Open-source), vert.x (Open-source),

MapBox (open-source), Cesium (Open-source). Additional functionality is

added using the GeoToolbox (commercial license).

URL Georocket.io,




Deployment type Client-Server architecture deployed on-premise


Documentation https://georocket.io/docs/

IPR GeoRocket Pro is a commercial product

Specific system

requirements


Data types

handled

Geospatial vector files, derived information / data from EO images

3.15 Service for Interactive Analysis and Aggregation This service allows to analyse and aggregate data in an interactive way. Starting with a simple

query (e.g. spatial extend), it will be possible to add more constraints, or aggregation

functions to build and refine a query for analysing geospatial data.

3.15.1 Narrative

For the implementation of the Common Agricultural Policy a data analyst wants to know the

distribution of crops in a given region in order to check if they comply with what was declared.

For this, the data analyst wants to be able to analyse fast, intuitively and interactively massive

amounts of data that represent crop distribution for a specific region derived from EO images.

These data are accumulated over time, thus increasing in volume and velocity.


The main purpose of GeoRocket is efficient storage and management of geospatial data using

standardized, well-structured formats such as GML or GeoJSON. Current solutions need to

pre-render these data to provide efficient visualisations, which in turn reduces the degree of

possible interaction dramatically. A new, innovative tiling approach which is provided by the

GeoToolbox was added to GeoRocket Pro to overcome these requirements and to enable the

visualization of millions of polygons, while keeping track of the relation between tiles and the

original data. This enables users to interactively explore huge datasets just as they would do

on the original data.

The multi-layered approach makes it possible to adapt the visualization to a particular use-

case scenario. By then end of the project it will be possible to build each layer from a highly

customizable query which allows to access all data stored in GeoRocket such as vector or

raster data, along with derived information such as colour coding or property-based elevation

for 2.5D Visualization.

https://georocket.io/docs/




Table 20: Summary of Service for Interactive Analysis and Aggregation

Service for Interactive Analysis and Aggregation

IPR Owner Fraunhofer IGD

Contact Ivo Senner


Technology The innovative tiling approach is provided by the GeoToolbox. This

component comprises various data processing capabilities which can be

licensed individually. More easily, the innovative tiling approach is also

integrated in the component GeoRocket Pro (see 3.14 “Tool fo Data

Management and Visualisation) and can be accessed from its administration

client called Groundstation.

URL Georocket.io,

Deployment type Script-based service deployed on premise (GeoToolbox) or Client-Server

architecture deployed on premise (GeoRocket Pro)


Documentation In progress

IPR GeoToolbox and GeoRocket Pro are commercial products

Specific system

requirements


Data types

handled

Vector and raster data along with derived information

3.16 Tool for visualization of 2D, 3D and 4D data of a high volume The tool can handle both vector and mosaic data. Moreover, it is possible to process and

visualize various types of georeferenced data structures:

• tables and relationships (entity-relationship-attribute model, database)

• tree (hierarchical data, such as XML or JSON based datasets)

• general graph (linked data, RDF)

Particularly, those data connectors exist:

• PostGIS database

• Web Map Service for raster and imagery data

• GeoJSON ~ for vector data




• Resource Description Framework (RDF) ~ for linked data

• OpenStreetMap live data pump ~ for vector data from OSM

3.16.1 Narrative

The following applications are best practice examples of processed data visualization that

were created using the above-mentioned framework.

1. Open Land Use Perspective Visualization

The Open Land Use perspective visualization is available at following address:

http://ng.hslayers.org/examples/3d-olu.

This application visualises the Open Land Use Map (OLU)4 on top of the EU-DEM terrain model

in a perspective view. Displaying the data in 3D environments helps users to explore an area

of interest in a more natural way than a traditional map. Moreover, the users can explore

other datasets, such as Open Transport Map5 or Smart Points of Interests6, or even add a Web

Map Service of their choice (as long as the WMS is in WGS84 coordinate system) to make

custom 3D mash-ups.

It is worth to mention that the application uses a strategy to visualize a continent-wide

dataset (OLU) in the 3D environment. The strategy consists of two features (adopted from

[REF-16]):

- Generalized data is portrayed when an observer is far from the Earth surface visualized

on the virtual globe visualized by Cesium plugin. Detailed data are segmented by

municipalities.

- When the observer moves closer, the following steps are processed:

o A visible spatial extent on Earth’s surface is calculated from an actual view.

o Bounding boxes for municipalities are then compared to the visible spatial

extent.

o Finally, displayed are the only municipalities, whose bounding boxes

intersected with the Earth’s surface visible spatial extent.

o These steps are done repeatedly each time when neither observer position nor

line of sight is changed, as it influences the actual view.

4 http://sdi4apps.eu/open_land_use 5 http://opentransportmap.info 6 http://sdi4apps.eu/spoi

http://ng.hslayers.org/examples/3d-olu

http://sdi4apps.eu/open_land_use

http://opentransportmap.info/

http://sdi4apps.eu/spoi



Figure 14: 3D Open Land Use visualisation

2. Perspective visualization of estimated yield - Rostěnice farm

As the yield is an integrator of landscape and climatic variability, it provides useful information

for identifying management zones [REF-17]. These zones are understood as areas with the

same or similar yield level within the fields. Such identified zones present a fundamental

delineation for site specific crop management.

The zones delineation is usually based on yield maps over the past few years. Similar to the

evaluation of yield variation from multiple yield data described by Blackmore et al. [REF-18]

the aim is to identify high yielding (above the mean) and low yielding areas related as the

percentage to the mean value of the field. Also, the inter-year spatial variance of yield data is

important for agronomists to distinguish between areas with stable or unstable yields. But

the presence of complete series of yield maps for all fields is rare, thus satellite imagery can

be analysed to determine the desired in field variability of crops thru vegetation indices where

yield maps are missing [REF-19].

The estimation of yield potential zones from multi-temporal satellite data is established as

the general model in FOODIE platform [REF-20]. As the main data source, ESPA repository of

LANDSAT satellite images is used, which offers surface reflectance products, main vegetation

indices (NDVI, EVI) and clouds identification by CFmask algorithm. A selection of scenes from

recent 8 years was made for the Rostěnice farm area to collect cloud-free data related to the

second half of the vegetation period. An estimated yield was calculated for separate scenes

as the relation of each pixel to the mean value of the whole field. In the last step all scenes

were combined, and the median value of yield potential was calculated. After full operation

of Sentinel 2A/B satellites, calculation of yield potential can be enhanced by its vegetation

products.



The Rostěnice application (available at http://ng.hslayers.org/examples/rostenice) visualises

the crop yield dataset calculated from (satellite imagery as mentioned above) on top of the

EU-DEM terrain model in a perspective view.

Such a visualization can help a farmer to better understand the farm fields. The farmer can

explore the relation of the estimated crop yield to the topography, slope, orientation and

topography wetness index in his/her field. The farmer can check the parts of fields with steep

slopes and how the machinery deals with them (by checking the machinery tracklogs, where

available), see more e.g. in [REF-21].

The application uses a digital terrain model of fifth generation and the digital surface model

of first generation produced by ČÚZK7.

Figure 15: Rostěnice Farm Visualisation

3. Linked data integration to 3D virtual environment - Vineyards example

This application (available at http://ng.hslayers.org/examples/produce-3d) visualises

management zones of different vine types, describing the type of vine by colour and label,

and the productivity of each zone using vertical exaggeration of a polygon representing

particular vineyards. After clicking on a vineyard, attribute data from the vineyard layer are

displayed together with the linked data from dbpedia8 and other semantic sources. The aim

of this example is to show integration of linked data and spatial data together.

7 http://www.cuzk.cz/en 8 http://wiki.dbpedia.org/

http://ng.hslayers.org/examples/rostenice

http://ng.hslayers.org/examples/produce-3d

http://www.cuzk.cz/en

http://wiki.dbpedia.org/



Figure 16: Linked data integration to 3D virtual environment


From the above-mentioned diversity of the data storage structures it comes clear that robust

and easily customizable applications should be used for data visualization.

The raw data visualization is usually done by a generic client application, handled by a

professional, who can deal with different data structures and formats. These applications are

not described here as there are many of them in the market or available as open source

solutions (QGIS9 for example, to mention at least one application of many).

The common technological framework selected for further best practises development

consists of HSLayers NG10 ~ a JavaScript based library for geodata visualisation, with an

integrated Cesium plugin11 for 3D visualisation. This framework was then extended for

handling various types of data, namely:

- Web Map Service ~ for raster and imagery data

- GeoJSON ~ for vector data

- Resource Description Framework (RDF) ~ for linked data

- OpenStreetMap live data pump ~ for vector data from OSM

The framework development was then divided into following steps:

- Integration of CESIUM with HSlayers NG (see folders examples/3d-olu and

components/cesium at https://github.com/hslayers/hslayers-ng repository for details).

9 https://www.qgis.org/ 10 https://github.com/hslayers/hslayers-ng 11 https://cesiumjs.org/

https://github.com/hslayers/hslayers-ng

https://www.qgis.org/

https://github.com/hslayers/hslayers-ng

https://cesiumjs.org/



- Installing own CESIUM server (http://cesium.cenia.cz)

- Publishing local DEMS on own CESIUM server

The created framework allows developers to create their own tailored applications. Last but

not least the framework supports responsive design.


Table 21: Summary of Tool for visualization of 2D, 3D and 4D data of a high volume

Data Management and Visualization tool

IPR Owner Lesprojekt s.r.o

Contact Karel Charvat


Technology The common technological framework selected for further best practises

development consists of HSLayers NG (Open Source) ~ a JavaScript based

library for geodata visualization, with an integrated Cesium plugin for 3D

visualisation (Open Source)

URL http://ng.hslayers.org/examples/, www.databio.eu

Deployment type Client-Server architecture deployed on-premise

Endpoint GUI

Documentation http://ng.hslayers.org/

IPR Software is OS, customization and deployment are paid

Specific system

requirements

3.17 Service and tool for Data management (Sensor data) This component is a web-based sensor data management application that can be used as tool

in standalone installation or can be used as service on general instance for DataBio.

The component is receiving sensor data from data producers’ components, stores data in the

database and publish stored data by the system of RESTful web services.

3.17.1 Narrative

SensLog will be used in several DataBio pilots from the agriculture domain as sensor data

management tool.

In the first case, farmers want to have a tool for mapping the crop vigor status to support

them in the decisions for variable rate application (VRA) of fertilizers and crop protection.

http://cesium.cenia.cz/


http://ng.hslayers.org/examples/


http://ng.hslayers.org/



SensLog will provide the data support for further data analysis and decision-making

processes. Stored sensor data can be visualized using other components like C02.03 HSLayers

[REF-05] and C03.01 WebGLayer [REF-05].

In another case, the pilot is focused mainly on collecting telemetry data from machinery and analysing them. SensLog will be used as tool for telemetry and sensor data storage. SensLog will receive observations and positions from agricultural machinery. Received data will contain telemetry data of the active machinery as well as observation data of the passive machinery. Data will be stored in the database with spatial extension and published for further analyses by other components. SensLog will serve as tracking data publisher for visualization components and the telemetry data support for further analysis in the FarmTelemetry component.


SensLog is web-based sensor data management system suitable for both static in-situ sensors

and sensors deployed on mobile carrier. It provides system of RESTful web services for data

receiving and publishing. It provides system of RESTful web services for data receiving and

publishing. It supports several standard interfaces, e.g. OGC SOS v1 or NGSI 9/10 (in process).

SensLog stores data in database in own data model. Scalability of the data model is supported

by table partitioning mechanism for the core tables. Scalability of the component is supported

by dividing of the application into two separate components - Receiver module and Publishing

module. Both modules are designed to be deployed on heterogeneous infrastructures.

Utilization of Docker container and Elasticsearch framework can increase scalability and

capacity of the instance. Performance of selection of particular data is supported by pre-

defined queries and prepared subsets of data.


Table 22: Summary of Service and tool for Data management (Sensor data)

Service and tool for Data management (Sensor data)

IPR Owner UWB, Lesprojekt

Contact Michal Kepka


Technology The SensLog is server-side application implemented in JAVA language.

URL www.senslog.org, www.databio.eu


Endpoint Not deployed yet on DataBio infrastructure

Documentation http://www.senslog.org/api/

IPR SensLog is open-source product under BSD license


http://www.senslog.org/


http://www.senslog.org/api/



Specific system

requirements

Java Runtime Environment (JRE), PostgreSQL instance

3.18 Tool for Metadata Management ((Open)Micka) (Open) Micka is a web application for management and discovery of geospatial metadata.

The following features are supported by an (Open) Micka: OGC Catalogue service (CSW 2.0.2),

Transactions and harvesting, Metadata editor, Multilingual user interface, ISO AP 1.0 profile,

Feature catalogue (ISO 19110), Interactive metadata profiles – management, WFS/Gazetteer

for defining metadata – extent, GEMET thesaurus built-in client, INSPIRE registry built-in

client, OpenSearch, INSPIRE ATOM download service - automatically creation from metadata.

3.18.1 Narrative

A user would like to know whether a newly published Sentinel 2 image covers his or her farm

plot with clouds or not. In other words, a user would like to know whether it is worth

downloading the image, what is its size and not being bothered with the first downloading at

least hundreds of megabytes and secondly processing the image to get the answer. Such task

is an example of a fitness-for-purpose analysis. The (Open)Micka software initiates the

Metadata, Linked Data and Graph Data Pipeline continuously. Such approach enables to (1)

download new resources when they become available, (2) process them in advance and (3)

create a derived (meta)information. When speaking specifically about this use case,

(Open)Micka regularly downloads Sentinel-2 images when they become available. Each image

is then processed with the CFMask algorithm to distinguish between areas covered by clouds

and those that remain visible in the satellite image. A derived (meta)information layer on the

geospatial extent of the visible parts of the image is made available. After this, a user searches

for his or her farm. (Open)Micka provides a user with the information on the overall

cloudiness percentage as well as shows explicitly whether his or her farm plot is covered by

clouds. Thus, a user receives an exact answer to his or her question in a form of simple

(meta)information. The basic goal of this use case was successfully finished, the user has an

answer on a detailed (meta)information level that was processed directly from the data

without any inference needed by the user.

Note that the above-mentioned use case is an initial demo use case as the (meta)data

processing and deriving relevant (meta)information offers dozens or even hundreds of use

cases. Those use cases are dependent on the kind of used resources. For instance, a different

use case could be deriving relevant (meta)information from raw sensor measurements of

traffic sensors to answer the questions on environment pollution.


(Open)Micka is a system for metadata management used for building spatial data

infrastructure (SDI) and geoportal solutions. It contains tools for editing and the management

of (geospatial) data and services metadata, and other sources (documents, websites, sensor

measurements etc.). (Open)Micka has been continuously developed within the last twenty



years as it is used as a metadata catalogue in a dozen of European research and innovation

projects, and also for instance in the Czech national INSPIRE geoportal. One of the newest

achievements, GeoDCAT RDF/XML is generated from existing ISO 19139 and/or INSPIRE

metadata in the catalogue according to the rules defined by the GeoDCAT-AP specification.

In addition to (Open)Micka, CKAN is also implemented. CKAN is an open source data

management and publishing tool supporting DCAT. It is a deployable as a Web portal that acts

as a data catalogue, where a user discovers and views datasets of his or her interest. Acting

as a catalogue, CKAN keeps track of the location of the actual data and their metadata. Using

such an extension, CKAN supports DCAT to import or export desired datasets. CKAN enables

harvesting data from OGC CSW catalogues, but not all mandatory INSPIRE metadata elements

are supported. Unfortunately, the DCAT output does not fulfil all INSPIRE requirements, nor

is GeoDCAT-AP fully supported.

In DataBio, the following developments are delivered in order to support the Metadata,

Linked Data and Graph Data Pipeline:

• Discovers relevant (meta)data from the Copernicus Open Access Hub through the SciHub API, e.g. through the following query https://scihub.copernicus.eu/dhus/search?q=footprint:%22Intersects(POLYGON((16.75%2049.03,%2017.12%2049.04,%2017.06%2049.30,%2016.78%2049.29,%2016.75%2049.03)))%22&FORMAT=json.

• Parses the following relevant information from the JSON format (meta)data obtained through the Copernicus Open Access Hub API:

o name of a platform like „Sentinel-2B“; o date of a measurement like “2018-02-17T09:50:49.027Z” that is being

transformed into a more readable way like „17.2.2018 9:50:49“; o instruments’ names like „MSI“ that are being transformed into a more

readable way in a form of a code list with the following values: multispectral image, radar image etc.

o file size like “416.90 MB“; o data format like „SAFE“; o cloud coverage percentage like „92.7“; o link for downloading the image like

https://scihub.copernicus.eu/dhus/odata/v1/Products('d0cb0602-7545-491a-9d73-4ae19936d4fc')/$value;

o four coordinates of a bounding box depicting the geospatial extent of an image.

• Enables to download an image with just one click;

• Enables to combine Earth Observation (meta)data with other (meta)data.

Figure 17 depicts the schematic overview of the deployment with the Copernicus Open Access

Hub, while Figure 18 depicts the current (18 May 2018) status of implementation.

https://scihub.copernicus.eu/dhus/search?q=footprint:%22Intersects(POLYGON((16.75%2049.03,%2017.12%2049.04,%2017.06%2049.30,%2016.78%2049.29,%2016.75%2049.03)))%22&FORMAT=json



https://scihub.copernicus.eu/dhus/odata/v1/Products('d0cb0602-7545-491a-9d73-4ae19936d4fc')/$value

https://scihub.copernicus.eu/dhus/odata/v1/Products('d0cb0602-7545-491a-9d73-4ae19936d4fc')/$value



Figure 17: Schematic overview of the DataBio Metadata, Linked Data and Graph Data Pipeline that re-uses (meta)data from the Copernicus Open Access Hub to display metadata directly in a map viewer

Figure 18: Screenshot of the current version (dated to 18 May 2018) of the progress in the DataBio Metadata, Linked Data and Graph Data Pipeline

The development has also been tightly connected to the hackathons and similar developers’

events, such as the INSIRE Hack12.

12 http://www.plan4all.eu/inspire-hack-2017/

http://www.plan4all.eu/inspire-hack-2017/

http://ng.hslayers.org/examples/featureFilter/?hs_panel=composition_browser&hs_x=1977293.8709483435&hs_y=6379524.90315378&hs_z=6&visible_layers=Base layer;Satellite imagery example



Further activities aim at four major directions:

1. Enhancing the support also for the NASA API

(https://api.nasa.gov/planetary/earth/imagery).

2. Enhancing the capabilities of near real-time processing.

3. Enhancing the functionality for notifications, as depicted in Fig. 3.

4. Enhancing the usage also in an open source desktop product, i.e. QGIS

(http://qgis.org) through a plug-in.

(Open)Micka in a form of the Metadata, Linked Data and Graph Data Pipeline is not used only

as a managing application of metadata. A typical metadata management deals with derived

(meta)information that does not fit into Big data definitions.

On the contrary, (Open)Micka follows user- and performance- oriented approaches that

require near real-time analyses and analytics of underlying (Big data) resources. Its primary

aim is to process massive volumes of data in order to support the decision making based on

a simplified version of the original resource, however still the same veracity.


Table 23: Summary of Tool for Metadata Management

Tool for Metadata Management

IPR Owner Lesprojekt, s.r.o.

Contact Karel Charvat


Technology Geospatial metadata catalogue and metadata editing tool under open (; the

BSD 3-Clause License) as well as commercial licenses.

Technical requirements:

• Any web server with mod_rewrite enabled

• PHP 5.6.x with XSL extension

• PostgreSQL >= 9.2

• Composer (https://getcomposer.org/) - for installation some

components.

URL https://github.com/hsrs-cz/Micka

http://www.bnhelp.cz/produkty/metadata/

http://databio.eu


Endpoint GUI, OGC CSW 2.0.2 (also with ISO Application Profile)

https://api.nasa.gov/planetary/earth/imagery

http://qgis.org/


https://github.com/hsrs-cz/Micka/blob/master/LICENSE

https://getcomposer.org/

https://github.com/hsrs-cz/Micka

http://www.bnhelp.cz/produkty/metadata/

http://databio.eu/



Documentation https://github.com/hsrs-cz/Micka,

A user manual is provided in the README file in this repository.

IPR Open Source as well as Commercial edition, customization and deployment

are paid in both cases

Specific system

requirements

https://github.com/hsrs-cz/Micka



Conclusion In the DataBio project, the technical partners provide and deliver innovative solutions for

handling big Earth Observation and Geospatial data. These solutions are adapted according

to the specific needs of the pilot applications that are implemented in the scope of the

DataBio project. These solutions for handling big EO and Geospatial data are not exclusively

built for the DataBio pilots. Their functionalities are accessible for any users. This document

provides a comprehensive summary of the services and tools delivered by the different

partners. It enables the readers to assess the applicability of the delivered software for their

specific purposes along with the details on how and under which conditions these services

and tools can be employed. This list of services and tools from the first year of the DataBio

project already shows an impressive portfolio of different functionalities for handling big EO

and Geospatial data. They can be used to offer directly functionalities in agriculture, forestry

and fishery applications or can be used, as part of the DataBio toolset, to build such

applications. In the fast-changing landscape of technologies, it can be a valuable guidance in

the search for flexible, scalable and spatially-enabled tools.



References Reference Name of document (include authors, version, date etc. where applicable)

[REF-01] DataBio D1.1: Agriculture Pilot Definition, v1.0, 2017-06-30.

[REF-02] DataBio D2.1: Forestry Pilot Definition, v1.0, 2017-06-30

[REF-03] DataBio D3.1: Fishery Pilot Definition, v1.0, 2017-10-20.

[REF-04] DataBio D4.1: Platform and Interfaces, v1.0, to be published.

[REF-05] DataBio D5.1: EO Component Specification, v1.0, 2017-12-29.

[REF-06] DataBio D4.2: Services for Tests, v1.0, 31/05/2018.

[REF-07] DataBio D5.2: EO Component and Interfaces, v1.0, to be published.

[REF-08] Wikipedia contributors, "Kubernetes," Wikipedia, The Free Encyclopedia,

https://en.wikipedia.org/w/index.php?title=Kubernetes&oldid=843061780

(accessed May 23, 2018).

[REF-09] ISO 19115:2003/Cor 1:2006, Geographic Information – Metadata – Implementation

specification, http://www.iso.org/iso/iso_catalogue/catalogue_tc/

catalogue_detail.htm?csnumber=44361

[REF-10] Standard Archive Format for Europe (SAFE), http://earth.esa.int/SAFE

[REF-11] FedEO Gateway Software Design Document, PDGS-FEDEO-OSGW-SPB-SDD-16-1263,

Issue 1, Revision 6, 12/03/2018.

[REF-12] OGC 10-032r8, OGC OpenSearch Geo and Time Extensions,


[REF-13] OGC 13-026r8, OGC OpenSearch Extension for Earth Observation Products,

http://docs.opengeospatial.org/is/13-026r8/13-026r8.html.

[REF-14] OGC 10-157r4, Earth Observation Metadata profile of Observations &

Measurements, Version 1.1, 09/06/2016, http://docs.opengeospatial.org/is/10-

157r4/10-157r4.html.

[REF-15] Häme, T., Tergujeff, R., Rauste, Y., Farquhar, C., van Zetten, P., Kershaw, P., de

Groof, A., Hämäläinen, J., van Bemmelen, J. & Seifert, F. M. (2017), Forestry-TEP

responds to user needs for sentinel data value adding in cloud, 2017 conference on

Big Data from Space : BiDS’17. European Commission, p. 239-242

[REF-16] Charvát, K., Mildorf, T., Bye, B., L., Berre, A.,J., Jedlička, K. Open Data, VGI and Citizen

Observatories INSPIRE Hackathon. In International Journal of Spatial Data

Infrastructures Research. In Press 2018

[REF-17] Kleinjan, J., Clay, D. E., Carlson, C. G., & Clay, S. A. (2007). Productivity zones from

multiple years of yield monitor data. In F. J. Pierce, & D. C. Clay, GIS applications in

agriculture. CRC Press, Boca Raton.

[REF-18] Blackmore, S., Godwin, R. J., & Fountas, S. (2003). The Analysis of Spatial and

Temporal Trends in Yield Map Data over Six Years. Biosystems Engineering.

[REF-19] Charvát K., Řezník T., Lukas V., Charvát K., Jr., Horáková, Š., Kepka, M., Šplíchal, M.

QUO VADIS PRECISION FARMING, In 13th International Conference on Precision

Agriculture Proceedings, July 31 – August 4, 2016, St. Louis, Missouri, USA

https://en.wikipedia.org/w/index.php?title=Kubernetes&oldid=843061780

http://www.iso.org/iso/iso_catalogue/catalogue_tc/

http://earth.esa.int/SAFE







[REF-20] Řezník, T., Charvát, K., LUKAS, V., Charvát Jr, K., Horáková, Š., & Kepka, M. (2015).

Open Data Model for (Precision) Agriculture Applications and Agricultural Pollution

Monitoring. Proceedings of the Enviroinfo and ICT for Sustainability.

[REF-21] Řezník, T., Lukas, V., Charvát, K., Charvát Jr, K., Horáková, Š., Křivánek, Z., & Herman,

L. (2016). Monitoring of in-field variability for site specific crop management through

open geospatial information. International Archives of the Photogrammetry, Remote

Sensing & Spatial Information Sciences, 41.

d5.3 eo services and tools - databio · d5.3 – eo services and tools h2020 contract no. 732064...

Documents