Research & Service Support

Delft Software Days 2015, Deltares

28/10/2015

Processing services for Sentinel-1 data

exploitation

RSS Team member: J. Manuel Delgado

Outline

1.Service description

2.Processing services

a. RSS processing on demand

b. RSS Cloud Toolbox

3.SNAP / Sentinels Toolbox

4.Examples of Sentinel-1A data processing

Service Objectives

• Support the Earth Observation community in exploiting EO data

• data provisioning

• data access

• data processing

• Empower effective and efficient research process

• Support exploitation environments

• enabling product generation

• specific derived information

instead of bringing “Big Data” to “users”

we bring “users” to “Big Data”

User community

RSS user community includes:

• Scientists (research centers, universities)

• Service developers (sme or industry developing technology projects)

• European or international institutions

who have started their work locally on their workstations with some data

samples and want to “scale up” to massive data processing or wide

service exposure.

Support in the Research Process

Support options

Provision of customized Sandboxes and Cloud Toolboxes

Customized Virtual Machines with pre-installed software (e.g. NEST, Sentinel

Toolbox, Matlab, IDL, other) to support user-specific processing requirements

can be provided on researcher request

Provision of EO data of interest

Data can be provided either in the sandbox/toolbox or on a dedicated Data

Farm for further processing. Reference datasets from ESA and Third Party

Earth Observation data on required areas of interest and time intervals can be

made available for download to the researcher's desktop

Processor Hosting / Processing

Algorithm integration and hosting service of selected processors on RSS

virtual infrastructure and support to processing campaigns on required areas

of interest and time intervals. Flexible processing resources (cloud) are used.

High Level Scientific and IT Competences

RSS team can support different stages of the algorithm/service development

with highly skilled competences

Research and Service Support

Sandboxes, Cloud toolboxes based on Customized Virtual Machines

with pre-installed software, mainly thought for post-processing and

analysis activities.

• Can be provided with the software required by the user

• Flexible amount of CPUs, RAM and dedicated storage, to be

tailored on the actual user needs and the type of machine

template required (virtual toolboxes)

Processing Resources

Grid and Cloud Processing On-Demand: g-pod

Generic GRID and Cloud-based processing on demand environment

with high-performance and sizeable computing resources.

Provides the necessary flexibility for building an application virtual

environment with quick accessibility to data, computing resources and

results.

RSS processing on demand

http://gpod.eo.esa.int

1. The Web Portal is the user access point where he/she can

a. Submit processing requests

b. Check the status of the task

(created/pending/active/completed/failed)

c. Perform some basic operations: abort/resubmit/copy the task

2. Catalogue Access Service (CAS) is the catalogue where the input files and

auxiliary data information (metadata and URL) are registered as well as the

path/address from which the data is stored

3. The Light Grid Engine (LGE) is the orchestrator of the GRID

a. It stores the applications (PI algorithms)

b. It receives processing requests from the G-POD web portal and

sends processing tasks to the computing clusters

4. Stores and datafarm are where the data belonging the different available

missions is physically stored

5. Grid and Cloud computing resources are where the processing are

performed

Architecture

RSS Flexible Resources

On-demand processing service:

Platform

Infrastructure

G-POD

ESRIN - 172 cores - 400 TB

UK-PAC - 96 cores - 300 TB

Flexible/ Unlimited Infrastructure - 10-200 cores - 1-10 TB

EO Scientists Principal Investigators

delivery Process

EO data

Volume accessed by PI projects in 2014: • Total Number Submitted Jobs 39,456 • Average Number of Products per Job: 48 • Average Product Size: 700 MB • Total Size Data Processed: 1326 TB

Catalogue

S1TBX service on G-POD

S1TBX service on G-POD

RSS CloudToolbox

RSS CloudToolbox Service Portal

http://eogrid.esrin.esa.int/cloudtoolbox

If approved the administrator creates the CloudToolbox

CloudToolbox request

start/stop button: the user manages the VM. Commands are launched through the APIs

User can require a machine update

CloudToolbox upgrade

1. Via SSH

a. Supporting the command lines operations

2. Via Desktop viewer (VNC viewer)

a. The user experience is not affected by the unpleasant

feeling of delay

3. Users are provided with detailed instructions on how to

connect

4. VMs are configured with an automatic shutdown after

30 minutes of inactivity.

How to connect

SNAP / Sentinels Toolbox

1. The Sentinel Toolboxes and full source code is distributed freely under the GNU

GPL license. The source code is available on Github.

2. The SENTINEL-1 Toolbox (S1TBX) consists of a collection of processing tools,

data product readers and writers and a display and analysis application to

support the large archive of data from:

a. ESA SAR missions including :

– SENTINEL-1, ERS-1 & 2 and ENVISAT, as well as

b. Third Party SAR data :

– ALOS PALSAR, TerraSAR-X, COSMO-SkyMed and RADARSAT-2.

c. The various processing tools could be run independently from the

command-line and also integrated within the graphical user interface.

d. The Toolbox includes tools for calibration, speckle filtering,

coregistration, orthorectification, mosaicking, data conversion,

polarimetry and interferometry.

3. The SENTINEL-1 Toolbox is being developed for ESA by Array in partnership

with DLR, Brockmann Consult and OceanDataLab.

Description

1. Shares the core architecture with the ESA BEAM and NEST toolbox;

2. Java implementation

3. Display and Analysis Tool (DAT): integrated graphical user-friendly

interface;

4. Graph Processing Framework (GPF): for creating user-defined processing

chains;

5. Graphical or command-line execution;

6. Tiled memory management for working with very large data products;

7. Data abstraction models to handle all SAR missions in a common way

8. Modular design for easy modifications and upgrades

9. Users are able to add their own modules via APIs

10.Multithreading and Multi-core processor support

Architecture Highlights

S1TBX Architecture: Data flow

Release of Sentinel Toolboxes

Sentinel Toolboxes

• Multi-mission Scientific Toolboxes

• Developed as open source software

• Common architecture (SNAP)

• Java/Python API and Java/Python plugins

• Portable engine to a Cloud infrastructure

Download https://sentinel.esa.int

Toolbox ServiceDeveloper

Toolbox Engine

Middleware

Processor

Plug-in A

Plug-in B

Plug-in C

Executable ZPlug-in C

Plug-in A

Plug-in B

Plug-in C

Plug-in C

Executable X

Executable Y

Executable Z

VisualisationUser Interface

AnalysisUser Interface

ProcessingUser Interface

Data I/OUser Interface

Toolbox Common Library and Extensibility Framework

Wrapper API(s)Stand-alone Tool

AdapterNative API

new processors (= new WPS capabilities)are deployed in the cloudwith the Toolbox Engine

as platform.

WebBrowser

MobileDevice

User’sToolbox

servicecall

Data Archives

cloudservice

interfaces

WPS

Toolbox “Server Configuration”

STEP

• EO science collaborative platform

• Technical forum and community animation

• Gathering user feedback and usage

• Communicating on results

• Cloud demonstrator (e.g. Land Training 2015)

Website http://step.esa.int

SentiNel Application Platform

Display and analysis tool: DAT

Graph Processing Tools

Readers

Utilities

SAR Tools

1. Absolute calibration (Sentinel-1A, Envisat ASAR, ERS 1&2,

ALOS, Radarsat-2, TerraSAR-X, Cosmo-skymed )

2. Coregistration of detected and complex products

3. Terrain Correction

4. Multilooking

5. Speckle filtering

6. Debursting of Sentinel IW SLC and ASAR WSS products

7. TOPSAR support

a. Split products in subswaths

b. Assembling consecutive products (same orbit and pass)

c. TOPSAR coregistration and interferogram formation

Example: Sentinel-1A data processing

Sentinel-1A data processing

1. Example 1:

a. TOPSAR Interferogram over The Netherlands

b. Georeferencing of Sentinel-1A derived products

2. Example 2:

a. Assembling of several Sentinel-1 slices (consecutive

products belonging to same orbit acquired during the same

pass)

b. Calibration

c. Thermal Noise Removal

d. Terrain correction

e. Subset

ESA UNCLASSIFIED – For Internal Use

Example 1

1. Using RSS processing on-demand:

a. Submission of a TOPS Interferogram

formation in G-POD using Sentinel-

1A IW SLC products over The

Netherlands

2. Using the RSS CloudToolbox:

a. Download of resulting interferogram

products and

– post-processing on RSS

CloudToolbox using the SNAP/

Sentinel-1 Toolbox.

– Visualization of results

TOPSAR Interferogram

TOPSAR Interferogram

1. Batch processing using Sentinel-1 Toolbox with xml graph (see figure).

TOPSAR Interferogram

S1TBX GUI on the RSS Cloud Toolbox

Georeferencing of Sentinel-1A derived products

Georeferencing of Sentinel-1A derived products

Sentinel-1A derived data visualization

Sentinel-1A derived data visualization

Example 2

1. By using the graph showed below, which has been created using the

Graph Builder module of SNAP/S1TBX, we obtain as output a subset

of two consecutive Sentinel products assembled, calibrated, thermal

noise free and terrain-corrected

Customized processing chain

Example 2

Example 2

Example 2

Example 2

Example 2

Example 2

RSS contacts

For general information:

RSS Website: rssportal.esa.int

Join&Share: wiki.services.eoportal.org

For feedback or enquiries:

RSS Team: rss_team@esa.int

G-POD Team: eo-gpod@esa.int