lerici, 8-10 october 2019 big data infrastructure for
TRANSCRIPT
1st Maritime Situation Awareness Workshop 2019
Lerici, 8-10th October 2019
Big Data Infrastructure for heterogeneous sources and Data fusion services applied to Maritime Surveillance
Engineering Ingegneria Informatica S.p.A., Speakers: Giuseppe Vella, Domenico Messina Co-authors: Giovanni Barone, Viviana Latino, Vito Morreale
@EU_H2020 #H2020 @MarisaProjectEU
2
The MARISA project The Big Data Infrastructure and the MARISA Data model The Inspiration The Big Data Infrastructure and the interactions with the
architectural layers The data fusion process in a nutshell
Outline
20/11/2019
3
MARISA Consortium
20/11/2019
The Engineering I.I. S.p.A. R&D
4 20/11/2019 The MARISA Project is coordinated by Leonardo SpA. The MARISA project has received funding from the European Union's Horizon 2020 Research and Innovation programme under Grant Agreement No 740698.
Founded in 1987
420 RESEARCHERS
Over 300 research projects,
successfully completed
40M Euro investments in 2018
+75 LIVE PROJECTS
4 LABORATORIES
The IS3 running projects
5 20/11/2019 The MARISA Project is coordinated by Leonardo SpA. The MARISA project has received funding from the European Union's Horizon 2020 Research and Innovation programme under Grant Agreement No 740698.
Smart and Secure Transport and Infrastructure
Smart Industry
and Agrifood
Fight Against Crime and Terrorism
Cyber Security
IS & Big Data
(*) New projects
Blockchain
Border and External Security
& Defence
Security
A4BLUE
AMELIE
BEinCPPS
EW-SHOPP
FAR-EDGE
FINDUSTRY4.0
ICOSAF (*)
L4MS
MARKET4.0
MIDIH
OEDIPUS
OK-INSAID (*)
QU4LITY (*)
SERENA
SIMONA
SIPARIO (*) ICARUS (*)
TOREADOR
STORM
DEFENDER
MAIA (*)
MARISA
ALFA
PYTHIA
COMPACT
HERMENEUT
DOGANA
PROTECT-ID
SURVANT
ASGARD
TRILLION
DANTE
ANITA
CONNEXIONS
DNT
CREEP (*)
REVEIL (*)
DESOLATE (*)
FAR-EDGE
DCoT
Industry
IPMDecisions (*)
SOLOMON (*)
PREVENT (*)
INFRASTRESS (*)
FASTER (*)
ANDROMEDA (*)
MUSKETEER
SAGRES
Promerc
PERSEUS
The MARISA project
6 20/11/2019 The MARISA Project is coordinated by Leonardo SpA. The MARISA project has received funding from the European Union's Horizon 2020 Research and Innovation programme under Grant Agreement No 740698.
Create improved situational awareness with a focus on delivering a complete and useful comprehension of the situation at sea
Support the practitioners along the complete lifecycle of situations at sea, from the observation of elements in the environment up to detection of anomalies and aids to planning
Ease a fruitful collaboration among adjacent and cross-border agencies operating in the maritime surveillance sphere in order to pull resources towards the same goal, leading to cost efficient usage of existing resources
Foster a dynamic eco-system of users and providers, allowing new data fusion services to be delivered to different actors at sea by the integration of a wide range of data and sensors
The MARISA project
7 20/11/2019 The MARISA Project is coordinated by Leonardo SpA. The MARISA project has received funding from the European Union's Horizon 2020 Research and Innovation programme under Grant Agreement No 740698.
Create improved situational awareness with a focus on delivering a complete and useful comprehension of the situation at sea
Support the practitioners along the complete lifecycle of situations at sea, from the observation of elements in the environment up to detection of anomalies and aids to planning
Ease a fruitful collaboration among adjacent and cross-border agencies operating in the maritime surveillance sphere in order to pull resources towards the same goal, leading to cost efficient usage of existing resources
Foster a dynamic eco-system of users and providers, allowing new data fusion services to be delivered to different actors at sea by the integration of a wide range of data and sensors
How to deliver
information to agencies
and to make an efficient exchange of
data?
The Big Data Infrastructure and the MARISA Data model
8 20/11/2019
MARISA
Data Model Entities
OSINT
IVEF
CISE
Meteo
Satellite
BDI
Streaming Platform
Data products Data products
Data products Distribute data storage A
Distribute data storage B
Distribute data storage C
The MARISA Project is coordinated by Leonardo SpA. The MARISA project has received funding from the European Union's Horizon 2020 Research and Innovation programme under Grant Agreement No 740698.
The Inspiration
9 20/11/2019
Big Data for maritime surveillance
acquired from various sources ranging from sensors, satellites, open source, internal sources
in order to extract information from these huge amount through advanced correlation that improves the quality knowledge and the management of critical situations
and providing capabilities for Big Data Analytics as a Service
The MARISA Project is coordinated by Leonardo SpA. The MARISA project has received funding from the European Union's Horizon 2020 Research and Innovation programme under Grant Agreement No 740698.
Some figures
10 20/11/2019 The MARISA Project is coordinated by Leonardo SpA. The MARISA project has received funding from the European Union's Horizon 2020 Research and Innovation programme under Grant Agreement No 740698.
Trials Data Fusion Services
Cross border interactions
Concurrent elaboration processes
11
26 5 2
User interactions
Big Data Infrastructure
Web console System administration Console Services administration Console
Data fusion services layer
Data fusion1 Services Data fusion2 Services Data fusion3 Services
Interfaces external data sources
CISE Adaptors IVEF Adaptors OSINT Adaptors
Web Application Service
Distributed storage
Redis
Cassandra
Distributed processing framework
SPARK
Streaming platform
Kafka Topics
MARISA BD REST API
Streams Services
Geoserver
Meteo Adaptors Satellite Adaptors
Data Fusion Distribution Services
Postgres
External geoservers
Geo
server AP
I
Geoserver Client
User interactions
Big Data Infrastructure
Web console System administration Console Services administration Console
Data fusion services layer
Data fusion1 Services Data fusion2 Services Data fusion3 Services
Interfaces external data sources
CISE Adaptors IVEF Adaptors OSINT Adaptors
Web Application Service
Distributed storage
Redis
Cassandra
Distributed processing framework
SPARK
Streaming platform
Kafka Topics
MARISA BD REST API
Streams Services
Geoserver
Data Fusion Distribution Services
Postgres
Geo
server AP
I
Geoserver Client
Real time ingestion
Meteo Adaptors Satellite Adaptors External geoservers
User interactions
Big Data Infrastructure
Web console System administration Console Services administration Console
Data fusion services layer
Data fusion1 Services Data fusion2 Services Data fusion3 Services
Interfaces external data sources
CISE Adaptors IVEF Adaptors OSINT Adaptors
Web Application Service
Distributed storage
Redis
Cassandra
Distributed processing framework
SPARK
Streaming platform
Kafka Topics
MARISA BD REST API
Streams Services
Geoserver
Meteo Adaptors Satellite Adaptors
Data Fusion Distribution Services
Postgres
External geoservers
Geo
server AP
I
Geoserver Client
Data fusion flow for multi steps processing
User interactions
Big Data Infrastructure
Web console System administration Console Services administration Console
Data fusion services layer
Data fusion1 Services Data fusion2 Services Data fusion3 Services
Interfaces external data sources
CISE Adaptors IVEF Adaptors OSINT Adaptors
Web Application Service
Distributed storage
Redis
Cassandra
Distributed processing framework
SPARK
Streaming platform
Kafka Topics
MARISA BD REST API
Streams Services
Geoserver
Meteo Adaptors Satellite Adaptors
Data Fusion Distribution Services
Postgres
External geoservers
Geo
server AP
I
Geoserver Client
Harvesting from external geoservers
User interactions
Big Data Infrastructure
Web console System administration Console Services administration Console
Data fusion services layer
Data fusion1 Services Data fusion2 Services Data fusion3 Services
Interfaces external data sources
CISE Adaptors IVEF Adaptors OSINT Adaptors
Web Application Service
Distributed storage
Redis
Cassandra
Distributed processing framework
SPARK
Streaming platform
Kafka Topics
MARISA BD REST API
Streams Services
Geoserver
Meteo Adaptors Satellite Adaptors
Data Fusion Distribution Services
Postgres
Geo
server AP
I
Geoserver Client
Converting geoservers data for Data fusion
External geoservers
User interactions
Big Data Infrastructure
Web console System administration Console Services administration Console
Data fusion services layer
Data fusion1 Services Data fusion2 Services Data fusion3 Services
Interfaces external data sources
CISE Adaptors IVEF Adaptors OSINT Adaptors
Web Application Service
Distributed storage
Redis
Cassandra
Distributed processing framework
SPARK
Streaming platform
Kafka Topics
MARISA BD REST API
Streams Services
Geoserver
Meteo Adaptors Satellite Adaptors
Data Fusion Distribution Services
Postgres
External geoservers
Geo
server AP
I
Geoserver Client
Data Fusion Distribution Service
User interactions
Big Data Infrastructure
Web console System administration Console Services administration Console
Data fusion services layer
Data fusion1 Services Data fusion2 Services Data fusion3 Services
Interfaces external data sources
CISE Adaptors IVEF Adaptors OSINT Adaptors
Web Application Service
Distributed processing framework
SPARK
Streaming platform
Kafka Topics
MARISA BD REST API
Streams Services
Geoserver
Meteo Adaptors Satellite Adaptors
Data Fusion Distribution Services
External geoservers
Geo
server AP
I
Geoserver Client
Data retrieval for the HMI
Distributed storage
Postgres
Redis
Cassandra
Big Data Infrastructure
Interfaces external data sources
CISE Adaptors IVEF Adaptors OSINT Adaptors
Distributed storage
Redis
Cassandra
Distributed processing framework
SPARK
Streaming platform
Kafka Topics
MARISA BD REST API
Streams Services
Geoserver
Meteo Adaptors Satellite Adaptors
Data Fusion Distribution Services
Postgres
External geoservers
Geo
server AP
I
Geoserver Client
Data Consuming for the Data Fusion Services
Data fusion services layer
Data fusion1 Services Data fusion2 Services Data fusion3 Services
User interactions
Web console System administration Console Services administration Console Web Application Service
Real Time and Cold data for Data fusion
Interfaces with external data sources
Distributed storage
Redis
Cassandra
Streaming Services Data fusion services layer
Kafka Consumers
Topic1 Topic2 Topic3 Topic4 Topic5
Kafka Producers
Geoservers API
Mappers for SAR and VHR optical data
Kafka Producers
20/11/2019 The MARISA Project is coordinated by Leonardo SpA. The MARISA project has received funding from the European Union's Horizon 2020 Research and Innovation programme under Grant Agreement No 740698.
Some figures on Results
20 20/11/2019 The MARISA Project is coordinated by Leonardo SpA. The MARISA project has received funding from the European Union's Horizon 2020 Research and Innovation programme under Grant Agreement No 740698.
Kafka 1.7 GiB -> daily
Redis 8 GiB -> three days
Cassandra -> two months (considering 200GiB of storage)
# of records on Cassandra
Response time (in seconds) – 1 worker
Response time (in seconds) – 2 workers
1.000 28’’ 29’’
10.000 28’’ 39’’
100.000 78’’ 62’’
1.000.000 92’’ 64’’
Between 20 and 50 messages per second
20/11/2019 21
Questions?
Thanks Border and External Security and Defence unit
IS3 Lab - Intelligent Systems and Social Software for Security, Enterprises, Transport and Infrastructures
Research & Development Department Research & Innovation Division [email protected] [email protected]
The MARISA Project is coordinated by Leonardo SpA. The MARISA project has received funding from the European Union's Horizon 2020 Research and Innovation programme under Grant Agreement No 740698.