getting started with vortex

Data Sharing in Industrial IoT

AngeloCorsaro,PhDChiefTechnologyOfficer

[email protected]

mailto:[email protected]

Vortex is a standard technology for efficient, ubiquitous, interoperable, secure and platform independent data sharing across network connected devices

in151 Characters

Vortex's Coordination Model

Applications can autonomously and asynchronously read and

write data enjoying spatial and temporal decoupling

DDS Global Data Space

...

Data Writer

Data Writer

Data Writer

Data Reader

Data Reader

Data Reader

Data Reader

Data Writer

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

Global Data Space

Built-in dynamic discovery isolates applications from

network topology and connectivity details


...

Data Writer

Data Writer

Data Writer

Data Reader

Data Reader

Data Reader

Data Reader

Data Writer

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

Dynamic Discovery

The relevant portion of the data space is projected on

the application address space. Each typed projection is commonly called a Cache

No single point of failure or bottleneck

Decentralised Data Space

Data Writer

Data Writer

Data Writer

Data Reader

Data Reader

Data Reader

Data Writer

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

TopicDQoS

TopicDQoS

TopicAQoS

Connectivity is dynamically adapted to chose the most

effective way of sharing data

Adaptive ConnectivityData

Writer

Data Writer

Data Writer

Data Reader

Data Reader

Data Reader

Data Writer

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

TopicDQoS

TopicDQoS

TopicAQoS

ThecommunicationbetweentheDataWriterandmatchingDataReaderscanbepeer-to-peerexploitingUDP/IP(UnicastandMulticast)orTCP/IP

ThecommunicationbetweentheDataWriterandmatchingDataReaderscanbe“brokered”butstillexploitingUDP/IP(UnicastandMulticast)orTCP/IP

A domain-wide information’s class A Topic defined by means

of a <name, type, qos>

TopicDDS Global Data Space

...

Data Writer

Data Writer

Data Writer

Data Reader

Data Reader

Data Reader

Data Reader

Data Writer

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

TopicTypeName

QoS

Topic Types

Topic Types: Language Independent

Definitions

Topic types can be expressed using different syntaxes,

including IDL and ProtoBuf

Topic Type struct CarDynamics { string cid; long x; long y; float dx; long dy; } #pragma keylist CarDynamics cid

IDL



Topic Type message CarDynamics { option (.omg.dds.type) = {name: "CarDynamics"}; required string cid = 0 [(.omg.dds.member).key = true]; required long x = 1; required long y = 2; required float dx = 3; required long dy = 4; }

ProtoBuf

Topic Types: Language Specific

Definitions



Topic Type class CarDynamics: constructor: (@cid, @x, @y, @dx, @dy) ->

CoffeeScript



Topic Type public struct CaDynamics { public string cid { get; set; } public int x { get; set; } public int y { get; set; } public int dx { get; set; } public int dy { get; set; } public CaDynamics (string cid, int x, int y, int dx, int dy) { this.cid = cid; this.x = x; this.y = y; this.dx = dx; this.dy = dy; } }

C#



Topic Type @KeyList ( topicType = "CarDynamics", keys = {"cid"})public class CarDynamics { public String cid; public int x; public int dx; public int y; public int dy; public CarDynamics(String s, int a, int b, int c,int d) { this.cid = s; this.x = a; this.dx = b; this.y = c; this.dy = d; } @Override public String toString() { … }}

Java

Topics, Instances and Samples

Cop

yrig

ht P

rism

Tech

, 201

5

Each unique key value identifies a stream of data

Vortex demultiplexes “streams” and provides lifecycle information

A DataWriter can write samples belonging to multiple instances

Topic Instances

CarDynamics

\

Instances

struct CarDynamics { @key string cid; long x; long y; float dx; long dy; }

Topic

Topic Type

Sample

https://twitter.com/acorsaro










http://www.linkedin.com/in/corsaro










http://slideshare.net/angelo.corsaro









QoS policies allow to express temporal and availability

constraints for data


...

Data Writer

Data Writer

Data Writer

Data Reader

Data Reader

Data Reader

Data Reader

Data Writer

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

QoS - Enabled

A collection of policies that control non-

functional properties such as reliability,

persistence, temporal constraints and priority

QoS

HISTORY

LIFESPAN

DURABILITY

DEADLINE

LATENCY BUDGET

TRANSPORT PRIO

TIME-BASED FILTER

RESOURCE LIMITS

USER DATA

TOPIC DATA

GROUP DATA

OWENERSHIP

OWN. STRENGTH

LIVELINESS

ENTITY FACTORY

DW LIFECYCLE

DR LIFECYCLE

PRESENTATION

RELIABILITY

PARTITION

DEST. ORDER

RxO QoS Local QoS

QoS Policies controlling end-to-end properties

follow a Request vs. Offered

QoS Domain

Participant

DURABILITY

OWENERSHIP

DEADLINE

LATENCY BUDGET

LIVELINESS

RELIABILITY

DEST. ORDER

Publisher

DataWriter

PARTITION

DataReader

Subscriber

DomainParticipant

offered QoS

Topicwrites reads

Domain Idjoins joins

produces-in consumes-from

RxO QoS Policies

requested QoS

Interacting with the Data Cache

Each Data Reader is associated with a Cache

The Cache stores the last n∊𝜨∞ samples for each

relevant instance

Data Cache

Data Writer

Data Writer

Data Writer

Data Reader

Data Reader

Data Reader

Data Writer

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

TopicDQoS

TopicDQoS

TopicAQoS

Each Data Reader is associated with a Cache

The Cache stores the last n∊𝜨∞ samples for each

relevant instance

Data Cache

DataReader Cache

DataReader

...

Samples

Instances

Cache

The action of reading samples for a Reader Cache

is non-destructive.

Samples are not removed from the cache

Reading Data

DataReader Cache

DataReader

...

DataReader Cache

DataReader

...read

The action of taking samples for a Reader Cache

is destructive.

Samples are removed from the cache

Taking Data

DataReader Cache

DataReader

...

DataReader Cache

DataReader

...take

Samples can be selected using composable content

and status predicates

Sample Selectors

DataReader Cache

DataReader

...

Filters allow to control what gets into a DataReader

cache

Filters are expressed as SQL where clauses or as

Java/C/JavaScript predicates

Content-Filtering

DataReader Cache

DataReader

...

Filter

Application

Network

Content Filters can be used to project on the

local cache only the Topic data

satisfying a given predicate

Content Filters structCarDynamics{

@keystringcid;longx;longy;floatdx;longdy;}

cid x y dx dyGR 33N GO 167 240 45 0LO 00V IN 65 26 65 0AN 637 OS 32 853 0 50AB 123 CD 325 235 80 0

“dx>50ORdy>50”

Type

CarDynamics

cid x y dx dyLO 00V IN 65 26 65 0AB 123 CD 325 235 80 0

Reader Cache

Queries allow to control what gets out of a

DataReader Cache

Queries are expressed as SQL where clauses or as

Java/C/JavaScript predicates

Content-Based Selection

DataReader Cache

DataReader

...

Query

DataReader Cache

DataReader

...

Application

Network

Reader Cache

Queries can be used to select out of the local cache

the data matching a given predicate

QueriesstructCarDynamics{@keystringcid;longx;longy;floatdx;longdy;}


“dx>50ORdy>50”

Type

CarDynamics


cid x y dx dyLO 00V IN 65 26 65 0AB 123 CD 325 235 80 0

query

State based selection allows to control what gets out of a DataReader Cache

State base selectors predicate on samples meta-

information

State-Based Selection

DataReader Cache

DataReader

...

State Selector

DataReader Cache

DataReader

...

Application

Network

Sample State (READ, NOT_READ) allows to distinguish between new

samples and samples that have already been read


DataReader Cache

DataReader

...

State Selector

DataReader Cache

DataReader

...

Application

Network

The View State (NEW, NOT_NEW) allows to distinguish a new instance

from an existing one


DataReader Cache

DataReader

...

State Selector

DataReader Cache

DataReader

...

Application

Network

The Instance State (ALIVE, NOT_ALIVE_DISPOSED,

NOT_ALIVE_NO_WRITERS) allows to track the life-cycle transitions of the instance to which a sample belongs


DataReader Cache

DataReader

...

State Selector

DataReader Cache

DataReader

...

Application

Network

Selector Example

// == ISO C++ DDS API ==

auto data = dr.select() .content(query) .state(data_state) .instance(handle) .read();

your first vortex app

Cop

yrig

ht P

rism

Tech

, 201

4

Anatomy of a DDS Application






























Cop

yrig

ht P

rism

Tech

, 201

5

Writing Data in C++#include <dds.hpp>

int main(int, char**) {

DomainParticipant dp(0); Topic<Meter> topic(“SmartMeter”); auto pqos = dp.get_default_pubqos() << Partition (“FR:IdF:Essonne:Gif:Essonne:Mnts:123”); Publisher pub(dp, pqos); DataWriter<Meter> dw(pub, topic);

while (!done) { auto value = readMeter() dw.write(value); std::this_thread::sleep_for(SAMPLING_PERIOD); }

return 0; }

enumUtilityKind{ ELECTRICITY, GAS, WATER};structMeter{ stringsn; UtilityKindutility; floatreading; floaterror;};#pragmakeylistMetersn






























Cop

yrig

ht P

rism

Tech

, 201

5

Reading Data in C++#include <dds.hpp>

int main(int, char**) {

DomainParticipant dp(0); Topic<Meter> topic(”SmartMeter”); Subscriber sub(dp); DataReader<Meter> dr(dp, topic);

LambdaDataReaderListener<DataReader<Meter>> lst; lst.data_available = [](DataReader<Meter>& dr) { auto samples = data.read(); std::for_each(samples.begin(), samples.end(), [](Sample<Meter>& sample) { std::cout << sample.data() << std::endl; } } dr.listener(lst); // Print incoming data up to when the user does a Ctrl-C std::this_thread::join(); return 0; }































Cop

yrig

ht P

rism

Tech

, 201

4

Writing Data in Scalaimport dds._import dds.prelude._import dds.config.DefaultEntities._

object SmartMeter { def main(args: Array[String]): Unit = { val topic = Topic[Meter](“SmartMeter”) val dw = DataWriter[Meter](topic) while (!done) { val meter = readMeter() dw.write(meter) Thread.sleep(SAMPLING_PERIOD) } }}































Cop

yrig

ht P

rism

Tech

, 201

4

Reading Data in Scala

import dds._import dds.prelude._import dds.config.DefaultEntities._

object SmartMeterLog { def main(args: Array[String]): Unit = { val topic = Topic[Meter](“SmartMeter”) val dr = DataReader[Meter](topic) dr listen { case DataAvailable(_) => dr.read.foreach(println) } }}































Cop

yrig

ht P

rism

Tech

, 201

5

Writing Data in Python

import dds import timeif __name__ == '__main__': topic = dds.Topic("SmartMeter", "Meter") dw = dds.Writer(topic) while True: m = readMeter() dw.write(m) time.sleep(0.1)































Cop

yrig

ht P

rism

Tech

, 201

5

Reading Data in Pythonimport ddsimport sys def readData(dr): samples = dds.range(dr.read()) for s in samples: sys.stdout.write(str(s.getData())) if __name__ == '__main__': t = dds.Topic("SmartMeter", "Meter") dr = dds.Reader(t) dr.onDataAvailable = readData































Vortex Technology Stack

Device implementations optimised for OT, IT and

consumer platforms

Native support for Cloud and Fog Computing Architectures

Device-2-DeviceDevice-2-Cloud

Fog-2-Cloud

Device-2-Fog

Cloud-2-Cloud

Fog-2-Fog

infra

structure

sdk

Available across IT, Consumer and OT platforms Device-2-DeviceDevice-2-Cloud

Fog-2-Cloud

Device-2-Fog

Cloud-2-Cloud

Fog-2-Fog

infra

structure

sdk

Polyglot and Interoperable across Programming

Languages Device-2-DeviceDevice-2-Cloud

Fog-2-Cloud

Device-2-Fog

Cloud-2-Cloud

Fog-2-Fog

infra

structure

sdk

Fully Independent of the Cloud Infrastructure

Private Clouds


Fog-2-Cloud

Device-2-Fog

Cloud-2-Cloud

Fog-2-Fog

infra

structure

sdk

Native Integration with the hottest real-time analytics

platforms and CEP Device-2-DeviceDevice-2-Cloud

Fog-2-Cloud

Device-2-Fog

Cloud-2-Cloud

Fog-2-Fog

infra

structure

sdk

Integration with mainstream Dashboard Technologies


Fog-2-Cloud

Device-2-Fog

Cloud-2-Cloud

Fog-2-Fog

infra

structure

sdk

High Performance 30 μs peer-to-peer latency

4+ Mmsgs/sec p2p throughput


Fog-2-Cloud

Device-2-Fog

Cloud-2-Cloud

Fog-2-Fog

infra

structure

sdk

>10 μs fog/cloud routing latency

High Performance Device-2-DeviceDevice-2-Cloud

Fog-2-Cloud

Device-2-Fog

Cloud-2-Cloud

Fog-2-Fog

infra

structure

sdk

SecureData-Level security with Pluggable Authentication Access Control and Crypto


Fog-2-Cloud

Device-2-Fog

Cloud-2-Cloud

Fog-2-Fog

infra

structure

sdk

Default Plug-insX.509 Public Key

Infrastructure (PKI) based authentication


Fog-2-Cloud

Device-2-Fog

Cloud-2-Cloud

Fog-2-Fog

infra

structure

sdk

Default Plug-insAccess Control List file

signed by a shared Certificate Authority for


Fog-2-Cloud

Device-2-Fog

Cloud-2-Cloud

Fog-2-Fog

infra

structure

sdk

Default Plug-ins

Crypto based on TLS Cipher Suite


Fog-2-Cloud

Device-2-Fog

Cloud-2-Cloud

Fog-2-Fog

infra

structure

sdk

SecureData-Security as opposed to simply Transport-Level

security

Arthur Dent

Arthur Dent

Ford Prefect

Zaphod Beeblebrox

Marvin

Trillian

left/A(r,w), left/B(r)

left/A(r,w), left/B(r,w), left/X(r)

left/*(r,w)

left/*(r), right/(w)

left/A(r,w), left/B(r,w), right/C(r,w)

Ford Prefect

Zaphod Beeblebrox

Trillian

Marvin

A

B

A,BX

*

*

A,B,C

Identity Access RightsSessions are authenticated and communication is encrypted

Only the Topic included as part of the access rights are visible and accessible

SecureFine-grained access control

over Partition/Topic/Instance regular expressions

Arthur Dent

Arthur Dent

Ford Prefect

Zaphod Beeblebrox

Marvin

Trillian



left/*(r,w)



Ford Prefect

Zaphod Beeblebrox

Trillian

Marvin

A

B

A,BX

*

*

A,B,C



SecureSupport for Symmetric and Asymmetric Authentication

Arthur Dent

Arthur Dent

Ford Prefect

Zaphod Beeblebrox

Marvin

Trillian



left/*(r,w)



Ford Prefect

Zaphod Beeblebrox

Trillian

Marvin

A

B

A,BX

*

*

A,B,C



Boundary SecurityBoundary security support is

enabled by Vortex-Fog

Device-to-Cloud Communication

Peer-to-Peer (Broker-less)

Device-to-Device Communication

Fog Computing Fog ComputingFog Computing

TLS

TLS

Boundary SecuritySeparates security concerns at

different scales and controls what information is exposed


Peer-to-Peer (Broker-less)

Device-to-Device Communication

Fog Computing Fog ComputingFog Computing

TLS

TLS

Typical Deployment

Vortex device, such as Lite, Café, and Web are used to share data between different kinds of applications within a car

Café and Web are typically used Android / HTML5 based infotainment

Lite in typically used in ECU, sensors and onboard analytics

Vortex Fog is used to transparently (for in car apps) decouple and control the data sharing within and across the car

Vortex Fog also helps defining security boundaries and policies

Vortex Fog efficiently and securely deals with car to car communication

Vortex Fog allows to decouple the transport used for in-car communication and that used for car-to-car communication

Vortex Fog efficiently and securely deals with cloud connectivity adapting traffic flows and protocols


Cloud Analytics

Fog AnalyticsFog Analytics

Fog Analytics

Vortex Cloud efficiently and securely makes data available to any device at an Internet Scale

Vortex Cloud efficiently and securely makes data available to any device at an Internet Scale


Cloud Analytics

Fog AnalyticsFog Analytics

Fog Analytics

Use Cases

Cop

yrig

ht P

rism

Tech

, 201

5

Connected Boulevard — Nice, FranceVortex enables is the data sharing platform

for the City Operating System

Tens of thousands of devices connected by Vortex

Autonomous Vehicles

coordination of fast moving autonomous vehicles

intermittent connectivity

dynamic pairing of devices

Cop

yrig

ht P

rism

Tech

, 201

4

Smart Factory0.5 TB of data

produced per day

Cop

yrig

ht P

rism

Tech

, 201

4

Smart-Grid

20ms deadline for phase alignment data

ESA Extremely Large & SMART Telescope (ELT)

1750 computing nodes

100.000 mirrors whose position is

adjusted 100 times per second!

ELT will allow astronomers to probe the earliest stages

of the formation of planetary systems and to detect water and organic

molecules in proto-planetary discs around

stars in the making

Launch System

80K+ data points with aggregate updates rate of ~400K msgs/sec

Connected Aircrafts

10 TB of data every 30m of flight

Oil Rig 30000 data pointsonly 1% of available data

used today

Smart Lightbulbs96Kbytes Memory

Connected Medical Devices

12 msec period for pleath data

Peer-to-Peer connectivity

Vortex enables Smart, Secure and Efficient Data Sharing in IoT

In Summary

Cop

yrig

ht P

rism

Tech

, 201

5