ODIP II – Data Streams and FlowsSeptember 2015

Adam Leadbetter, Damian Smyth, Rob Fulleradam.leadbetter@marine.ie

http://www.confluent.io/blog/apache-kafka-samza-and-the-unix-philosophy-of-distributed-data

20th Century 21st Century

Single machine Distributed processing

One-to-one communication Publish-subscribe pattern

No fault tolerance Replication, auto-recovery

Fixed schema, encoding Schema management, evolvable encoding

20th Century 21st CenturyByte stream Messages

In-memory Durable

Blocking / backpressure Buffering

One-to-one Multi-subscriber

https://github.com/peterataylor/om-json

Further Work

• Look at the architectures for reprocessing historic data

• Incorporate Sensor Web Enablement, OM-JSON

• Deploy on vessels / mobile, remote platforms

• Investigation of Apache NiFi