thingmonk 2016 - concursus event sourcing for the iot by tareq abedrabbo & dominic fox
TRANSCRIPT
![Page 1: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/1.jpg)
ConcursusEvent Sourcing for the Internet of ThingsThingMonk 2016
![Page 2: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/2.jpg)
IntroductionsDominic Fox
Twitter: @dynamic_proxy
Email: [email protected]
Tareq Abedrabbo
Twitter: @tareq_abedrabbo
Email: [email protected]
Concursus
Page: https://opencredo.com/publications/concursus/
Github: http://github.com/opencredo/concursus
![Page 3: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/3.jpg)
What is Concursus?
A framework (or toolkit) for processing and organising messy data in an distributed context.
![Page 4: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/4.jpg)
Event Sourcing
“Event Sourcing ensures that all changes to application state are stored as a sequence of events. Not just can we query these events, we can also use the event log to reconstruct past states, and as a foundation to automatically adjust the state to cope with retroactive changes.”
http://martinfowler.com/eaaDev/EventSourcing.html
![Page 5: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/5.jpg)
What is Concursus?Problems Concursus addresses:
Processing events in a scalable and reliable way
Processing guarantees and ordering: exactly once, out of order, repeated or missed delivery, etc..
Building meaningful domain models to reason about and build business logic around
Flexibility: building additional views as needed
![Page 6: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/6.jpg)
Building Blocks• Java 8 (Kotlin is supported too)
• Cassandra as an event store (other backbends are supported)
• Kafka or RabbitMQ as message brokers
• Hazelcast for transient state
![Page 7: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/7.jpg)
Sources of InspirationStream processing frameworks such as Apache Storm and Spark
Google papers: Cloud dataflow, MillWheel
Apache Spark papers
The Axon CQRS framework
Domain Driven Design
![Page 8: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/8.jpg)
Tendencies:
• From internet of users to internet of things
• From “presence” to “presents”
• From monoliths to microservices
Why Concursus?
![Page 9: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/9.jpg)
From Internet of Users to Internet of Things
![Page 10: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/10.jpg)
From Presence to Presents
![Page 11: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/11.jpg)
From Monoliths to Microservices
![Page 12: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/12.jpg)
“Write First, Reason Later”
![Page 13: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/13.jpg)
“Write First, Reason Later”
![Page 14: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/14.jpg)
Example
![Page 15: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/15.jpg)
Domain Model: Events
![Page 16: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/16.jpg)
aggregateType: lightbulbaggregateId: 69016fb5-1d69-4a34-910b-f8ff5c702ad9
eventTimestamp: 2016-03-31T10:31:17.981Zparameters: { “wattage”: 60 }
Domain Model: Events
![Page 17: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/17.jpg)
aggregateType: lightbulbaggregateId: 69016fb5-1d69-4a34-910b-f8ff5c702ad9
eventTimestamp: 2016-03-31T10:36:42.171Zparameters: { “location”: “hallway”}
Domain Model: Events
![Page 18: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/18.jpg)
aggregateType: lightbulbaggregateId: 69016fb5-1d69-4a34-910b-f8ff5c702ad9
eventTimestamp: 2016-03-31T10:36:42.171ZprocessingTimestamp: 2016-03-31T10:36:48.3904Zparameters: { “location”: “hallway”}
Domain Model: Events
![Page 19: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/19.jpg)
Domain Model: SummaryEvery Event occurs to an Aggregate, identified by its type and id.Every Event has an eventTimestamp, generated by the source of the event.An Event History is a log of Events, ordered by eventTimestamp, with an additional processingTimestamp which records when the Event was captured.
![Page 20: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/20.jpg)
Network
Event sources
Event processors
Events arrive:• Partitioned• Interleaved• Out-of-order
Processing Model: Ordering
![Page 21: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/21.jpg)
Log is:• Partitioned by aggregate id• Ordered by event timestamp
Processing Model: Ordering
![Page 22: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/22.jpg)
CREATE TABLE IF NOT EXISTS concursus.Event ( aggregateType text, aggregateId text, eventTimestamp timestamp, streamId text, processingId timeuuid, name text, version text, parameters map<text, text>, characteristics int, PRIMARY KEY((aggregateType, aggregateId), eventTimestamp, streamId)) WITH CLUSTERING ORDER BY (eventTimestamp DESC);
Cassandra Schema
![Page 23: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/23.jpg)
CassandraEvent Store
RabbitMQ Topic
DownstreamprocessingLog
events
Publish events
Cassandra & AMQP
![Page 24: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/24.jpg)
CassandraEvent Store
RabbitMQ Topic
Downstreamprocessing
out-of-order events
ordered query results
Cassandra & AMQP
![Page 25: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/25.jpg)
CassandraEvent Store
Kafka Topic
Downstreamprocessing
Event store listener
Publish events
Log events
Cassandra & Kafka
![Page 26: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/26.jpg)
Processing Model: SummaryEvents arrive partitioned, interleaved and out-of-order.Events are sorted into event histories by aggregate type and id.Events are sorted within event histories by event timestamp, not processing timestamp.Event consumers need to take into account the possibility that an event history may be incomplete at the time it is read – consider using a watermark to give incoming events time to “settle”.
![Page 27: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/27.jpg)
Programming Model: Core Metaphor
![Page 28: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/28.jpg)
Consumer<Event>
Programming Model: Core Metaphor
![Page 29: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/29.jpg)
You give me a Consumer<Event>, and I send Events to it one at a time:
Emitting Events
![Page 30: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/30.jpg)
I implement Consumer<Event>, and handle Events that are sent to me.
Handling Events
![Page 31: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/31.jpg)
Java 8 Mapping
![Page 32: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/32.jpg)
Java 8 Mapping
![Page 33: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/33.jpg)
Java 8 Mapping
![Page 34: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/34.jpg)
Event-handling middleware is a chain of Consumer<Event>s that transforms, routes, persists and dispatches events. A single event submitted to this chain may be:■ Checked against an idempotency filter (e.g. a Hazelcast distributed cache)■ Serialised to JSON■ Written to a message queue topic■ Retrieved from the topic and deserialised■ Persisted to an event store (e.g. Cassandra)■ Published to an event handler which maintains a query-optimised view of part of the
system■ Published to an event handler which maintains an index of aggregates by event property
values (e.g. lightbulbs by wattage)
Event-Handling Middleware
![Page 35: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/35.jpg)
Thank you for listeningAny questions?
![Page 36: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/36.jpg)
Three Processing Schedules
1.Transient
2.Durable
3.Persistent
![Page 37: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/37.jpg)
Three Processing Schedules
1.Transient
2.Durable
3.Persistent
![Page 38: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/38.jpg)
Three Processing Schedules
1.Transient
2.Durable
3.Persistent
![Page 39: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/39.jpg)
Three Processing Schedules
1.Transient - what happens
2.Durable - what’s happening
3.Persistent - what happened
![Page 40: ThingMonk 2016 - Concursus Event sourcing for the IOT By Tareq Abedrabbo & Dominic Fox](https://reader035.vdocuments.site/reader035/viewer/2022062905/586f781c1a28ab10258b6a53/html5/thumbnails/40.jpg)
“Write First, Reason Later”