build your own service bus v2
DESCRIPTION
Our feedback on the creation of a distributed enterprise service bus.TRANSCRIPT
Service Bus
ALT.NET Paris October 2013 ABC Arbitrage
Build your own
Olivier Coanet OMG this guy does not tweet
Kevin Lovato @alprema
ABC Arbitrage Asset Management www.abc-arbitrage.com
ALT.NET France www.altnetfr.org www.meetup.com/altnetfr
Service Bus Build your own
Application 1
ESB
Application 2
Application 3
Copyright Bullshit Consulting ®
The tube is a lie
Qpid Service Bus V1: broker-based
Application 1 Application 2
Qpid.Client Abc.ServiceBus
Application
Qpid
Broker-based Transport
layer
Bus library
Qpid.Client Abc.ServiceBus Application 1
Qpid
Qpid.Client Abc.ServiceBus Application 2
Application 3 Abc.ServiceBus
Qpid.Client ESB + infrastructure services
Qpid.Client Abc.ServiceBus
Service 1
Qpid
Qpid.Client Abc.ServiceBus
Service 2
Service 3 Abc.ServiceBus
Qpid.Client
And they were blissfully, blissfully happy forever and ever
And they were blissfully, blissfully happy forever and ever
NEW REQUIREMENTS
Higher troughtput
Better latency
Custom messaging usage was missing Service Bus features
and the stack was kind of
version
LEFT DEAD
.NET client
Buy or Build?
Brokerless transport recipe
• Peer to peer communication protocol
• Peer directory
Brokerless transport recipe
• Optional and parallel persistence
Brokerless transport recipe
Chef tip: • Use interoperable ingredients
Brokerless transport recipe
The cherry on the cake: • Build a gateway for Qpid communication
Brokerless transport recipe
Maybe you want implementation details?
P2P communication
Protocol Buffers
“ØMQ sockets are the world-saving superheroes of the networking world”
Pieter Hintjens
Service 1 Service 2 Service 3
Service 4
PUSH PUSH PUSH
PULL
Who do we connect to?
Connecting to the directory
Service 1 Service 2 Service 3
Directory
Peers list + subscriptions
Service 1 Service 2 Service 3
Directory
Connecting to the directory
RegisterCommand + I listen to HelloEvent
Service 1
New service ID + subscriptions
Service 2 Service 3
Directory
Connecting to the directory
Service 1 Service 2 Service 3
Directory
Connecting to the directory
HelloEvent
The directory, our new SPOF
Connecting to the directories
Directory 1 Directory 2
Service 1 Service 2 Service 3
RegisterCommand + I listen to HelloEvent
Connecting to the directories
Directory 1 Directory 2
Service 1 Service 2 Service 3
RegisterCommand + I listen to HelloEvent
Connecting to the directories
Directory 1 Directory 2
Service 1 Service 2 Service 3
HelloEvent listeners: Service 2
HelloEvent listeners: Service 1
Connecting to the directories
Directory 1 Directory 2
Service 1 Service 2 Service 3
HelloEvent listeners: Service 2
HelloEvent listeners: Service 1
RegisterCommand Peers list + subscriptions
Connecting to the directories
Directory 1 Directory 2
Service 1 Service 2 Service 3
HelloEvent listeners: Service 2
HelloEvent listeners: Service 1
HelloEvent listeners: Service 2
Cassandra to the rescue
Directory 1 Directory 2
What if a service goes down?
HelloEvent
Service 1
Persistence service
Service 2
Copy of HelloEvent
Persisting messages
Stored messages Stored messages Stored messages
Persisting messages
Service 1
Persistence service
Service 2
HelloEvent Ack
• Persistence service acting as frontend
• Cassandra for storage
Persistence service
Messages persistence model
Row Key
Abc_Service1 2013-10-03 08:25:38 2013-09-03 11:03:52 …
Serialized message …
Abc_Service2 2012-03-09 23:01:00 ...
..
…
Acks persistence model
Row Key
Abc_Service1 2013-10-03 08:25:38 2013-09-03 11:03:52 …
null …
Abc_Service2 2012-03-09 23:01:00 ...
..
…
Service 1
Persistence service
Service 2
Error recovery
Messages sent to Service 2 while it was down
Stored messages Stored messages Stored messages
• Phase 1 : Missed messages recovery
Error recovery
Service 1
Persistence service
Service 2
Missed messages recovery Stored messages Stored messages Stored messages
Service 1
Persistence service
Service 2
• Phase 1 : Missed messages recovery • Phase 2 : Mixed recovery
Normal message reception
Streaming of the messages from the persistence
Error recovery
Service 1
Persistence service
Service 2
• Phase 1 : Missed messages recovery • Phase 2 : Mixed recovery • Phase 3 : Normal phase
Error recovery
Normal message reception
Meet our latest SPOF, the persistence
• Persistance only required at startup • Messages are also stored in the ZMQ outbound buffer
But actually, no
Migration time !
Service 1
Inter-bus gateway
Gateway
Service 2
Service 3
Service 4
QPID
Zebus
Broker
Summary • Running in production • Migration: 40% done • Gateway under heavy load • Owning the transport layer is double edged
Thank you