build your own service bus v2

Service Bus

ALT.NET Paris October 2013 ABC Arbitrage

Build your own

Olivier Coanet OMG this guy does not tweet

Kevin Lovato @alprema

ABC Arbitrage Asset Management www.abc-arbitrage.com

ALT.NET France www.altnetfr.org www.meetup.com/altnetfr

Service Bus Build your own

Application 1

ESB

Application 2

Application 3

Copyright Bullshit Consulting ®

The tube is a lie

Qpid Service Bus V1: broker-based

Application 1 Application 2

Qpid.Client Abc.ServiceBus

Application

Qpid

Broker-based Transport

layer

Bus library

Qpid.Client Abc.ServiceBus Application 1

Qpid

Qpid.Client Abc.ServiceBus Application 2

Application 3 Abc.ServiceBus

Qpid.Client ESB + infrastructure services

Qpid.Client Abc.ServiceBus

Service 1

Qpid

Qpid.Client Abc.ServiceBus

Service 2

Service 3 Abc.ServiceBus

Qpid.Client

And they were blissfully, blissfully happy forever and ever

And they were blissfully, blissfully happy forever and ever

NEW REQUIREMENTS

Higher troughtput

Better latency

Custom messaging usage was missing Service Bus features

and the stack was kind of

version

LEFT DEAD

.NET client

Buy or Build?

Brokerless transport recipe

• Peer to peer communication protocol

• Peer directory

Brokerless transport recipe

• Optional and parallel persistence

Brokerless transport recipe

Chef tip: • Use interoperable ingredients

Brokerless transport recipe

The cherry on the cake: • Build a gateway for Qpid communication

Brokerless transport recipe

Maybe you want implementation details?

P2P communication

Protocol Buffers

“ØMQ sockets are the world-saving superheroes of the networking world”

Pieter Hintjens

Service 1 Service 2 Service 3

Service 4

PUSH PUSH PUSH

PULL

Who do we connect to?

Connecting to the directory

Service 1 Service 2 Service 3

Directory

Peers list + subscriptions

Service 1 Service 2 Service 3

Directory

Connecting to the directory

RegisterCommand + I listen to HelloEvent

Service 1

New service ID + subscriptions

Service 2 Service 3

Directory

Connecting to the directory

Service 1 Service 2 Service 3

Directory

Connecting to the directory

HelloEvent

The directory, our new SPOF

Connecting to the directories

Directory 1 Directory 2

Service 1 Service 2 Service 3

RegisterCommand + I listen to HelloEvent

Connecting to the directories

Directory 1 Directory 2

Service 1 Service 2 Service 3

RegisterCommand + I listen to HelloEvent

Connecting to the directories

Directory 1 Directory 2

Service 1 Service 2 Service 3

HelloEvent listeners: Service 2

HelloEvent listeners: Service 1

Connecting to the directories

Directory 1 Directory 2

Service 1 Service 2 Service 3

HelloEvent listeners: Service 2

HelloEvent listeners: Service 1

RegisterCommand Peers list + subscriptions

Connecting to the directories

Directory 1 Directory 2

Service 1 Service 2 Service 3

HelloEvent listeners: Service 2

HelloEvent listeners: Service 1

HelloEvent listeners: Service 2

Cassandra to the rescue

Directory 1 Directory 2

What if a service goes down?

HelloEvent

Service 1

Persistence service

Service 2

Copy of HelloEvent

Persisting messages

Stored messages Stored messages Stored messages

Persisting messages

Service 1

Persistence service

Service 2

HelloEvent Ack

• Persistence service acting as frontend

• Cassandra for storage

Persistence service

Messages persistence model

Row Key

Abc_Service1 2013-10-03 08:25:38 2013-09-03 11:03:52 …

Serialized message …

Abc_Service2 2012-03-09 23:01:00 ...

..

…

Acks persistence model

Row Key

Abc_Service1 2013-10-03 08:25:38 2013-09-03 11:03:52 …

null …

Abc_Service2 2012-03-09 23:01:00 ...

..

…

Service 1

Persistence service

Service 2

Error recovery

Messages sent to Service 2 while it was down

Stored messages Stored messages Stored messages

• Phase 1 : Missed messages recovery

Error recovery

Service 1

Persistence service

Service 2

Missed messages recovery Stored messages Stored messages Stored messages

Service 1

Persistence service

Service 2

• Phase 1 : Missed messages recovery • Phase 2 : Mixed recovery

Normal message reception

Streaming of the messages from the persistence

Error recovery

Service 1

Persistence service

Service 2

• Phase 1 : Missed messages recovery • Phase 2 : Mixed recovery • Phase 3 : Normal phase

Error recovery

Normal message reception

Meet our latest SPOF, the persistence

• Persistance only required at startup • Messages are also stored in the ZMQ outbound buffer

But actually, no

Migration time !

Service 1

Inter-bus gateway

Gateway

Service 2

Service 3

Service 4

QPID

Zebus

Broker

Summary • Running in production • Migration: 40% done • Gateway under heavy load • Owning the transport layer is double edged

Thank you