Events & Microservices

Saul Caganoff, CTO Sixtree@scaganoff

Yamen Sader@yaamehn

Introduction• Asynchrony is considered a microservice characteristic– But not often discussed

• In the top 20 Google results– 7 mentioned async/choreography or events– 3 of those by Chris Richardson or Martin Fowler

• Mostly mentioned in passing• Honourable Mentions:

– http://www.infoq.com/articles/microservices-intro– http://

highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html

WHY EVENTS?

Bounded Contexts

Finance

Sales

Fulfilment

Product

Online bookstore is the “Hello World” of enterprise systems

http://maribyrnong.com.au

Sales Context

Wishlist

Book

ISBNTitleAuthorPrice

Order

StatusTimestampSub-totalGSTTotal

Item

Quantity

History

CustomerNameAgeLoyalty Status

Finance Context

Wishlist

Book

ISBNTitleAuthorPrice

Order

StatusTimestampSub-totalGSTTotal

Item

Quantity

History

CustomerNameAgeLoyalty Status

Credit CardHolder NameNumberExpiryCode

Fulfilment Context

Book

ISBNTitle

CustomerName

ReceiptOrder NumberSub-totalGSTTotal

AddressStreet NumberStreet NameSuburbPostcode

CourierDepotLocationDriver

Shipment

Dispatch TimeETALocation

Product Context

Book DescriptionAbstractPicture

Review

Author

NameBiography

InventoryLocationQuantity

ISBNTitlePrice

Coupling & Cohesion• Cohesion – how well a subsystem forms a

uniting whole– How well a bounded context does its job

• Coupling – the propensity for change in one part of a system to disrupt another– how badly “stuff” leaks across the boundaries

• Coupling is evil – it sneaks into solutions in all kinds of insidious ways– But… distributed monolith is the “worst kind of evil”

Coupling & Cohesion• Microservices strive to eliminate coupling• Insidious sources of coupling– Shared databases– Single Common Data Model (vs bounded contexts)– DRY – shared code– APIs generated from code– Poor client code– Misunderstood ‘transactions’– Assumptions about business processes– Organizational structure– Services…

Business Processes• End-to-end business processes require

coordination across bounded contexts

Finance

Sales Ful-filment

Product

Business Processes• Browse the inventory• Create order• Enter Shipping Details• Quote shipping charge• Make the payment• Update inventory• Create Shipment• Pick & Pack• Courier• Accept delivery

GET /productsPOST /orderPUT /order/:id/addressGET /3pl/charges?zip=…POST /finance/payment?order=POST /productsPOST /3pl/shipmentGET /3pl/shipment/:idGET /3pl/shipment/:id/addressPUT /3pl/shipment/:id?status=

Business Processes• Orchestration – Centralized coordination– E.g. the user-agent, BPM engine, service facade– the “god” object

• Hypermedia– HATEOAS– But what about at the edges?– How much should one BC know about another BC?

“microservices architecture will never be SOA done right unless it means building hypermedia systems of microservices” – Jason Bloomberg

http://intellyx.com/2015/07/20/are-microservices-soa-done-right/

Some Perspective

DB Triggers Monolith Orchestration Hypermedia EDA0

2

4

6

8

10

12

Coupling

Data Synchronization• How to handle data “owned” by multiple

domains– Glibness: “don’t” have multiple data owners– …in reality it is impossible to eliminate– …especially if legacy or COTS systems involved.

• Some data synchronization is naturally conveyed by the business process

• …some is not• Another topic!

Choreography• Use events to coordinate activity across bounded

contexts• Publishers & Subscribers are highly decoupled– Event subscribers can come & go minimal or no code or

configuration changes• Publishers & Subscribers are more resilient to

change – either planned or unplanned– E.g. events redelivered in the event of subscriber failure

– either to the restored subscriber, or a replacement

A Dependency View

Finance

Sales

Sales Tells Finance

Finance

Sales

Finance Asks Sales

Sales

Sales Announces

Choreography• But trade-offs• Asynchronous events lie at the heart of some of the

microservices challenges– Centralised orchestration is harder to change, but easier

to control/monitor– Event-based choreography is easier to change but

harder to control/monitor• Distributed logging/monitoring systems are needed

to provide a view into asynchronous interactions

An Aside• No global view may seem like a major downside• But very rare for anyone to know this anyway

• When X happens, we do A and B• When A happens, we do F• Etc

• The more ‘inter-domain’ concerns can be solved by just depending on events, the more likely you are to have the ‘correct’ boundaries

Coordination• Think about hypermedia inside boundaries• Events outside the boundaries

Finance

Sales Ful-filment

Product

MODELLING EVENTS

Commands & Events• Duality between commands & events– Commands cause state transitions– Events reflect state transitions

• An external observer can infer the state of a resource from its event-stream– This is the essence of event-sourcing & CQRS

State ModellingEmptyCart

FullCart

Quoted

Paid

POST /cart

PUT /cart/:id { book: … }

PUT /cart/:id { book: … }

DELETE /cart/:id/books/:id

PUT /cart/:id { address: … }

POST /payments { cart: …, card: {…}}

Commands (HATEOAS){create: { cart: … } … }

{add: { cart: …, book: {…}} ..}

{add: { cart: …, book: {…}} ..}

{del: { cart: …, book: {…}} ..}

{quoted: { cart: … } … }

{paid: { cart: … } … }

Events (EATNOAS)

Commands & Events

Finance

Sales Ful-filment

ProductInventoryMarketing

CommandsEvents

Reporting

Logistics

Event Structure• Immutable structures• Typical properties– Timestamp– Source– Entity URI– Action (state transition)

• The MVP is to notify that a resource has changed and allow subscribers to pull the update if needed

• Don’t forget Postel’s Law, versioning etc

Event Granularity• Customer details changed?• Or email address changed?

• Some considerations:– Event storm with the business or the technical

specialists, they will call out ‘special’ events– ‘Eye of the Beholder’ ie the subscriber determines

deltas important to them– Expect to get this wrong once or twice

Concurrency• Race Conditions• The importance of order• Single Master vs Multi-Master• Deltas vs Copies• Action indicators• CRDTs – conflict-free replicated data types

• …think in terms of events rather than copies

EVENT TRANSPORTS

Pull vs Push• Pull = subscribers beat– Subscriber never get overwhelmed– Messages pile up on publisher

• Push = publishers beat– Publisher never gets backed up– Subscribers can be overwhelmed

• Answer: use an intermediary tuned to this problem

Intermediary

Known Quantity• Events are highly regular– Append only series– Generic attributes (topics, metadata)– Timestamped– Consumed in sequence– Like a very simple database

• Common to standardise on an intermediary– Like standardising on a database– Usually exposed directly, a platform service

Syndication (Pull)• Service exposes an RSS/atom feed• Consumers do all the hard work:

– Subscription & Concurrency– State Management (where am I up to?)– Retries

• Positives– Very easy…well-known pattern– Web-scale

• Negatives– Polling– High latency– Consumer concurrency very difficult

Web Hooks (Push)• Service POSTs events to subscribers• Publishers do all the hard work:– Manage subscribers– Managed guaranteed delivery/ retries

• Positives– Not polling, relatively lower latency– Web technology (HTTP endpoints and load balancers)

• Negatives– Consumer concurrency a little harder– Retries and history difficult

Message Queues (Push)• Events published to queues/topics– E.g JMS, AMQP, ZMQ, SQS/SNS, IronMQ

• Positives– Mature, well-known technologies– Good open-source implementations available– Competing consumer pattern

• Negatives– Extra infrastructure– Reliability depends on reliable file-systems

• SANs, Distributed file-lock managers– No history or replay– Slow consumer problem

Pull vs Push (redux)• Pull– Subscriber tracks state– Intermediary simplified– Consumer concurrency more difficult– Natural history is kept

• Push– Intermediary tracks consumer state– Can create a bottleneck and contention– Consumer concurrency simpler– Generally no history or replay

Apache Kafka• Distributed log

– Consumers tail the logs

• Consumers tail this log, but:– Server manages consumer position (if desired)– Server manages consumer concurrency– Subscription is very simple– Pull is very low latency (push-like)– Replay and history is native– Application level replication

• Others (eg Event Store) provide some similar features (history, pull + push), but none as rich in capability

Event Processing• Simple events are very useful for data synchronization or

process coordination• Continuum of complexity

• Some tools: Drools, Riemann.io, Apache Storm, Apache Spark (Streaming), Samza, etc

• This is a whole ‘nother topic

Cardinality State Tools

Single event Simple Seriously?

Multiple events/stream Simple Stream processor

Multiple events/stream Complex / Rules Complex Event Processor

Wrap-Up• Bounded contexts are important

– they promote cohesion and reduce coupling• You need to decide how to coordinate data & processes across

boundaries– Orchestration (relatively higher coupling)– Hypermedia (couples across boundaries)– Events

• Prefer Hypermedia within bounded contexts & events across bounded contexts

Events have many benefits but also some downsides - they are an important tool in your toolbox