building a bank out of microservices (ndc sydney, august 2016)

Building a

Bank out of

Microservices

Microservices is a stupid word

‘Stupid V’ by Laura Lewis

What are we doing?

What are you doing?

James Lewis

James Lewis

Stefan Tilkov

James Lewis

Stefan Tilkov

James Lewis

Stefan TilkovMark Hibberd

James Lewis

Stefan TilkovMark Hibberd

Twitter

James Lewis

Stefan TilkovMark Hibberd

Twitter

Fred George

James Lewis

Stefan TilkovMark Hibberd

Twitter

Fred George

Adrian Cockcroft

James Lewis

Stefan TilkovMark Hibberd

Twitter

Fred George

Adrian Cockcroft

PricewaterhouseCoopers

James Lewis

Stefan TilkovMark Hibberd

Twitter

Fred George

Adrian Cockcroft

PricewaterhouseCoopers

Martin Fowler

James Lewis

Stefan TilkovMark Hibberd

Twitter

Fred George

Adrian Cockcroft

PricewaterhouseCoopers

Martin Fowler

Yamen Sader

James Lewis

Stefan TilkovMark Hibberd

Twitter

Fred George

Adrian Cockcroft

PricewaterhouseCoopers

Martin Fowler

Yamen Sader

Alvaro Sanchez

James Lewis

Stefan TilkovMark Hibberd

Twitter

Fred George

Adrian Cockcroft

PricewaterhouseCoopers

Martin Fowler

Yamen Sader

Alvaro Sanchez

Microservices?

micro - services

very small services

What’s a service?

W3C Web Services Architecture

W3C Web Services Architecture

RESTful services

W3C Web Services Architecture

RESTful services

W3C Web Services Architecture

RESTful services

Something entirely different?

How small?

“small enough and no smaller” - Sam Newman

We all agree on one thing…

Kenn

y Ba

stan

i ke

nnyb

asta

ni.c

om

Rod

Cur

rie

open

softw

ares

olut

ions

.com

Kenn

y Ba

stan

i ds

guid

e.bi

z

Susa

n H

all

then

ewst

ack.

io

Gra

ham

Lea

gr

aham

lea.

com

Jaso

n C

ham

bers

m

icro

svcs

.io

Sam

New

man

sa

mne

wm

an.io

Striving for microservices is stupid

“Cra

nes

near

the

Alex

ande

rpla

tz in

Ber

lin” b

y Ti

ll Kr

ech

Building a

Bank out of

Microservices

Some experts might think we’re stupid

‘Stupid V’ by Laura Lewis

Adrian Cockcroft - Battery Ventures

‘Kevin Rudd official portrait’ by Australian Department of Foreign Affairs and Trade

Questions

‘Kevin Rudd official portrait’ by Australian Department of Foreign Affairs and Trade

QuestionsWho are we, and why are we building a bank?

What exactly are we building?

How are we building it, and why that way?

What’s going well and what lessons have we learned?

Where have we got to?

Other questions welcome at the end

Who are we, and why are we building a bank?

What exactly are we building?

How are we building it, and why that way?

What’s going well and what lessons have we learned?

Where have we got to?

2003:

“Specialist Credit Card Institution” license

In-store Payments Competition

Revolutionise banking in Australia for small & medium businesses

“Home Market: 1941 in Worthington, Ohio” - Don O’Brien

2007: ……

2007: ……

First product launched

2007: ……

First product launched

Increasing revenue

Profitable!

Break-even took a while

Six years of losses

“I want my money sooner.”

Experiments!

Early 2014

Boss: “We’d like to start building a bank.

Early 2014

Boss: “We’d like to start building a bank.

Can your team start on that next week?”

Me: “Okay”

Who are we, and why are we building a bank?

What exactly are we building?

How are we building it, and why that way?

What’s going well and what lessons have we learned?

Where have we got to?

Core banking system for deposit accounts

Core banking system for deposit accounts

Interest-bearing

Core banking system for deposit accounts

Interest-bearing

Integrated with EFTPOS settlements

Core banking system for deposit accounts

Interest-bearing

Integrated with EFTPOS settlements

Xero integration: reconciliation & bill payments

Core banking system for deposit accounts

Interest-bearing

Integrated with EFTPOS settlements

Xero integration: reconciliation & bill payments

Mobile app for approving bills and other stuff

Who are we, and why are we building a bank?

What exactly are we building?

How are we building it, and why that way?

What’s going well and what lessons have we learned?

Where have we got to?

AWS

AngularJS React.js

HaskellScala

GoNodeJS

GitHub TravisCI Docker

MongoDB Cassandra

Hadoop

Spark

Euereka< 1,000 line

services!

AWS

AngularJS React.js

HaskellScala

GoNodeJS

GitHub TravisCI Docker

MongoDB Cassandra

Hadoop

Spark

Euereka

NOT< 1,000 line

services!

Our hardware in our racks at HA data centres

JS / jQuery

JavaGit

Jenkins

VMWare

MySQL

Spring + Spring MVC

Deployment Control

RabbitMQ

AndroidiOS C C#.NET

Tech stack != How are we building it?

Microservices!

Distributed system

Software SoftwareNetwork

Single-responsibility services

Single-responsibility services

An autonomous piece of software that interacts with other software

Single-responsibility services

‘Good work guys, why don't you take a break’ by Meme Binge

Single-responsibility services

Define the responsibility up front, document it

Single-responsibility services

Cohesion at the architecture level

Well-named services

“What is this service responsible for?”

Well-named services

Self-documenting code at the architecture level

Responsible (Autonomous) Services

Design services to maximise their chance

of completing their responsibility despite what’s going on around them.

Design services to minimise their dependency

on other services while doing work.

Responsible (Autonomous) Services

Agents at the architecture level

Data-owning services

Data-owning services

Encapsulation at the architecture level

Limited communication options

RabbitMQ for asynchronousREST for synchronous

Limited communication options

RabbitMQ for asynchronousREST for synchronous

Coding standards at the architecture level?

Preference for asynchronous comms

Preference for asynchronous comms

Background processing at the architecture level

Live-live redundancy is table stakes

Live-live redundancy is table stakes

Multi-threading at the architecture level

Deploys happen duringbusiness hours on weekdays

Deploys happen duringbusiness hours on weekdays

Deploys happen duringbusiness hours on weekdays

Architected for work/life balance

Deployed Incrementally

Some things we don’t have…

Continuous Delivery

(Fortnightly deploy)

Continuous Delivery

(Fortnightly deploy)

Stupid?

Service Discovery

(Host names in properties files)

Service Discovery

(Host names in properties files)

Stupid?

Strong service ownership

(Collective Code Ownership)

Strong service ownership

(Collective Code Ownership)

Stupid!

Non-Java services

(i.e.“best tool for the job” isn’t easy)

Non-Java services

(i.e.“best tool for the job” isn’t easy)

Missed the point?

Lots of production incidents

Why are we buildingthings this way?

Distributed system since 2006

2011

Backoffice webapp too big

Solution:

Smaller, more specific webapps for

new work

Backoffice webapp had too many responsibilities

Early 2013

Transaction switch deployed with links

Solution:

Links as separate services

Transaction switch + links =

too many responsibilities

Late 2013

Backend monolith

causing a lot of pain

Rapidly Growing Team

50% growth in a year … and more planned

Backend monolith had too many responsibilities

Solution:

No new responsibilities in the monolith

More webapps +More links +More backends +————————

More webapps +More links +More backends +————————Operations pain

First step:

Sit down and talk to each other

Retrospective on “First Deploys”

TRADE-OFF WARNING!

http://xkcd.com/1205/

Solution:

Automation for some things +

Repeatable manual processes

Early 2014

Let’s build a bank!

Early 2014

Let’s build a bank!

… with dozens of features and many teams all working

on it at the same time

Sounds great

Microservices can work for large greenfield projects

But…

maybe you need to

already be well along the

road to microservices?

‘Sherwin Range, Benton Crossing’ by “TheDailyNathan”

We got there in stages

Micro(?)servicesSmaller Services

DistributedMonolith

‘Hum

an e

volu

tion

sche

me’

by

José

-Man

uel B

enito

s

Key to progress:

Kill your monolith addiction

Adding to monoliths is easy Do the hard thing!

Who are we, and why are we building a bank?

What exactly are we building?

How are we building it, and why that way?

What’s going well and what lessons have we learned?

Where have we got to?

Small services + small teams =

productivity

Small services + small teams =

productivityLimited in responsibility

Small services + small teams =

productivityLimited in responsibility

3 pairs + tester + lead

Is this small?

Median about 2,500 LOC

Data ownership

Smaller, easier to understand schemas

Data ownership

Prevents inappropriate joining and data re-purposing

Preference for asynchronous comms

Why should your process wait for a response if no one is waiting for your

process?

Why should your process wait for a response if no one is waiting for your

process?

Temporal Coupling

Synchronous comms seem simple, but that’s an illusion

Increase service autonomy (and decrease temporal coupling) by caching data where processing

happens

TRADE-OFF WARNING!

We just increased service autonomy.

TRADE-OFF WARNING!

We just increased service autonomy.

We also introduced eventual consistency.

Has to be okay if this is not 100% up to date.

Data duplication becomes the norm

Synchronisation becomes a concern

Sometimes it’s better to move processing to where the data is

Whole tables of data?

Whole tables of data?

Distinguishing betweencommand messages and event messages

Use both to managecoupling of bounded contexts

When should I send a command?

When should I send an event?

How can I decide whether to send a command or an event?

Domain Coupling =

Knowledge of existence, responsibilities and domain language

of other components

What’s the difference between a Command and an Event?

What’s the difference between a Command and an Event?

The direction of the Domain Coupling

Heuristics• Cardinality • Responsibility Cohesion • Weight of Responsibility • Architectural Layers

Heuristics• Cardinality • Responsibility Cohesion • Weight of Responsibility • Architectural LayersTRA

DE-OFF

WARNING

!

Use commands and events.

Careful decisions can be used to control domain coupling.

Avoid distributed transactions

Be on the lookout for them!

Example of a distributed transaction hiding in plain sight

MessageBroker

TyroService

Business data Messages

Avoiding distributed transactions, preventing data loss with “Async Rabbit” pattern

TyroService

1. Business data & messages

2a. Messages 2b. Messages

MessageBroker

Idempotence

Idempotence

“Doing it twice = doing it once” - Graham Lea

Idempotence in Software Systems

Service A MessageBroker Service B

Receive ACK timeout & re-transmit

?

Service A MessageBroker Service B

Send ACK timeout & re-transmit

?

Idempotence in Software Systems

Service A MessageBroker Service B

Batch job re-run

⏲

Idempotence in Software Systems

Service A

MessageBroker

Service B

Redundant delivery

MessageBroker

Idempotence in Software Systems

Service

Many similar scenarios with REST

?

Idempotence in Software Systems

Hint: Duplicate detection counts

Deploy Independence

Let’s change this and deploy it!

Deploy Independence

Hope none of these break!

😬

Backwards Compatibility Testing

Backwards Compatibility Testing

Got a tester who can regression test every flow

that goes through here?

Backwards Compatibility Testing

(Do you even know every

activity that flows through here?!)

Backwards Compatibility Testing

It’s crucial

Automate it

Backwards Compatibility Testing

How?

For REST: Pact (& variants)

“Consumer-driven contracts”

Backwards Compatibility Testing with Pact

Source: https://github.com/realestate-com-au/pact

Backwards Compatibility Testing with Pact

Source: https://github.com/realestate-com-au/pact

Backwards Compatibility Testing with Pact

Polyglot implementations

Ruby, JVM, JS, growing…

https://github.com/realestate-com-au/pact

Backwards Compatibility Testing for message brokers

It’s hard!

Backwards Compatibility Testing for message brokers

Layer of indirection

Service A MessageBroker Service B

Messaging

Backwards Compatibility Testing for message brokers

https://www.flickr.com/photos/sydneyhistory/6899990333MessagingREST

Backwards Compatibility Testing for message brokers

Command: Senders are the “consumers”

Event:Receivers are the “consumers”

Who are we, and why are we building a bank?

What exactly are we building?

How are we building it, and why that way?

What’s going well and what lessons have we learned?

Where have we got to?

Do your own testing &

benchmarking of middleware

in your environment

https://www.flickr.com/photos/r00s/22248842680

Don’t have 2 - 3 different teams working in one

single-page webapp

‘Another rolling maul gets going’ by “Charlie”

Organise the work so that different

teams aren’t messing in the

same part of the codebase

Alternatively, organise the teams around the work or architecture

(“Inverse Conway Manoeuvre”)

Should You Do Integration Tests?

Should You Do Integration Tests?

(Cross-System Tests)

Contract

Contract

Journey

Cross System testsare always expensive

Cross System testsare always expensive

Some can be highly valuable

‘Gol

d-Ba

rs’ b

y M

ark

Her

pel

Cross System testsare always expensive

Some can be highly valuableTRA

DE-OFF W

ARNING!

TRADE-O

FF WARN

ING!

‘Gol

d-Ba

rs’ b

y M

ark

Her

pel

Mean time between failure

Mean time to recover

Mean time between failure

Mean time to recover

Merchantstaking payments

Mean time between failure

Mean time to recover

Cross-system tests

Merchantstaking payments

Mean time between failure

Mean time to recover

Cross-system tests

Merchantstaking payments

Financerunning a report

Mean time between failure

Mean time to recover

Cross-system tests Cheaper tests

Merchantstaking payments

Financerunning a report

Heuristic:

If we’d all be in serious trouble if it broke in Production, write a Cross-System Test

Heuristic:

If we’d all be in serious trouble if it broke in Production, write a Cross-System Test

Otherwise use unit testing, Pact testing and great monitoring for fast recovery

Keep reviewing your services’ responsibilities

Some may get “too big”

Some may be so small they cause issues

If Ops appears slow, don’t complain

Do something to help

‘Pushing the Car’ by Marina del Castell

Fortnightly deploys start to feel too slow

Get serious about security early

“Banksy Feb2011 131” by Lord Jim

Get expert help with security

Get expert help with security

Unless you do security full-time, you are not an expert

Layer Method Granularity

Application Message Authentication Message

Application Resource Authorisation User + Resource

Application User Authorisation Resource Type*

Application User Authentication User

Application Client Authorisation Resource Type*

Transport Encryption Connection

Transport Client Authentication Client

Transport Server Authentication vHost

Network Firewalling vHost

Physical Physical Security Rack

*Resource Type = URL pattern, e.g. /customers/{id}/transactions

UnauthorisedAccess

Eavesdropping

Message Tampering

Message Replay

How Do You Secure a Banking Service?

Bake security into your development process

Automate testing of security

‘Application field automotive’ by KUKA Systems GmbH

Automate testing of security

Best Option:

Failing test:“exploit possible!”

Automate testing of security

Second-Best Option:

Failing test: “protective tool is not integrated

correctly!”

Don’t expect much help on security from the web!

Don’t expect much help on security from the web!

Google: “microservices security questions”

Just assume villains are already inside your infrastructure

“Anonymous Hacker” by Brian Klug

Who are we, and why are we building a bank?

What exactly are we building?

How are we building it, and why that way?

What’s going well and what lessons have we learned?

Where have we got to?

18months

18months

7teams (by the end)

18months

7teams (by the end)

300Klines of code

18months

7teams (by the end)

300Klines of code

6,500automated tests

18months

7teams (by the end)

300Klines of code

6,500automated tests

21new services

18months

7teams (by the end)

300Klines of code

6,500automated tests

41person years of dev

21new services

August 2015

banking license

October 2015

product launched

August 2015

banking license

October 2015

product launchedDecember 2015

$100m investment

August 2015

banking license

October 2015

product launchedDecember 2015

$100m investment

August 2015

banking license

Can we build on top of it?

Can we build on top of it?

Yes we can!

‘Bob the Builder’ by OpenCage Systems

February 2016

lending kick-off

May 2016

pilot ready

February 2016

lending kick-off

May 2016

pilot readyJuly 1, 2016

first loan accepted

February 2016

lending kick-off

Summary

Summaryhexagons

Stupid!loosely coupled

bounded contexthoneycomb

distributed

single responsibility

well-named

autonomousdata encapsulation

asynchronous comms

operations painautomationmonolith

eventual consistency

domain coupling

commands & events

heuristicsidempotence

backwards compatibility

Pact

benchmarking

cross-system tests

security

One last question…

Should you copy us?

NO

Learn from us, but…

Context

Context!

2011: More webapps

2013: More links

2013: More backends

Operational scaling addressed

Monolith first?

Monolith first?

TRADE-OFF WARNING!

Growing teams and growing products

are threatened by monoliths

Microservices is…

Microservices is…

a collection of approaches

Microservices is…

a collection of approachesto modern problems

Microservices is…

a collection of approachesto modern problems

in distributed computing

Microservices is…

a collection of approachesto modern problems

in distributed computingand organising collaborative work.

Do you have the right problems?

Startup?

Microservices probably doesn’t solve your problems!

Microservices techniques are not a package deal

Use them selectively depending on your current problems

You’re not stupid for doing something different

if it works in your context

More questions?

Graham Lea @evolvable http://www.grahamlea.com/

Tyro Payments @tyro

http://www.tyro.com/