building a bank out of microservices (ndc sydney, august 2016)
TRANSCRIPT
Building a
Bank out of
Microservices
Microservices is a stupid word
‘Stupid V’ by Laura Lewis
What are we doing?
What are you doing?
James Lewis
James Lewis
Stefan Tilkov
James Lewis
Stefan Tilkov
James Lewis
Stefan TilkovMark Hibberd
James Lewis
Stefan TilkovMark Hibberd
James Lewis
Stefan TilkovMark Hibberd
Fred George
James Lewis
Stefan TilkovMark Hibberd
Fred George
Adrian Cockcroft
James Lewis
Stefan TilkovMark Hibberd
Fred George
Adrian Cockcroft
PricewaterhouseCoopers
James Lewis
Stefan TilkovMark Hibberd
Fred George
Adrian Cockcroft
PricewaterhouseCoopers
Martin Fowler
James Lewis
Stefan TilkovMark Hibberd
Fred George
Adrian Cockcroft
PricewaterhouseCoopers
Martin Fowler
Yamen Sader
James Lewis
Stefan TilkovMark Hibberd
Fred George
Adrian Cockcroft
PricewaterhouseCoopers
Martin Fowler
Yamen Sader
Alvaro Sanchez
James Lewis
Stefan TilkovMark Hibberd
Fred George
Adrian Cockcroft
PricewaterhouseCoopers
Martin Fowler
Yamen Sader
Alvaro Sanchez
Microservices?
micro - services
very small services
What’s a service?
W3C Web Services Architecture
W3C Web Services Architecture
RESTful services
W3C Web Services Architecture
RESTful services
W3C Web Services Architecture
RESTful services
Something entirely different?
How small?
“small enough and no smaller” - Sam Newman
We all agree on one thing…
Kenn
y Ba
stan
i ke
nnyb
asta
ni.c
om
Rod
Cur
rie
open
softw
ares
olut
ions
.com
Kenn
y Ba
stan
i ds
guid
e.bi
z
Susa
n H
all
then
ewst
ack.
io
Gra
ham
Lea
gr
aham
lea.
com
Jaso
n C
ham
bers
m
icro
svcs
.io
Sam
New
man
sa
mne
wm
an.io
Striving for microservices is stupid
“Cra
nes
near
the
Alex
ande
rpla
tz in
Ber
lin” b
y Ti
ll Kr
ech
Building a
Bank out of
Microservices
Some experts might think we’re stupid
‘Stupid V’ by Laura Lewis
Adrian Cockcroft - Battery Ventures
‘Kevin Rudd official portrait’ by Australian Department of Foreign Affairs and Trade
Questions
‘Kevin Rudd official portrait’ by Australian Department of Foreign Affairs and Trade
QuestionsWho are we, and why are we building a bank?
What exactly are we building?
How are we building it, and why that way?
What’s going well and what lessons have we learned?
Where have we got to?
Other questions welcome at the end
Who are we, and why are we building a bank?
What exactly are we building?
How are we building it, and why that way?
What’s going well and what lessons have we learned?
Where have we got to?
2003:
“Specialist Credit Card Institution” license
In-store Payments Competition
Revolutionise banking in Australia for small & medium businesses
“Home Market: 1941 in Worthington, Ohio” - Don O’Brien
2007: ……
2007: ……
First product launched
2007: ……
First product launched
Increasing revenue
Profitable!
Break-even took a while
Six years of losses
“I want my money sooner.”
Experiments!
Early 2014
Boss: “We’d like to start building a bank.
Early 2014
Boss: “We’d like to start building a bank.
Can your team start on that next week?”
Me: “Okay”
Who are we, and why are we building a bank?
What exactly are we building?
How are we building it, and why that way?
What’s going well and what lessons have we learned?
Where have we got to?
Core banking system for deposit accounts
Core banking system for deposit accounts
Interest-bearing
Core banking system for deposit accounts
Interest-bearing
Integrated with EFTPOS settlements
Core banking system for deposit accounts
Interest-bearing
Integrated with EFTPOS settlements
Xero integration: reconciliation & bill payments
Core banking system for deposit accounts
Interest-bearing
Integrated with EFTPOS settlements
Xero integration: reconciliation & bill payments
Mobile app for approving bills and other stuff
Who are we, and why are we building a bank?
What exactly are we building?
How are we building it, and why that way?
What’s going well and what lessons have we learned?
Where have we got to?
AWS
AngularJS React.js
HaskellScala
GoNodeJS
GitHub TravisCI Docker
MongoDB Cassandra
Hadoop
Spark
Euereka< 1,000 line
services!
AWS
AngularJS React.js
HaskellScala
GoNodeJS
GitHub TravisCI Docker
MongoDB Cassandra
Hadoop
Spark
Euereka
NOT< 1,000 line
services!
Our hardware in our racks at HA data centres
JS / jQuery
JavaGit
Jenkins
VMWare
MySQL
Spring + Spring MVC
Deployment Control
RabbitMQ
AndroidiOS C C#.NET
Tech stack != How are we building it?
Microservices!
Distributed system
Software SoftwareNetwork
Single-responsibility services
Single-responsibility services
An autonomous piece of software that interacts with other software
Single-responsibility services
‘Good work guys, why don't you take a break’ by Meme Binge
Single-responsibility services
Define the responsibility up front, document it
Single-responsibility services
Cohesion at the architecture level
Well-named services
“What is this service responsible for?”
Well-named services
Self-documenting code at the architecture level
Responsible (Autonomous) Services
Design services to maximise their chance
of completing their responsibility despite what’s going on around them.
Design services to minimise their dependency
on other services while doing work.
Responsible (Autonomous) Services
Agents at the architecture level
Data-owning services
Data-owning services
Encapsulation at the architecture level
Limited communication options
RabbitMQ for asynchronousREST for synchronous
Limited communication options
RabbitMQ for asynchronousREST for synchronous
Coding standards at the architecture level?
Preference for asynchronous comms
Preference for asynchronous comms
Background processing at the architecture level
Live-live redundancy is table stakes
Live-live redundancy is table stakes
Multi-threading at the architecture level
Deploys happen duringbusiness hours on weekdays
Deploys happen duringbusiness hours on weekdays
Deploys happen duringbusiness hours on weekdays
Architected for work/life balance
Deployed Incrementally
Some things we don’t have…
Continuous Delivery
(Fortnightly deploy)
Continuous Delivery
(Fortnightly deploy)
Stupid?
Service Discovery
(Host names in properties files)
Service Discovery
(Host names in properties files)
Stupid?
Strong service ownership
(Collective Code Ownership)
Strong service ownership
(Collective Code Ownership)
Stupid!
Non-Java services
(i.e.“best tool for the job” isn’t easy)
Non-Java services
(i.e.“best tool for the job” isn’t easy)
Missed the point?
Lots of production incidents
Why are we buildingthings this way?
Distributed system since 2006
2011
Backoffice webapp too big
Solution:
Smaller, more specific webapps for
new work
Backoffice webapp had too many responsibilities
Early 2013
Transaction switch deployed with links
Solution:
Links as separate services
Transaction switch + links =
too many responsibilities
Late 2013
Backend monolith
causing a lot of pain
Rapidly Growing Team
50% growth in a year … and more planned
Backend monolith had too many responsibilities
Solution:
No new responsibilities in the monolith
More webapps +More links +More backends +————————
More webapps +More links +More backends +————————Operations pain
First step:
Sit down and talk to each other
Retrospective on “First Deploys”
TRADE-OFF WARNING!
http://xkcd.com/1205/
Solution:
Automation for some things +
Repeatable manual processes
Early 2014
Let’s build a bank!
Early 2014
Let’s build a bank!
… with dozens of features and many teams all working
on it at the same time
Sounds great
Microservices can work for large greenfield projects
But…
maybe you need to
already be well along the
road to microservices?
‘Sherwin Range, Benton Crossing’ by “TheDailyNathan”
We got there in stages
Micro(?)servicesSmaller Services
DistributedMonolith
‘Hum
an e
volu
tion
sche
me’
by
José
-Man
uel B
enito
s
Key to progress:
Kill your monolith addiction
Adding to monoliths is easy Do the hard thing!
Who are we, and why are we building a bank?
What exactly are we building?
How are we building it, and why that way?
What’s going well and what lessons have we learned?
Where have we got to?
Small services + small teams =
productivity
Small services + small teams =
productivityLimited in responsibility
Small services + small teams =
productivityLimited in responsibility
3 pairs + tester + lead
Is this small?
Median about 2,500 LOC
Data ownership
Smaller, easier to understand schemas
Data ownership
Prevents inappropriate joining and data re-purposing
Preference for asynchronous comms
Why should your process wait for a response if no one is waiting for your
process?
Why should your process wait for a response if no one is waiting for your
process?
Temporal Coupling
Synchronous comms seem simple, but that’s an illusion
Increase service autonomy (and decrease temporal coupling) by caching data where processing
happens
TRADE-OFF WARNING!
We just increased service autonomy.
TRADE-OFF WARNING!
We just increased service autonomy.
We also introduced eventual consistency.
Has to be okay if this is not 100% up to date.
Data duplication becomes the norm
Synchronisation becomes a concern
Sometimes it’s better to move processing to where the data is
Whole tables of data?
Whole tables of data?
Distinguishing betweencommand messages and event messages
Use both to managecoupling of bounded contexts
When should I send a command?
When should I send an event?
How can I decide whether to send a command or an event?
Domain Coupling =
Knowledge of existence, responsibilities and domain language
of other components
What’s the difference between a Command and an Event?
What’s the difference between a Command and an Event?
The direction of the Domain Coupling
Heuristics• Cardinality • Responsibility Cohesion • Weight of Responsibility • Architectural Layers
Heuristics• Cardinality • Responsibility Cohesion • Weight of Responsibility • Architectural LayersTRA
DE-OFF
WARNING
!
Use commands and events.
Careful decisions can be used to control domain coupling.
Avoid distributed transactions
Be on the lookout for them!
Example of a distributed transaction hiding in plain sight
MessageBroker
TyroService
Business data Messages
Avoiding distributed transactions, preventing data loss with “Async Rabbit” pattern
TyroService
1. Business data & messages
2a. Messages 2b. Messages
MessageBroker
Idempotence
Idempotence
“Doing it twice = doing it once” - Graham Lea
Idempotence in Software Systems
Service A MessageBroker Service B
Receive ACK timeout & re-transmit
?
Service A MessageBroker Service B
Send ACK timeout & re-transmit
?
Idempotence in Software Systems
Service A MessageBroker Service B
Batch job re-run
⏲
Idempotence in Software Systems
Service A
MessageBroker
Service B
Redundant delivery
MessageBroker
Idempotence in Software Systems
Service
Many similar scenarios with REST
?
Idempotence in Software Systems
Hint: Duplicate detection counts
Deploy Independence
Let’s change this and deploy it!
Deploy Independence
Hope none of these break!
😬
Backwards Compatibility Testing
Backwards Compatibility Testing
Got a tester who can regression test every flow
that goes through here?
Backwards Compatibility Testing
(Do you even know every
activity that flows through here?!)
Backwards Compatibility Testing
It’s crucial
Automate it
Backwards Compatibility Testing
How?
For REST: Pact (& variants)
“Consumer-driven contracts”
Backwards Compatibility Testing with Pact
Source: https://github.com/realestate-com-au/pact
Backwards Compatibility Testing with Pact
Source: https://github.com/realestate-com-au/pact
Backwards Compatibility Testing with Pact
Polyglot implementations
Ruby, JVM, JS, growing…
https://github.com/realestate-com-au/pact
Backwards Compatibility Testing for message brokers
It’s hard!
Backwards Compatibility Testing for message brokers
Layer of indirection
Service A MessageBroker Service B
Messaging
Backwards Compatibility Testing for message brokers
https://www.flickr.com/photos/sydneyhistory/6899990333MessagingREST
Backwards Compatibility Testing for message brokers
Command: Senders are the “consumers”
Event:Receivers are the “consumers”
Who are we, and why are we building a bank?
What exactly are we building?
How are we building it, and why that way?
What’s going well and what lessons have we learned?
Where have we got to?
Do your own testing &
benchmarking of middleware
in your environment
https://www.flickr.com/photos/r00s/22248842680
Don’t have 2 - 3 different teams working in one
single-page webapp
‘Another rolling maul gets going’ by “Charlie”
Organise the work so that different
teams aren’t messing in the
same part of the codebase
Alternatively, organise the teams around the work or architecture
(“Inverse Conway Manoeuvre”)
Should You Do Integration Tests?
Should You Do Integration Tests?
(Cross-System Tests)
Contract
Contract
Journey
Cross System testsare always expensive
Cross System testsare always expensive
Some can be highly valuable
‘Gol
d-Ba
rs’ b
y M
ark
Her
pel
Cross System testsare always expensive
Some can be highly valuableTRA
DE-OFF W
ARNING!
TRADE-O
FF WARN
ING!
‘Gol
d-Ba
rs’ b
y M
ark
Her
pel
Mean time between failure
Mean time to recover
Mean time between failure
Mean time to recover
Merchantstaking payments
Mean time between failure
Mean time to recover
Cross-system tests
Merchantstaking payments
Mean time between failure
Mean time to recover
Cross-system tests
Merchantstaking payments
Financerunning a report
Mean time between failure
Mean time to recover
Cross-system tests Cheaper tests
Merchantstaking payments
Financerunning a report
Heuristic:
If we’d all be in serious trouble if it broke in Production, write a Cross-System Test
Heuristic:
If we’d all be in serious trouble if it broke in Production, write a Cross-System Test
Otherwise use unit testing, Pact testing and great monitoring for fast recovery
Keep reviewing your services’ responsibilities
Some may get “too big”
Some may be so small they cause issues
If Ops appears slow, don’t complain
Do something to help
‘Pushing the Car’ by Marina del Castell
Fortnightly deploys start to feel too slow
Get serious about security early
“Banksy Feb2011 131” by Lord Jim
Get expert help with security
Get expert help with security
Unless you do security full-time, you are not an expert
Layer Method Granularity
Application Message Authentication Message
Application Resource Authorisation User + Resource
Application User Authorisation Resource Type*
Application User Authentication User
Application Client Authorisation Resource Type*
Transport Encryption Connection
Transport Client Authentication Client
Transport Server Authentication vHost
Network Firewalling vHost
Physical Physical Security Rack
*Resource Type = URL pattern, e.g. /customers/{id}/transactions
UnauthorisedAccess
Eavesdropping
Message Tampering
Message Replay
How Do You Secure a Banking Service?
Bake security into your development process
Automate testing of security
‘Application field automotive’ by KUKA Systems GmbH
Automate testing of security
Best Option:
Failing test:“exploit possible!”
Automate testing of security
Second-Best Option:
Failing test: “protective tool is not integrated
correctly!”
Don’t expect much help on security from the web!
Don’t expect much help on security from the web!
Google: “microservices security questions”
Just assume villains are already inside your infrastructure
“Anonymous Hacker” by Brian Klug
Who are we, and why are we building a bank?
What exactly are we building?
How are we building it, and why that way?
What’s going well and what lessons have we learned?
Where have we got to?
18months
18months
7teams (by the end)
18months
7teams (by the end)
300Klines of code
18months
7teams (by the end)
300Klines of code
6,500automated tests
18months
7teams (by the end)
300Klines of code
6,500automated tests
21new services
18months
7teams (by the end)
300Klines of code
6,500automated tests
41person years of dev
21new services
August 2015
banking license
October 2015
product launched
August 2015
banking license
October 2015
product launchedDecember 2015
$100m investment
August 2015
banking license
October 2015
product launchedDecember 2015
$100m investment
August 2015
banking license
Can we build on top of it?
Can we build on top of it?
Yes we can!
‘Bob the Builder’ by OpenCage Systems
February 2016
lending kick-off
May 2016
pilot ready
February 2016
lending kick-off
May 2016
pilot readyJuly 1, 2016
first loan accepted
February 2016
lending kick-off
Summary
Summaryhexagons
Stupid!loosely coupled
bounded contexthoneycomb
distributed
single responsibility
well-named
autonomousdata encapsulation
asynchronous comms
operations painautomationmonolith
eventual consistency
domain coupling
commands & events
heuristicsidempotence
backwards compatibility
Pact
benchmarking
cross-system tests
security
One last question…
Should you copy us?
NO
Learn from us, but…
Context
Context!
2011: More webapps
2013: More links
2013: More backends
Operational scaling addressed
Monolith first?
Monolith first?
TRADE-OFF WARNING!
Growing teams and growing products
are threatened by monoliths
Microservices is…
Microservices is…
a collection of approaches
Microservices is…
a collection of approachesto modern problems
Microservices is…
a collection of approachesto modern problems
in distributed computing
Microservices is…
a collection of approachesto modern problems
in distributed computingand organising collaborative work.
Do you have the right problems?
Startup?
Microservices probably doesn’t solve your problems!
Microservices techniques are not a package deal
Use them selectively depending on your current problems
You’re not stupid for doing something different
if it works in your context
More questions?
Graham Lea @evolvable http://www.grahamlea.com/
Tyro Payments @tyro
http://www.tyro.com/