goto conference - monolithic batch goes eaming · monolithic batch goes eaming a story about one...
TRANSCRIPT
![Page 1: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/1.jpg)
Monolithic Batch GoesMicroservice Streaming
A story about one transformation
Charles Tye & Anton Polyakov
![Page 2: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/2.jpg)
![Page 3: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/3.jpg)
Who are We?
3 •
Anton Polyakov
Head of ApplicationDevelopment
2 years in Nordea
Charles Tye
Head of Core Services& Risk IT
17 years in Nordea
Develop solutions forMarket RiskCredit Risk
Liquidity RiskStress Testing
Messaging
Together with around 70 other people fromall over the world
What We Do
![Page 4: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/4.jpg)
Market Risk
4 •
The high level view
Quantify potentiallosses and exposures
Do many smallrisks add up to a bigrisk?
Can risks combine inunusual andunexpected ways?
![Page 5: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/5.jpg)
Market Risk
5 •
Line of Defence
Protect Nordea andour customers
Daily internal reportingand external reporting toregulators
Independent function
Analysis and insight intothe sources of risk
Control of risk
Management of capital
![Page 6: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/6.jpg)
Examples of Risk Analysis
6 •
Value at Risk
Look at last 2 years ofmarket history
Average of the worst 1%of outcomes
Simulate if the samething happened againtoday.
Highly non-linear butrequirement to drill inand find the drivers
![Page 7: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/7.jpg)
Examples of Risk Analysis
7 •
Stress Scenarios
“Black Swan” worst casescenarios Unexpected outcomes fromfuture events
Example: Brexit
Simulate if it happened
![Page 8: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/8.jpg)
An Interesting Technology Problem
8 •
Consistent
Non-linear
Volume
Speed
Risk Analysis:Everything has to be included= know when you are complete
Risk does not sumover hierarchies Drill-down is nontrivial Traditional OLAPaggregate &incrementdoesn’t work
10,000,000 ,000 ,000
Reactive nearreal-timecalculations Streaming dataFast correctionsand “what-if” Interactivesub-secondqueries on hugedata sets
![Page 9: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/9.jpg)
Challenge No 1.
Find the seams
Break it up
Reusable components
Replace a piece at a time
9 •
Spaghetti
![Page 10: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/10.jpg)
Challenge No 2.
10 •
Develop a new service
Integrate into the legacy system Reconcile the output Find and fix legacy bugs Fight complification
![Page 11: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/11.jpg)
Challenge No 3.
Batch is synchronous state transfer. Theonly way to achieve consistency?
11 •
Consistency is seriously hard to combine with streaming
Event sourced and streaming approach
More robust, scalable and faster,especially for recovery
Comes with a cost
![Page 12: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/12.jpg)
Challenge No 4.
Legacy SQL was slow
12 •
Partitions and horizontally scales out across commodity hardware.
Tougher challenges on terabyte-scale hardware due to NUMA limitations. Somecubes already > 200gb and larger ones planned.
Replace with in-memoryaggregation
Aggregate billions ofscenarios in-memory andpre-compute total vectorsover hierarchies (linear)
Non-linear measurescomputed lazily
Reactive and continuousqueries
![Page 13: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/13.jpg)
Solution: Microservices!Well almost…
Single responsibility – replace pieces of legacy from the inside out
Self contained with business functional boundaries• Independent and rapid development – team owns the whole stack• Organisationally scalable – horizontally scale your teams
Flexible and maintainable – evolve the architecture
Smart endpoints and dumb pipes
Innovation and short lifecycles
13 •
![Page 14: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/14.jpg)
The problem• Business:
• Multi-model Market Risk calculator for Nordea portfolio• VaR on different organization levels with 5-6 different models in parallel
• IT:• 7000 CPU hours of grid calculation• More than 4000 SQL jobs
• Graph with more than 10000 edges• Nightly batch flow
14 •
![Page 15: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/15.jpg)
How did it look like?
• Well, you know. 10 years of development
• In SQL
• No refactoring(who needs it?)
15 •
![Page 16: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/16.jpg)
Precisely, how did it look?
16 •
![Page 17: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/17.jpg)
Logical architecture
Monolith staged app
17 •
![Page 18: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/18.jpg)
Now a little of complication
Sloo-o-o-ow Fat. So it breaksCan be parallel?
18 •
![Page 19: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/19.jpg)
So what to do?
We all know the answer probably (sincewe are at this section ☺ )
- Find logically isolated blocks- Keep an eye on non-functional aspect- Think of how they communicate- Think about what happens if something dies
19 •
![Page 20: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/20.jpg)
Not quite a “classical” microservices…or?
produce enrich aggregate
- Request/response is not feasible- Synchronous interaction is too long- Some results are expensive to reproduce
20 •
![Page 21: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/21.jpg)
So we need…
A middleware which
- “Glues” services together- Caches important results- Serves as a coordinator and work distributor
21 •
![Page 22: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/22.jpg)
Scale out
Fast pub/sub Queues and setspull and dedup Distributed locks
22 •
![Page 23: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/23.jpg)
Scale out
Fast pub/sub Queues and setspull and dedup Distributed locks
Locks? Whoneeds locks?
23 •
![Page 24: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/24.jpg)
store store store
Pub/sub messaging as notifier
Producer Enricher Aggregator
consumer
Redis pub/sub
24 •
![Page 25: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/25.jpg)
But…
25 •
There are two main problems in distributed messaging:2) Guarantee that each message is only delivered once1) Guarantee messages order2) Guarantee that each message is only delivered once
![Page 26: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/26.jpg)
Enricher
Redis pub/sub
Incoming queue
Processing queue
EnricherProducer
store
Queues with atomic operations
BRPOPLPUSH
26 •
![Page 27: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/27.jpg)
Sets and Hmaps – all good for dedup
In eventually consistent world dedupis your best friend
store - HSET
EnricherMultiple inserts due to recovery
Consistent state due to dedup
27 •
![Page 28: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/28.jpg)
So how to scale out?
logically concurrently
Enricher<type A>Enricher
<type B>Enricher<type X>
Redis pub/sub
Aggregator<day 1>Aggregator
<day 2>Aggregator<day 3>Steal workFilter my events
RedLock + TTL
28 •
![Page 29: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/29.jpg)
Demo
store store store
Producer Enricher Aggregator
consumer
Redis pub/sub
Incoming queueProcessing queue
RedLock + TTL
29 •
![Page 30: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/30.jpg)
The Result and What We LearnedSuccess!
• Aggregate and produce risk: 5 hours → 30 mins• Corrections: 40 mins → 1 second• Earlier deliveries – more time to manage the risks• Faster recovery from problems• Happy risk managers
Important (and painful) to integrate new services into the existing system
Consistency is hard to combine with streaming (subject of another talk maybe)
When distributing remember first law of distributed objects architecture(do you remember it?)
30 •
![Page 31: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/31.jpg)
The Result and What We Learned
First Law of Distributed Object Design:
"don't distribute your objects"
31 •
![Page 32: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/32.jpg)
And of course…
32 •
https://dk.linkedin.com/in/charles-tye-a8aa88b
https://github.com/parallelstream/
![Page 33: GOTO Conference - Monolithic Batch Goes eaming · Monolithic Batch Goes eaming A story about one transformation ye & Anton Polyakov?](https://reader033.vdocuments.site/reader033/viewer/2022060502/5f1c3480ee4629070e039b66/html5/thumbnails/33.jpg)