Real Time Investment Alerts using Apache Kafka at ING Bank

Marcos Maia & Tim van Baarsen

GOTO AMSTERDAM – JUNE 2019

Introduction

Tim van BaarsenSoftware Engineer @ ING

@TimvanBaarsen

Marcos MaiaSoftware Engineer @ ING

@thegroo

ING is active in more than 40 counties

4

https://www.ing.jobs

Agenda

5

• Use case @ ING• Mifid II• Mobile Investments App

• Overall Architecture• Demo• Metrics / Monitoring• Kafka streams• Local development / Code walk through• Lessons learned• Questions

Disclaimer

6

Both the code and high level architecture we show here today is

not the actual product we developed within ING!This example is inspired by our work

Domain Lingo

7

Tick Price update on stock=

Mifid II: “Inform customers when a stock drops -10% in price at the end of the day!”

8

Mifid IIRegulations

At the end of the day?!

Markets in Financial Instruments Directive II = Regulatory reform financial markets European Union came into effect on January 3, 2018

Mifid II: “Inform customers when a stock drops -10% in price at the end of the day!”

9

Mifid IIRegulations

"No, in real time!Markets in Financial Instruments Directive II =

Regulatory reform financial markets European Union came into effect on January 3, 2018

Screens of our mobile Investments app

10

price going up!

specific price

-10% drop in price

Recurring alerts

2.000 – 12.000 stock updates a second!

11

# $

The perfect “match”

12

❤

Overall architecture

13

Alert serviceAPI

Matching Service

Stocks update receiver

Internet

“SMS Platform”

API

Receive stock price updates ~ 12.000/sec

Publish stock price updates to Kafka

Create alertsDelete alerts

List alerts Customer receives

investment alert

Consume updates & match against alerts created by

customers

In case of a matchsend a sms to

customer

Customer alert(s) stored in database

Notification Service

Schema Registry Zookeeper

“Push Platform”

API

"

Quick Demo

14

Monitoring - Metrics

15

• Micrometer

• Think SLF4J, but for metrics (Jon Schneider)

• Application metrics

• Out of the box metrics

• Business metrics

• Vendor Neutral

Monitoring – Real-time?

16

Kafka Streams Join

17

Matching Service

raw-stock-quote stock-quote-tick stock-quote-metalog-compaction

Raw-tick

stock-quote-ticker RocksDB

Key value

ING Raw-tick

GOOGL Raw-tick

AAPL Raw-tick

Key value

ING Meta-data

GOOGL Meta-data

AAPL Meta-data

GlobalKTableKStream

Local development

18

• Docker• Kafka• Zookeeper• Kafka manager• All our Spring Boot applications

• Producers for development• Custom Kafka command line producer

• Lab project

Spring Kafka

19

• Producing: • ProducerFactory• KafkaTemplate

• “The producer is thread safe and sharing a single producer instance across threads will generally be faster than having multiple instances.”

• Consuming• Kafka consumers are not thread safe• @KafkaListener• MessageListener (many different interfaces)• Access to ConsumerRecord• Access to manual control to commit the offset• Scale up multiple consumer (threads) within the application

Spring Kafka

20

• Integration tests

• Junit rule: EmbeddedKafkaRule• @EmbeddedKafka

• Metrics

• Kafka consumer metrics out of the box (Micrometer)

Lessons learned

21

• Use Spring Kafka after you understand the Kafka APIs

• Error handling • (De)serialization -> Apache Avro

• Think about # of partitions per topic upfront if possible (routing)• Can your consumers keep up? Can you scale up?

• Monitor your topics / streams in production• Based on metrics -> Micrometer

Lessons learned

22

• Kafka clients are backward compatible with Kafka brokers

• Kafka Streams• KTables• GlobalKTables• Interactive Queries• RocksDB build for ssd• Describe• processing.guarantee=exactly_once

• Kafka topology• Plan ahead• small & multiple topologies

Questions

23

&

Contact

Tim van BaarsenSoftware Engineer @ ING

@TimvanBaarsen

Marcos MaiaSoftware Engineer @ ING

@thegroo

Backup slides

Matching in memory

28

Matching Service

Customer alert(s) stored in database

Matching Service

Matching Service

Matching Service

Database state of alerts in

memory cache

In memory cache is

updated based on event

published to Kafka

Alert created event

Alert matched event

Alert deleted event

ING AAPL NFLX ESTC

Kafka Topology

29