atmosphere conference 2015: pubsub++ - few tips that make your life with kafka easier

Krzysztof DębskiAtmosphere 2015

PubSub++or how to make life with Kafka easier

@DebskiChris Kraków, 19.05.2015, Atmosphere

Why do we need a PubSub?


Service

Service

Service

Service

Service

Service

Service

Service

Service

Service

Service

Distributed environment


Service

Service

Service

Service

Service

Service

Service

Service

Service

Service

Service

Domain

Domain

Domain

Distributed environment


Orchestration vs. Choreography


Kafka


Kafka architecture

Service

Producer

Service

Consumer

KafkaBroker

Zookeeper< Kafka 0.9


Kafka data1

0 9 78 6 5 34 2 1

8 5 4 2 1

10 9 7 6 3

5 4 2

8 1

Data Topic Partition


Consumer groups

Consumer 1

Broker

P0

Broker

P1

Broker

P2

Broker

P3

Consumer 2

Consumer 3

Consumer 4

Consumer 5

Consumer 6

Consumer group 1 Consumer group 2


Publish & subscribe

Producer_1…Producer_n

Consumer_group_1Consumer_group_2

Remove old messages

Remove old events

Publish event

Read eventRead event


Leader & Replicas

Service

Producer

Service

Consumer

Broker

Zookeeper

Broker

Broker

P1 P0

P2 P1

P0 P2


Rebalancing leadersBroker 1

Broker 2

Broker 3

P1 P0

P2 P1

P0 P2

Topic: test Partition count: 3 Replication factor: 1Configs: retention.ms=86400000

Topic: test Partition: 0 Leader: 3Replicas: 3, 1 ISR: 3, 1Topic: test Partition: 1 Leader: 1Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2Replicas: 2, 3 ISR: 2, 3



Broker 2

Broker 3

P1 P0

P2 P1

P0 P2


Topic: test Partition: 0 Leader: 1Replicas: 3, 1 ISR: 1Topic: test Partition: 1 Leader: 1Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2Replicas: 2, 3 ISR: 2



Broker 2

Broker 3

P1 P0

P2 P1

P0 P2


Topic: test Partition: 0 Leader: 1Replicas: 3, 1 ISR: 1, 3Topic: test Partition: 1 Leader: 1Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2Replicas: 2, 3 ISR: 2, 3


Acknowledges

0 - don’t wait for response from broker

1 - only leader needs to respond

-1 - all replicas in sync

Spee

d

Safety


Events security

Basically, there is no authentication

You can create any topic

You can publish everywhere

You can subscribe to any topic


Synchronous world

RESTService

AService

B


Asynchronous made simple

Service A

Service B

REST REST/JMS


Hermes

Hermes Frontend

Hermes Frontend

Hermes Frontend

Hermes Consumer

Hermes Consumer

REST

REST, JMS


Event identification

Hermes Frontend

KafkaBroker

POST{“event”: ”test”}

{ "id": "58d7ff07-dd0e-4103-9b1f-55706f3049e6", "timestamp”: 1430443071995, “data”: {“event”: ”test”}}

HTTP 201 CreatedMessage-id: 58d7ff07-dd0e-4103-9b1f-55706f3049e6


100 ms timeout(on the client side)


Network

Nagle’s algorithm 42 ms -> 6 ms

High load on client serverswrong timeout calculation


65 ms timeout(on the server side)


Normal operation

Hermes Frontend

Producer

Hermes Consume

rConsumer

KafkaBroker

Zookeeper

POST

HTTP201Created


Slow kafka response

Hermes Frontend

Producer

Hermes Consume

rConsumer

KafkaBroker

Zookeeper

POST

HTTP202Accepted


Improved offset management

HermesProducer

Hermes consumer

Remove old messages

Publish event

Commited

Local unsent queue

Read event


Resend messages

Hermes consumer

Remove old messages

Local unsent queue

New event

Serviceinstanc

e


Resend messages

Hermes consumer

Remove old messages

Local unsent queue

New event

Serviceinstanc

e

HTTP 503 Unavailable


Resend messages

Remove old messages

Local unsent queue

New event

Serviceinstanc

e

HTTP 503 Unavailable

Check TTL & Add to queue

Hermes consumer


Resend speed

100% adapt 1/s 1/min


Lost events

Hermes Frontend

Producer

Hermes Consume

rConsumer

KafkaBroker

Zookeeper

Tracker

Publicationdata Deliv

ery

attempts


Groups and Topics

pl.allegro.hermes.demo.basic Group Topic


Security

Authentication and authorization interfaces

You can create any topic in your group

You can publish everywhere

Group owner defines subscriptions


Multi datacenter

Hermes Frontend

Hermes Manager

Hermes Frontend

Hermes Consumer

Hermes Consumer D

C 2D

C 1


What’s in it for me?


What we plan to do?


Lower overhead

Avro

RFC 7540


Avro

RFC 7540 => HTTP/2

Lower overhead


Improve security

Service authentication


What’ve we learned?


Tradeoffs

You cannot have exactly one delivery

http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/


Clients

Not all clients are the same.


Early adopters are your friends

You know exactly what features are needed.

@debskichris

allegrotech.io

atmosphere conference 2015: pubsub++ - few tips that make your life with kafka easier

Internet

debskichris krakw

p1 p0 p2 p1 p0 p2 topic

test partition count

data topic partition

atmosphere orchestration

atmosphere kafka data

n consumer

atmosphere events security