going event drive + kafka a rabbitmq
DESCRIPTION
Kratko o event-driven architekture, zaklady messagingu, porovnanie mq systemov a scenarov pouzitia RabbitMQ a Kafka.TRANSCRIPT
1
going event driven+ kafka a rabbit-mq
java group #13 bratislava @danielharcek
2
Agenda
1. Case study2. Retrospektiva3. Event-driven(ness)4. Messaging5. JMS6. RMQ7. Kafka8. Otazky?
3
Pred
lb
web
rtrtrtrtrt
rt
r loader
p loader
MySQLreq
multithreadeddaemons
MySQLprof
I/O
I/Oread
I/Omov
I/Omov
network
I/Owrite
network
I/Owrite
network
reportscampaign
s
MySQLad
MySQLsys
reportsaggregate
reportscustom
reportsvisiitors
web
sync
tab delimited files chunked per minutemostly cron-scheduled once per day
network I/O
write
MySQLreq
archive
4
Po
lb
web
rtrtrtrtrt
rt
rabbit
MySQLreqarch
MySQLprof
I/Oread
network
I/Owrite
network
I/Owrite
network
reportscampaign
s
MySQLad
MySQLsys
reportsaggregate
reportscustom
reportsvisiitors
web
sync
request sent, persisted and
forwarded over network
daemons & batch (cron jobs)
network I/O
write
5
#pred_a_po
rtrtrtrtrt
rt
r loader
p loader
MySQLreq
MySQLprof
reportscampaign
s
reportsaggregate
reportscustom
reportsvisiitors
rabbit
reportscampaign
s
reportsaggregate
reportscustom
reportsvisiitors
rtrtrtrtrt
rt
versus
6
Co sme ziskali• usetrili sme VELA disk i/o• usetrili load na DB a zredukovali mnozstvo
DB hostov (zostal vlastne uz iba “archiv” requestov)
• zlepsili distribuovatelnost• reporty sa stali menej previazane (netrebalo
zdielany diskovy caching)• moznost jednoduchsie pridavat nove typy
worker reportov• moznost signalovat potrebu recountu• skoro-real time countre (predtym
komplikovane koli loadu na DB)• ak nam padol tracker tak sme nestratili
(takmer) ziadne requesty• zjednodusila sa nam distribucia a mohli sme
sa viac sustredit na optimalizaciu reportingu• ak padol report spravy si nan pockaju (to ale
nebol problem predtym)
rabbit
reportscampaign
s
reportsaggregate
reportscustom
reportsvisiitors
rtrtrtrtrt
rt
7
“Reaktivita” je dnes proste
KAJŠMENTKE
asynchronous, non-blocking, real-time, highly-available, loosely
coupled, scalable, fault-tolerant, concurrent, reactive, event-driven, push instead of pull, distributed,
low latency, high throughput
8
Event-driven(ness)
But now a new architecture has evolved to let developers conceptualize and build applications that satisfy today’s demands.We call these Reactive Applications. This architecture allows developers to build systems that are event-driven, scalable, resilient and responsive.http://www.reactivemanifesto.org/
NIC NOVE!
9
Potreba
Komplexne systemy pracujuce s tokmi velkeho objemu dat s potrebou odozvy v takmer realnom case, “neustalym” uptimom nasadzovane do cloudoveho prostredia s flexibilnym modelom skalovania a spravy.
napr. IoT, mobilne appky, automaticke tradingove systemy
10
Messaging 101Push based (zvacsa)Producer (agent)Consumer (sink)“Event consumers subscribe to middleware which receives notification of an event from producer(s).”Vyuziva messaging middleware (backbone)• vacsinou menej alebo viac sofistikovany Broker (message manager)• alebo broker-less framework, jednoducho “Queue”* (p2p, napr. ZMQ)Durability vs. persistence (subscription vs. message)Garancia radenia (ziadna, FIFO)*Garancia dorucenia (at-most-once, at-least-once, exactly once)
11
Messaging broker / broker-less
O(n2) O(n)
http://www.eaipatterns.com/ramblings/03_hubandspoke.html
12
Messaging broker / broker-less
broker ako adresar distribuovany broker distribuovany adresar
http://zeromq.org/whitepapers:brokerless
13
Messaging - rozhodujúce kritériá• potrebna priepustnost (velkost spravy a pocet sprav), latencia• clustering / HA• aka topologia (broker, nebroker, aky workflow)• perzistencia (mat consumerov ktori nie su pern. online)• moznosti routingu, filtrovania, fransformacie• push a konzumovania batchov sprav• delivery a ordering garancie• “multiplatformovost” (klienti, drivers) a vyspelost• podpora protokolov• ttl, delayed / scheduled messages, prioritizacia messagov• acknowledgment / receipt notification
14
Messaging basic patterns: Queue
• point-to-point• by default FIFO• message je spracovany PRAVE
jednym consumerom
• logovanie udalosti / monitoring• load leveling (cakaju vo fronte
tak ako system stiha)• load balancing (pridanie
novych async workerov)
producer queue consumer
producer queue consumer
consumer
consumer
15
Messaging basic patterns: Publish-Subscribe • hub / broadcast
• sprava od publishera je forwardovana na vsetkych subscriberov
• * topic
• propagacia udalosti (napr. MMO, push notifikacie, live updaty skore zapasu a podobne)
• integracie
producer topic consumer
consumer
consumer
16
Messaging basic patterns: Request-reply
• ring • blizke RPC• asynchronne spracovanie
odpovede
• klientska aplikacia posle search query, backend searchne, vysledok sa vrati naspat
• aplikacia si vyziada stav inej aplikacie (napr. progress tasku)
producer
request q
consumer
reply q
17
JMS• Prva verzia 1.0.2b v 2001, 2.0 v 2013• Java Message Service API• MOM rozhranie (Message Oriented Middleware)• Nepopisuje protokol! (teda dve implementacie JMS nemusia vediet
komunikovat)• Provider, Client – Producer / Consumer, Message, Queue (per
Consumer), Topic (distribucny mechanizmus)• Nema garantovany ordering, dorucenie at-most-once alebo once-
and-only-once (ak je message persistentny)• Point-to-point a Publish-Subscribe• Implentacie: ActiveMQ, Qpid, Redis, …
18
RMQ• Vyspely broker (komplexne moznosti routovania, workflows), dokumentacia,
tooling• AMQP +plugins• Drivre do vsetkych relevantnych jazykov a kopec pluginov• Publisher, Exchange, Route, Queue, Consumer• Echanges: Direct, Fanout, Topic, Headers• Exchange su by default transientne, ich durabilitu a persistenciu je treba
explicitne deklarovat• Consumeri maju push aj pull API• ACK, a volba kedy poslat, nie je 100%, expiracia atd• pub-sub je Fanout• clustering, federation (plugin) a queues replication• Vhodny ak menej ako 20k+/sec* a ak potreba komplexnych routing
scenarov
19
Kafka• distribuovany pub-sub messaging system, skor klient-server ako broker (urceny pre
konkretny typ use-casov – spracovanie real time aktivity streamov, zber metrik, logovanie)• 0.8.x, meni sa pod rukami• klienti pre kazdy relevantny jazyk• messages su persistovane na servri, kafka zapise a potom zisti kto fetchuje,
konfigurovatelny “rolling window of time” (cas alebo miesto na hdd), jedna kopia stremu pre N consumerov
• dorucenie v poradi a at-least-once garancia• producer-centric (t.j. producer si strazi svoj stav – rmq ma metadata na strane brokera• pull-based (data fetchujeme a mozme aj specifikovat offset – robit rewind)• Cosumer, producer, topic, partition (ak su consumer groups)• Potrebuje Zookeeper – distribuovany register / adresar, je pouzity na koordinaciu clustra
producerov, consumerov a “brokera”• Od 0.8 replikacia partitions, partitioning definovany producerom, Kafka rozhoduje ako
rozhodi partitions na brokerov, aj ACK• Vhodny ak treba vysoku priepustnost (100k+/sec)*
20
Event-driven(ness)Umoznuje navrhovat systemy, kde volne previazane komponenty (asynchronne) komunikuju prostrednictvom sprav a takyto dizajn vediet k implementacii ktora zjedodusuje rozsirovanie a spravu systemu.
Pouzitie je teda na distribuciu uloh (routing), spravu (management) systemu, transformaciu* a kontrolu (monitoring).
Asynchronicita umoznuje efektivne zdielat prostriedky “jedneho HW vlakna” / vypoctovej jednotky resp. komunikacneho kanala. Non-blocking vedie k nizsej latencii a vyssej priepustnosti. Konkurentnost priamo v dizajne.
21
Otázky?
deh'-ku-yemza pozornost
a chlapcom z mrkvosoftu za to ze powerpoint NEMA autosave recovery-save sa nepocita!