apache kafka
TRANSCRIPT
Yogesh..BG27th Nov 2015
• Introduction• Features• Architecture• Storage• Topic• Producer and consumer• Broker• Zookeeper• Use Cases• Comparison with RMQ & AMQ
05/01/2023 Confidential 2
Agenda
Introduction
05/01/2023 Confidential 3
Open Source messaging Q written in scala.LinkedIn Corp -> Apache
Producer1
Producer2Queue
Consumer1
Consumer2
Consumer3
Features
05/01/2023 Confidential 4
• Brokered
• Distributed
• high throughput for publish and subscribe
• easy scalable
• fast
• replicated commit log service
• partitioned
• stores messages on disk
• In order delivery, per partition.
Architecture
05/01/2023 Confidential 5
Elements are:• Messages• Producer• Consumer• Topic• Broker [Cluster]• Zookeeper
Broker
05/01/2023 Confidential 6
• Stateless brokers.
• Transient System – deletes data
• Time-based SLA for the retention policy.
• Zero copy
Storage
05/01/2023 Confidential 7
Topic
05/01/2023 Confidential 8
• Unique identification for messages – offset• Consumer can change the offset to re consume or skip the message• Replicated among the configurable number of servers.• Retention policy• Parallelism at partitions level
Producers and Consumers
05/01/2023 Confidential 9
Producers:• Decides which message to which partition - LB.• Batch the messages• Async Send
Consumers:• Pull• Queuing and pub-sub• Consumer groups - cluster of consumers• Ordered per partition
Zookeeper
05/01/2023 Confidential 10
Use Cases
05/01/2023 Confidential 11
• Messaging
• Website activity tracking
• Metrics
• Log Aggregation
• Stream Processing
• Event Publishing
Comparison
05/01/2023 Confidential 12
Sooo fast… How come?
05/01/2023 Confidential 13
1. No wait.
2. efficient storage format. Header -> 9bytes (144 bytes in AMQ(As per JMS)).
3. Zero copy 4. Compressing multiple messages – message set
Thank YouYogesh..BG
05/01/2023 Confidential 14