Download - Apache Kafka Lightning Talk
![Page 1: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/1.jpg)
Q
Apache KafkaQ
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
QQ Q
Q
Q
Q
Q
Q
Q
QQ Q
Q
Q
Q
Jeff KunkleNov 22, 2013
Tuesday, December 3, 13
![Page 2: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/2.jpg)
Q
What is Kafka?
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
QQ Q
Q
Q
Q
Q
Q
Q
QQ Q
Q
Q
QTuesday, December 3, 13
![Page 3: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/3.jpg)
Kafka is a distributed, partitioned, replicated commit log service.
Tuesday, December 3, 13
![Page 4: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/4.jpg)
Tuesday, December 3, 13
![Page 5: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/5.jpg)
?Tuesday, December 3, 13
![Page 6: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/6.jpg)
... a non-JMS messaging system with a unique design.
Tuesday, December 3, 13
![Page 7: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/7.jpg)
Key Concepts1 Topics, Logs, and Partitions2 Consumer Groups3 Replication
Tuesday, December 3, 13
![Page 8: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/8.jpg)
Topics and Logs
Tuesday, December 3, 13
![Page 9: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/9.jpg)
partition 0
partition 1
partition 2
Topics and Logs
Tuesday, December 3, 13
![Page 10: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/10.jpg)
partition 0
partition 1
partition 2
old new
Topics and Logs
1 2 3 4 5 6 7log
1 2 3 4 5 6log
1 2 3 4 5log
Tuesday, December 3, 13
![Page 11: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/11.jpg)
partition 0
partition 1
partition 2
writes
old new
Topics and Logs
1 2 3 4 5 6 7log
1 2 3 4 5 6log
1 2 3 4 5log
Tuesday, December 3, 13
![Page 12: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/12.jpg)
1234567
partition 0
Pro
du
cers
123456
partition 1
12345
partition 2
P1
P2
Tuesday, December 3, 13
![Page 13: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/13.jpg)
1 2
34567
partition 0
Pro
du
cers
1
23456
partition 1
1
2345
partition 2
P1
P2
Tuesday, December 3, 13
![Page 14: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/14.jpg)
1 2
34567
partition 0
Pro
du
cers
1
23456
partition 1
1
2345
partition 2
P1
P2
bulk publishing
Tuesday, December 3, 13
![Page 15: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/15.jpg)
1 2
34567
partition 0
Pro
du
cers
1
23456
partition 1
1 2 3 4
5
partition 2
P1
P2
bulk publishing
Tuesday, December 3, 13
![Page 16: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/16.jpg)
1 2
34567
partition 0
Pro
du
cers
1
23456
partition 1
1 2 3 4
5
partition 2
P1
P2
parallel publishing
Tuesday, December 3, 13
![Page 17: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/17.jpg)
1 2 3 4
567
partition 0
Pro
du
cers
1 2
3456
partition 1
1 2 3 4
5
partition 2
P1
P2
parallel publishing
Tuesday, December 3, 13
![Page 18: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/18.jpg)
1 2 3 4 5 6 7partition 0
Pro
du
cers
1 2 3 4 5 6partition 1
1 2 3 4 5partition 2
P1
P2
Tuesday, December 3, 13
![Page 19: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/19.jpg)
1 2 3 4 5
1 2 3 4 5 6
1 2 3 4 5 6 7 partition 0
partition 1
partition 2
Co
nsu
me
rs
1 2 3 4 5 6 7
1 2 3 4 5 6
1 2 3 4 5
consumer group 2
consumer group 1
C3
C2
C1
C4
Tuesday, December 3, 13
![Page 20: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/20.jpg)
1 2 3 4 5
1 2 3 4 5 6
1
2 3 4 5 6 7 partition 0
partition 1
partition 2
Co
nsu
me
rs
1 2 3 4 5 6 7
1 2 3 4 5 6
1 2 3 4 5
consumer group 2
consumer group 1
C3
C2
C1
C4
Tuesday, December 3, 13
![Page 21: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/21.jpg)
1 2 3 4 51
2 3 4 5 6
1
2 3 4 5 6 7 partition 0
partition 1
partition 2
Co
nsu
me
rs
1 2 3 4 5 6 7
12 3 4 5 6
1 2 3 4 5
consumer group 2
consumer group 1
C3
C2
C1
C4
Tuesday, December 3, 13
![Page 22: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/22.jpg)
1 2 3 4 51
2 3 4 5 6
1
2 3 4 5 6 7 partition 0
partition 1
partition 2
Co
nsu
me
rs
1 2 3 4 5 6 7
12 3 4 5 6
1
2 3 4 5
consumer group 2
consumer group 1
C3
C2
C1
C4
Tuesday, December 3, 13
![Page 23: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/23.jpg)
12 3 4 5123
4 5 6
12
3 4 5 6 7 partition 0
partition 1
partition 2
Co
nsu
me
rs
1
2
3 4 5 6 7
1
2
3
4 5 6
1
2
3 4 5
consumer group 2
consumer group 1
C3
C2
C1
C4
Tuesday, December 3, 13
![Page 24: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/24.jpg)
12 3 4 5123
4 5 6
12
3 4 5 6 7 partition 0
partition 1
partition 2
Co
nsu
me
rs
1
2
3 4 5 6 7
1
2
3
4 5 6
1
2
3 4 5
consumer group 2
consumer group 1
C3
C2
C1
C4
bulk consumption
Tuesday, December 3, 13
![Page 25: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/25.jpg)
12 3 4 5123
4 5 6
123456
7 partition 0
partition 1
partition 2
Co
nsu
me
rs
1
2
3456
7
1
2
3
4 5 6
1
2
3 4 5
consumer group 2
consumer group 1
C3
C2
C1
C4
bulk consumption
Tuesday, December 3, 13
![Page 26: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/26.jpg)
12 3 4 5123
4 5 6
123456
7 partition 0
partition 1
partition 2
Co
nsu
me
rs
1
2
3456
7
1
2
3
4 5 6
1
2
3 4 5
consumer group 2
consumer group 1
C3
C2
C1
C4
parallel consumption
Tuesday, December 3, 13
![Page 27: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/27.jpg)
123 4 512345
6
123456
7 partition 0
partition 1
partition 2
Co
nsu
me
rs
1
2
3456
7
1
2
345
6
1
23
4 5
consumer group 2
consumer group 1
C3
C2
C1
C4
parallel consumption
Tuesday, December 3, 13
![Page 28: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/28.jpg)
123451234561234567
partition 0
partition 1
partition 2
Co
nsu
me
rs
1
2
3456
71
2
345
61
2345
consumer group 2
consumer group 1
C3
C2
C1
C4
Tuesday, December 3, 13
![Page 29: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/29.jpg)
A1A2A3A4
A5
A6A7
server A
Replication
server Bserver C
B1B2B3B4
B5B6C1C2C3C4
C5
P2
P1
A master(all reads and writes)
B master(all reads and writes)
C master(all reads and writes)
Tuesday, December 3, 13
![Page 30: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/30.jpg)
A1
A2A3A4
A5
A6A7
server A
Replication
server Bserver C
B1B2B3B4
B5B6C1C2C3C4
C5
P2
P1
A master(all reads and writes)
B master(all reads and writes)
C master(all reads and writes)
Tuesday, December 3, 13
![Page 31: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/31.jpg)
A1
A2A3A4
A5
A6A7
server A
Replication
server Bserver C
B1B2B3B4
B5B6C1C2C3C4
C5
P2A1
P1
A master(all reads and writes)
B master(all reads and writes)
C master(all reads and writes)
Tuesday, December 3, 13
![Page 32: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/32.jpg)
A1 A2
A3A4
A5
A6A7
server A
Replication
server Bserver C
B1B2B3B4
B5B6C1C2C3C4
C5
P2A1 A2
P1
A master(all reads and writes)
B master(all reads and writes)
C master(all reads and writes)
Tuesday, December 3, 13
![Page 33: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/33.jpg)
A1 A2
A3A4
A5
A6A7
server A
Replication
server Bserver C
B1B2B3B4
B5B6
C1
C2C3C4
C5
P2A1 A2
P1C1
A master(all reads and writes)
B master(all reads and writes)
C master(all reads and writes)
Tuesday, December 3, 13
![Page 34: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/34.jpg)
A1 A2 A3 A4
A5
A6A7
server A
Replication
server Bserver C
B1 B2 B3 B4 B5 B6
C1 C2 C3 C4 C5
P2A1 A2 A3 A4
P1C1 C2 C3 C4 C5
B1 B2 B3 B4 B5 B6
A master(all reads and writes)
B master(all reads and writes)
C master(all reads and writes)
Tuesday, December 3, 13
![Page 35: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/35.jpg)
A1 A2 A3 A4 A5 A6 A7
server A
Replication
server Bserver C
B1 B2 B3 B4 B5 B6
C1 C2 C3 C4 C5
P2A1 A2 A3 A4
A5 A6 A7
P1C1 C2 C3 C4 C5
B1 B2 B3 B4 B5 B6
A master(all reads and writes)
B master(all reads and writes)
C master(all reads and writes)
Tuesday, December 3, 13
![Page 36: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/36.jpg)
A1 A2 A3 A4 A5 A6 A7
server A
Replication
server Bserver C
B1 B2 B3 B4 B5 B6
C1 C2 C3 C4 C5
P2A1 A2 A3 A4 A5 A6 A7
P1C1 C2 C3 C4 C5
B1 B2 B3 B4 B5 B6
A master(all reads and writes)
B master(all reads and writes)
C master(all reads and writes)
Tuesday, December 3, 13
![Page 37: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/37.jpg)
123 Messages in a partition are assigned a sequential id number, referred to as the offset.
Consumers keep track of their consumed offset in the partition. Kafka doesn’t maintain any metadata.
Kafka maintains all messages for a configurable time, regardless of whether they’ve been consumed.
Tuesday, December 3, 13
![Page 38: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/38.jpg)
Alternativeshttp://queues.io
Tuesday, December 3, 13
![Page 39: Apache Kafka Lightning Talk](https://reader033.vdocuments.site/reader033/viewer/2022061221/54bd19c54a795932448b4697/html5/thumbnails/39.jpg)
Questions?
Tuesday, December 3, 13