robust stream processing with apache flink
TRANSCRIPT
![Page 2: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/2.jpg)
Who am I?• Director of Applications Engineering at data
Artisans• Previously working on streaming computation
at Twitter, Gnip and Boulder Imaging• Involved in various kinds of stream
processing for about a decade• High-speed video, social media streaming,
general frameworks for stream processing
![Page 3: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/3.jpg)
Overview• What is Apache Flink?• What is Stateful Stream Processing?• Windowed computation over streams• Robust Time Handling (Event Time vs Processing
Time)• Robust Failure Handling• Robust Planned Downtime Handling• Robust Reprocessing
![Page 4: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/4.jpg)
What isApache Flink?
Apache Flink is an open source platform for distributed stream and batch data processing.
![Page 5: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/5.jpg)
What isApache Flink?
![Page 6: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/6.jpg)
Stream Processing
Your Code
Data Stream Data Stream
![Page 7: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/7.jpg)
StatefulStream Processing
Your Code
Data Stream Data Stream
State
![Page 8: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/8.jpg)
More ComplexExample
RabbitMQ
Files
Kafka
Filter
Map
Join / Sum
InfluxDB
C*
![Page 9: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/9.jpg)
Distributed and Parallel Deployment
MapR Stream
s
Files
Kafka
Filter
Parse
Join / Sum
InfluxDB
C*
![Page 10: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/10.jpg)
Benchmarking onHPC Cluster
Se-ries1
10 Machines with 40 GigE
Throughput: msgs/sec
72 Million msgs/sec
![Page 11: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/11.jpg)
Robust Stream Processingwith Apache Flink
![Page 12: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/12.jpg)
Code Example!
![Page 13: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/13.jpg)
Amplifier Function
Amplifier
Control StreamAmplified Stream
State*
Data Stream
*State: Amplification factors for each key
![Page 14: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/14.jpg)
Windowing
![Page 15: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/15.jpg)
Processing Timevs
Event Time
![Page 16: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/16.jpg)
Windowing in Processing Time
0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9
Processing Time
Event Time
![Page 17: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/17.jpg)
Windowing in EventTime
0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9
Event Time
![Page 18: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/18.jpg)
Processing Time = Errors!
![Page 19: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/19.jpg)
Event Time = Accuracy
![Page 20: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/20.jpg)
Failure Handling
![Page 21: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/21.jpg)
Downtime Handling
![Page 22: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/22.jpg)
Data Reprocessing
![Page 24: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/24.jpg)
Questions?
![Page 25: Robust Stream Processing With Apache Flink](https://reader035.vdocuments.site/reader035/viewer/2022062503/5880023e1a28ab3a1e8b7d9b/html5/thumbnails/25.jpg)
Thanks!