flink community update december 2015: year in review
TRANSCRIPT
Berlin Apache Flink Meetup #12Community Update & 2015 – The Year in Review
December 2015
Robert MetzgerCommitter and PMC
@rmetzger_
2
What happened?• 0.10.0 and 0.10.1 released– Event-time streaming (out of order streams,
sessions)– Master HA no single point of failure– DataStream API graduated– New monitoring interface– Offheap memory– Outer joins– Gelly Scala API
• Otto released a Flink testing framework: https://github.com/ottogroup/flink-spector
flink.apache.org
3
Now in master (1.0-SNAPSHOT)
flink.apache.org
• Kafka Connector improvements: Multiple topics, keys, deleted messages, periodic offset commit
• Flink in firewalled setups• Out of core state backend for JDBC• Major improvements in the flink-storm
module• RabbitMQ Source with exactly-once• Many other operational and usability
improvements
4
Pending Pull Requests• Savepoints: Trigger state checkpoints
manually and restore a job from a savepoint
• Submit Flink jobs through JobManager web interface
• Add range partitioner with sampling (FLINK-7)
• “stop” signal for streaming topologies• Monitoring: JMX beans & checkpoint metrics
flink.apache.org
5
Reading List• Realtime dashboards with Flink and ElasticSearch
https://www.elastic.co/blog/building-real-time-dashboard-applications-with-apache-flink-elasticsearch-and-kibana
• Introduction to streaming windowshttp://flink.apache.org/news/2015/12/04/Introducing-windows.html
• Flink + Storm: Run Storm topologies on Flinkhttp://flink.apache.org/news/2015/12/11/storm-compatibility.html
• New streaming use caseshttp://data-artisans.com/how-apache-flink-enables-new-streaming-applications-part-1/
• Streaming with Flink Intro (reading twitter data) http://blog.brakmic.com/stream-processing-with-apache-flink/
flink.apache.org
6
Flink 2015: The journey to mainstream
2015 – The Year in Review
flink.apache.org
7
Let the numbers speak: GitHub
flink.apache.org
280%+
Stars 340%+
Forks
February 2015
December 2015
8
Let the numbers speak: Meetup Groups
flink.apache.org
From one Meetup group to a global community
9
Let the facts speak:Organizations at Flink Forward 2015
flink.apache.org 9
Let the facts speak:Articles about Flink featured in
10
Let the facts speak:Hadoop Summit 2016: 17 Flink talks
11
12flink.apache.org
• Check out the slides: http://flink-forward.org/?post_type=session
• Video recordings on YouTube, “Flink Forward” channel
Apropos
13
Feature Timeline 2015• January
– Flink 0.8• February
– Gelly– Table API
• March– Flink-Google Data
Flow Runner (batch)– FlinkML
• April– Python API
• May– Static Code Analysis– Zeppelin Integration
• June– 0.9: Akka, batch
exchange, exactly-once
• July– New dashboard– first HA features
flink.apache.org
14
Feature Timeline 2015• August
– Gelly Scala API– DataSet.sample()
• September– Cascading on Flink– Scala 2.11 (and 2.10)– Offheap memory– Reworked windowing– Rolling file sink– Elasticsearch sink– High Availability
flink.apache.org
• October– Reworked Kafka
Consumer– Streaming API
graduation• November– Flink 0.10– JDBC backend
• December– Flink behind firewalls
15
git Stats 2015git log --pretty=oneline --after=1/1/2015 | wc -l
2201 commits in 2015
flink.apache.org
Lines of code Commits per day
Commits per hour
16
We are not done yet• Flink Forward 2016• Roadmap 2016 (pending community discussion)
– Autoscaling– Stream SQL–Managed Memory State… much more
Let’s make 2016 the Year of Flink!flink.apache.org