Download - Real time analytics with Netty, Storm, Kafka
![Page 1: Real time analytics with Netty, Storm, Kafka](https://reader033.vdocuments.site/reader033/viewer/2022050903/540d59d18d7f72767e8b492b/html5/thumbnails/1.jpg)
Real-time Analytics with Netty, Apache Kafka and StormCase study with “lambda architecture”
http://nguyentantrieu.infoUpdate: 07/06/2013
![Page 2: Real time analytics with Netty, Storm, Kafka](https://reader033.vdocuments.site/reader033/viewer/2022050903/540d59d18d7f72767e8b492b/html5/thumbnails/2.jpg)
Agenda1. Overview Architecture2. Log HTTP-Handler and producer: Netty 43. Kafka 0.8 (Stream Data Log Storage)4. Storm Analytics Cluster
![Page 3: Real time analytics with Netty, Storm, Kafka](https://reader033.vdocuments.site/reader033/viewer/2022050903/540d59d18d7f72767e8b492b/html5/thumbnails/3.jpg)
Overview System Architecture
![Page 4: Real time analytics with Netty, Storm, Kafka](https://reader033.vdocuments.site/reader033/viewer/2022050903/540d59d18d7f72767e8b492b/html5/thumbnails/4.jpg)
Concept Flow
![Page 5: Real time analytics with Netty, Storm, Kafka](https://reader033.vdocuments.site/reader033/viewer/2022050903/540d59d18d7f72767e8b492b/html5/thumbnails/5.jpg)
Concept FlowJavaScript Tracking
Mobile SDK
Http Log Server
Kafka
![Page 6: Real time analytics with Netty, Storm, Kafka](https://reader033.vdocuments.site/reader033/viewer/2022050903/540d59d18d7f72767e8b492b/html5/thumbnails/6.jpg)
S2 HTTP Log Servernetty framework 4
![Page 7: Real time analytics with Netty, Storm, Kafka](https://reader033.vdocuments.site/reader033/viewer/2022050903/540d59d18d7f72767e8b492b/html5/thumbnails/7.jpg)
Netty.ioNetty is a non-blocking I/O (NIO) client-server framework for the development of Java network applications such as protocol servers and clients. The asynchronous event-driven network application framework and tools are used to simplify network programming such as TCP and UDP socket servers.[2] Netty includes an implementation of the reactor pattern of programming.
http://en.wikipedia.org/wiki/Netty_(software)http://nguyentantrieu.info/blog/backend-system-with-netty-io
![Page 8: Real time analytics with Netty, Storm, Kafka](https://reader033.vdocuments.site/reader033/viewer/2022050903/540d59d18d7f72767e8b492b/html5/thumbnails/8.jpg)
Apache Kafka (version 0.8)https://cwiki.apache.org/confluence/display/KAFKA/Index
![Page 9: Real time analytics with Netty, Storm, Kafka](https://reader033.vdocuments.site/reader033/viewer/2022050903/540d59d18d7f72767e8b492b/html5/thumbnails/9.jpg)
In Production● Clustering ( 4 nodes)● Partitions
○ user-activity: 24 partitions● Producer and consumer are replication
aware
![Page 10: Real time analytics with Netty, Storm, Kafka](https://reader033.vdocuments.site/reader033/viewer/2022050903/540d59d18d7f72767e8b492b/html5/thumbnails/10.jpg)
Core Producer Class (S2 HTTP Log Server)
![Page 11: Real time analytics with Netty, Storm, Kafka](https://reader033.vdocuments.site/reader033/viewer/2022050903/540d59d18d7f72767e8b492b/html5/thumbnails/11.jpg)
Storm Analytics Cluster
![Page 12: Real time analytics with Netty, Storm, Kafka](https://reader033.vdocuments.site/reader033/viewer/2022050903/540d59d18d7f72767e8b492b/html5/thumbnails/12.jpg)
The Storm Topology
Kafka Clustertopic: user-activity
Tokenizer Bolt
Parser Bolt
Aggregate Bolt
Redis Statistics Bolt
Save DWH Bolt
Raw Data
Kafka Consumer Spout
![Page 13: Real time analytics with Netty, Storm, Kafka](https://reader033.vdocuments.site/reader033/viewer/2022050903/540d59d18d7f72767e8b492b/html5/thumbnails/13.jpg)
Useful Links● https://storm.incubator.apache.org● https://www.youtube.com/watch?
v=DRgs8L4AOgc● https://cwiki.apache.
org/confluence/display/KAFKA/Index● http://nguyentantrieu.info/blog/backend-
system-with-netty-io●