analyse en temps réel de bigdata par l'exemple
TRANSCRIPT
![Page 1: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/1.jpg)
L’analyse en temps réel de Big Data Le monitoring de flux par l’exemple
#DevoxxMA #BigData @a_bouchama
Abdellatif BOUCHAMA
![Page 2: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/2.jpg)
Abdellatif BOUCHAMA
@a_bouchama
• Middleware architect, and passionate about the new technology: Big Data,IoT and contributor in the Open source
• Co-founder of BusHorn.com
@a_bouchama#DevoxxMA #BigData
![Page 3: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/3.jpg)
Agenda
Big Data in 2015: Trends & Statistics
The emergence of real time
Use Case: Flow Activity Monitoring
Demo
@a_bouchama#DevoxxMA #BigData
![Page 4: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/4.jpg)
What is Big Data?
Big Data
Velocity 30KB/s --> 30GB/s· Batch · Real-Time · Streaming
Volume
1-2 Terabytes à ∞
Variety
· Structured · Semi-Structured · UnStructured
![Page 5: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/5.jpg)
Big Data Expectations:Things That Excite Executives About Big Data
It’s the “next oil.” from the Ginni Rometty, CEO of IBM
The potential to revolutionize industries, and change business models (e.g., Uber and Airbnb), with what we learn from the data.
The ability to provide real-time graphing solutions for data relationships.
Real-time operational and business data allows people to make decisions faster thereby saving significant money.
The opportunities it gives us to help clients solve real business problems.
https://dzone.com/articles/14-things-that-excite-executives-about-big-data?oid=big_data
![Page 6: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/6.jpg)
Big Data view by Gartner:
@a_bouchama#DevoxxMA #BigData
![Page 7: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/7.jpg)
Big Data in 2015:
75% of Companies Are Investing or Planning to Invest in Big Data in the Next Two Years
Goals for Big Data initiatives:1. Enhancing the customer experience2.Streamlining existing processes3.Achieving more targeted marketing and reducing costs
Last year, 37% of big data projects were initiated by the CIO, while 25% were initiated by business unit heads. In 2015, the roles are nearly tied, at 32% and 31 %, respectively.
http://www.gartner.com/newsroom/id/3130817@a_bouchama#DevoxxMA #BigData
![Page 8: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/8.jpg)
The emergence of Real-Time
Low
Pure BatchOperationalizing
Near Real TimeOr InteractiveAnalytics
High
Real Time Analytics
• Right time
Real time
• Smart data
Big Data
« It is the right time for Real Time to use Big Data as a Smart Data »
![Page 9: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/9.jpg)
Use case: Flow Activity Monitoring
@a_bouchama#DevoxxMA #BigData
![Page 10: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/10.jpg)
ESB
ESB ESB
![Page 11: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/11.jpg)
ObjectivesMonitor all data flows which goes through the different layers (transport, mediation, ..etc).
Analyze and centralize data
Perform the correlation between the functional and technical data
Provide a centralized view in real time.
DealsIntegrate all data flows, analyze them and correlate them quickly.
Manage a very high data rate, near millions of events
Manage a number of messages per second
@a_bouchama#DevoxxMA #BigData
![Page 12: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/12.jpg)
Constraints & RequirementsConstraints• Non intrusive system• No modification on business flows.• We can plug it and unplug it easily.Requirements• System cost should be mastered
and adaptable• System automatized• Measurement
@a_bouchama#DevoxxMA #BigData
![Page 13: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/13.jpg)
Architecture
CHESBUS
ESB
FRESB
ElasticSearch
![Page 14: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/14.jpg)
Why Elasticsearch stack?Open source
Easy to deploy
Distributed
Linear scalability
REST Interface
Kibana & Logstash to complete the Landscape
@a_bouchama#DevoxxMA #BigData
![Page 15: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/15.jpg)
Data flow and constraints
Collect
JMX
Store & transport
Transform & Access
Modelize & Analyze
Visualize & Predict
JMS
![Page 16: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/16.jpg)
Demo Time!Be prepared for it to fail, because demos always do
@a_bouchama#DevoxxMA #BigData
![Page 17: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/17.jpg)
What’s Next ?• Enrichment analysis, after adding
business information in the logs
• Integration of backend and frontend applications
• Audit
@a_bouchama#DevoxxMA #BigData
![Page 18: Analyse en temps réel de BigData par l'exemple](https://reader036.vdocuments.site/reader036/viewer/2022062311/58ed50d41a28abf0508b4657/html5/thumbnails/18.jpg)
THANKS FOR LISTENING
Any Questions?
#DevoxxMA #BigData @a_bouchama