متنبازسازی کلانداده
TRANSCRIPT
![Page 1: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/1.jpg)
![Page 2: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/2.jpg)
Open Sourcing Big Data
Hadi
Sotudeh
![Page 4: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/4.jpg)
About Us
Dr. Sharif
Big Data: From a Business & Managerial Perspective
![Page 5: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/5.jpg)
Bigdata.blog.ir
![Page 6: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/6.jpg)
About Us
Torob.ir Co-Founder : Ali Babei
![Page 7: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/7.jpg)
About Us
B.S Project : (DRPC)Distributed Real Time Processing Crawler using Apache Storm
Dr. Goudarzi
![Page 8: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/8.jpg)
![Page 9: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/9.jpg)
Everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it.
Dan Ariely
![Page 10: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/10.jpg)
News
![Page 11: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/11.jpg)
Big Data Definition
Is there any standard definition?
![Page 12: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/12.jpg)
Big Data Definitions Gartner Mckinsey ….
![Page 13: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/13.jpg)
Gartner Big data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.
![Page 14: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/14.jpg)
Mckinsey datasets whose size is beyond the ability of
typical database software tools to capture,
store, manage, and analyze
![Page 15: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/15.jpg)
![Page 16: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/16.jpg)
![Page 17: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/17.jpg)
• Sensors
• Transactions
• GPS
• Social Network
• Sound Files
• Video
• Image
• Telescope
• Log
• Tex
• ....
Data Sources
![Page 18: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/18.jpg)
Tim Berners Lee
Open Data Movement
![Page 19: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/19.jpg)
Open Data:
19
State/Org Website
UAE http://government.ae/web/guest/uae-data
UK http://data.gov.uk
US http://data.gov
World Bank http://data.worldbank.org/
India http://data.gov.in
Russia http://opengovdata.ru
EU Open-data.Europa.eu/en/data
• Google.com/trends/explore
• Google.com/finance
![Page 20: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/20.jpg)
20
![Page 21: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/21.jpg)
![Page 22: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/22.jpg)
![Page 23: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/23.jpg)
Close Data!
23
![Page 24: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/24.jpg)
شبکه های اجتماعی
24
![Page 25: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/25.jpg)
A Tweet
![Page 26: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/26.jpg)
![Page 27: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/27.jpg)
![Page 28: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/28.jpg)
![Page 29: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/29.jpg)
![Page 30: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/30.jpg)
![Page 31: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/31.jpg)
Edward Snowden
![Page 32: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/32.jpg)
![Page 33: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/33.jpg)
NSA
![Page 34: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/34.jpg)
Log or Dark Data
34
![Page 35: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/35.jpg)
35
![Page 36: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/36.jpg)
Importance
![Page 37: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/37.jpg)
![Page 38: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/38.jpg)
Analytics is the discovery and communication ofmeaningful patterns in data
Analytics
![Page 39: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/39.jpg)
Types of Analytics Cube Analytics Multi Dimensional Product Date Price
BI Predictive Analytics Statistics and Machine Learning Linear Regression Data Clustering Find Association
![Page 40: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/40.jpg)
Dimensions of Analytics Variants
Real Time Ability to Analyze the data instantly
Batch Ability to provide insights after several
hours/days when a query is posted
![Page 41: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/41.jpg)
TOOLS
![Page 42: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/42.jpg)
![Page 43: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/43.jpg)
Do It
Real Time
![Page 44: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/44.jpg)
![Page 45: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/45.jpg)
Problems
Scaling is painfulPoor fault-tolerance
Coding is tedious
![Page 46: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/46.jpg)
What We Want
Guaranteed Data ProcessingHorizontal scalabilityFault-tolerance“just works”
![Page 47: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/47.jpg)
What Is The Key?
![Page 48: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/48.jpg)
Hadoop
Batch Oriented System
![Page 49: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/49.jpg)
![Page 50: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/50.jpg)
![Page 51: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/51.jpg)
Storm
Guaranteed Data ProcessingHorizontal scalabilityFault-tolerance“just works”
![Page 52: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/52.jpg)
Use cases
![Page 53: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/53.jpg)
![Page 54: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/54.jpg)
Streams
![Page 55: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/55.jpg)
Spouts
![Page 56: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/56.jpg)
Bolts
![Page 57: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/57.jpg)
Topology
![Page 58: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/58.jpg)
Word Count
![Page 59: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/59.jpg)
Tuple Tree
![Page 60: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/60.jpg)
![Page 61: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/61.jpg)
Resources
Book Apache Storm website
![Page 62: متنبازسازی کلانداده](https://reader031.vdocuments.site/reader031/viewer/2022030312/58ed96ce1a28abb4408b464f/html5/thumbnails/62.jpg)
Conclusion
• Data, Data, and Data
• Data Gathering
• Analytics
• Visualization
• Action
• Bottleneck is Creativity not Technology
• Discover Use Cases