big data using public cloud
TRANSCRIPT
Big Data Using Public Cloud
Assoc. Prof. Dr. Thanachart NumnondaExecutive DirectorIMC Institute18 August 2015
2
Internet of ThingsCloud Computing
Big Data
3
Data created every minute
Source: Trendwise Analytics
4
Data!
The New York Stock Exchange generate about1TB of new trade data per day.
A commercial aircraft generates 3GB of flightsensor data in 1 hour.
Vodafone generates 3TB of Call Detail Record(CRDs) per day.
Between 2009 and 2014, the total number of U.S.online banking households will increase from 54million to 66 million.
5
Big Data
Source: http://www.datasciencecentral.com/
6
7
Sample Data v.s. Big Data
Can you judge a persons life expectancy?
Given:
– DNA
– Medical records
– Food
– Lifestyle (smoking, drinking, driving, exercise)
8
IT Infrastructure
Analytics
Data Sources
9
“B ัy 2015, 20% of Global 1000 organizationsWill have established a strategic focus on
information infrastructure ”
Gartner
10
Big Data Technology !!
11
Big Data Landscape
Source: Big Data in the Enterprise. When to Use What?
12
A scalable fault-tolerant distributed system for data storage and processing
Completely written in javaOpen source & distributed under Apache license
What is Hadoop?
13
Hadoop Environment
Source: Hadoop in Practice; Alex Holmes
14
Hadoop Distribution
Microsoft Azure
15
Big Data Future Architecture
Sscial Media Images e-mails Crawlers ERP CRM LOB APPs
Unstructured and Structured Data
Data Warehouse / NewSQL
Hadoop OnCloud
Hadoop OnPrivateServer
Connectors
SSRS
BI Platform
Familiar End User ToolsSpreadsheet Predictive Analytics
Data Market Place
NoSQL
Petabytes of Data(Unstructured)
Hundreds of TB of Data(structured)
16
Issue with Big Data Infrastructure
Large investment
Scalabilty
ROI
Business Cases
17
Big Data on Cloud
Using IaaS to leverage Cloud Vms
Using Big Data as a Services
18
Big Data using IaaS
19
20
21Source: http://www.datadansandler.com/
22
Big Data Services on Cloud
Amazon Elastic Mapreduce
Microsoft Azure Hadoop
23
Big Data as a Service
24
Database as a Service
Amazon RDS
IBM SQL Database for Bluemix
Microsoft SQL Database
Google CloudSQL
25
NoSQL as a Service
Amazon DynomoDB
Google Cloud DataStore
Microsoft Azure DocumentDB
Cloudant on IBM Bluemix.
Mongo DB on Heroku
26
Amazon DynomoDB
27
Hadoop as a Service
Amazon Elastic Map Reduce
Rackspace Cloud Big Data Platform
Qubole
Google Cloud Platform
IBM Bluemix: Analytic on Hadoop
Microsoft Azure HDInsight
28
29
30
Big Data on Amazon EMR
31
Amazon EMR
32
33
34
Hadoop on Google
35
36
Analytic as a Service
Google Big Query
Amazon Machine Learning
Azure Machine Learning
BIME: BI as a Service
IBM Watson Analytic
37
38
39
Google BigQuery
40
Big Data on Cloud Roadmap
Step 1: Build the business case
Step 2: Assess your Big Data applicationworkloads
Step 3: Develop a technical approach fordeploying and managing Big Data in the cloud
Step 4: Address governance, security, privacy,risk,
Step 5: Deploy, integrate, and operationalizeyour cloud-based Big Data infrastructure
Source : Deploying Big Data Analytics Applications to the Cloud: Roadmap for Success: CSCS
41
Sample applications
Enterprise applications already hosted in thecloud
High-volume external data sources thatrequire considerable preprocessing
Tactical applications beyond your on-premises, Big Data capabilities
Elastic provisioning of very large but short-lived analytic sandboxes
Source : Deploying Big Data Analytics Applications to the Cloud: Roadmap for Success: CSCS
42
www.facebook.com/imcinstitute
