hadoop, oracle and the industrial revolution of data

© 2012 Quest Software Inc. All rights reserved.

Hadoop, Oracle and the industrial revolution of data

Guy HarrisonVP R&D, Database Management

Hadoop, Oracle and the industrial revolution of data

Guy HarrisonExecutive Director, R&D Business Intelligence Software

Pg. 3© 2012 Quest Software Inc. All rights reserved.

Introductions

www.guyharrison.net [email protected]

http://twitter.com/guyharrison

http://www.guyharrison.net/

http://www.guyharrison.net/

mailto:[email protected]

http://twitter.com/guyharrison


Quest


Blue

Yellow

Red

0 10 20 30 40 50 60 70 80

Star trek shirt fatality analysis

Pct


What is Big Data?


The 3-4 V’s

VolumeTerabytesPetabytesExabytesZetabytes

VarietyStructuredUnstructuredHuman GeneratedMachine Generated

VelocityUser populations xTransaction rates xMachine data

Value Competitive or Community advantage


Volume Data volumes have always been increasing

2006 Perspective


But the vastness is becoming mind boggling

Human Brain

Google

Living Human Genomes

Digital information 2008

Total Digital capacity

Digital information created 2011

1.00E+09 1.00E+11 1.00E+13 1.00E+15 1.00E+17 1.00E+19 1.00E+21 1.00E+23

2.81E+15

1.10E+17

5.48E+18

4.87E+18

1.18E+21

2.13E+21

Gigabyte Terabyte Petabyte Exabyte zettabyte


Velocity


Fail whales


The Industrial Revolution of Data

Variety


Big Data is driven by the smallest devices


Samsung Galaxy S IIII specifications

Quad-core 1.4 GHz CPU

1GB RAM

64GB Storage

1080p display

GSM/Bluetooth/WiFi Network

8MP Camera

GPS & Compass

35

Name: Willy Bowman

Nationality: German

DON’T MENTION THE WAR


Data Input

From now on, I’ll call you ‘An Ambulance’. OK?

“Siri call me an ambulance”

I found 14 bridges nearby:

“I want to jump off a bridge”

Siri


Brain Control


All of this requires and Generates Big Datasets

But what are they good for?


Value?

Achieve competitive advantage

From Big Data using

Collective Intelligence,

Machine Learning

and Predictive Analytics

Machine LearningPrograms that evolve with “experience”

Collective IntelligencePrograms that use inputs from “crowds’ to seem intelligent

Predictive AnalyticsPrograms that extrapolate from existing data into the future

Big Data AnalyticsHow do we derive value from the data?


Applications

Collective Intelligence

Search Optimization

Recommendation Systems

Security•Vulnerability•Penetration Detection

Fraud Detection

Predictive Analytics•Churn •Defaults

Medical•Risk analysis•Diagnosis•Prognosis

Game optimization

Advertising•Targeting•Tailoring


Collective Intelligence beats Artificial Intelligence

?


For the past 40 years, AI has been consistently disappointing


Google: pioneers of big data


Google File System (GFS)

Map Reduce BigTableChubby

Google Applications

Google Software Architecture


START REDUCEMAPMAP

MAPMAP

MAPMAP

MAPMAP

MAPMAP

MAPMAP

MAP

MAPMAP

MAPMAP

MAPMAP

MAPMAP

MAPMAP

MAPMAP

MAPMAP

MAPMAP

MAPMAP

MAPMAP

MAPMAP

Map Reduce


HDFS

MAPPER

MAPPER

MAPPER

MAPPER

MAPPER

MAPPER

MAPPER

MAPPER

SCANSORT

MAPPER

MAPPER

MAPPER

MAPPER

AGGREGATE

REDUCECLIENT

Multi-stage Map-Reduce


Hadoop: Open Source Map-Reduce Stack


Hadoop at Yahoo!

Yahoo! Hadoop cluster:− 4000 nodes− 16PB disk− 64 TB of RAM− 32,000 Cores


MAP REDUCE (DISTRIBUTED PROCESSING)

HADOOP CLIENT (JAVA, PIG, HIVE)

HDFS (DISTRIBUTED

STORAGE)

JOB TRACKER

DATA NODE TASK TRACKER




NAME NODE





SECONDARY NAME NODE








Hadoop Architecture(1.0)


Schema on Read vs Schema on Write


Data

Analyse

Aggregate

Normalize

Cleanse

Code

Extract Load TransformData Warehouse

Utilize

Data LoadHadoop

Analyse

Cleanse

Code

Utilize

Schema on Write

Schema on Read


Hadoop Ecosystem

Hadoop File System (HDFS)

Hadoop Map ReduceHbase

(Database)ZooKeeper(Locking)

SQOOP(RDBMS loader)

Hive(Query)

Pig(Scripting)

Flume(Log Loader)

Oozie (Workflow manager)


HBase


HBase is a real-time database built on Hadoop

HBase

ASM

Datafiles

Buffer Cache

Table Table

Redo

Disks

LogBuffer

HDFS

HFile

MemStore

Table Table

WA Log

Disks

HFile

Name Site Counter

Dick Ebay 507,018

Dick Google 690,414

Jane Google 716,426

Dick Facebook 723,649

Jane Facebook 643,261

Jane ILoveLarry.com 856,767

Dick MadBillFans.com 675,230

NameId Name

1 Dick

2 Jane

SiteId SiteName

1 Ebay

2 Google

3 Facebook

4 ILoveLarry.com

5 MadBillFans.com

NameId SiteId Counter

1 1 507,018

1 3 690,414

2 3 716,426

1 3 723,649

2 3 643,261

2 4 856,767

1 5 675,230

Id Name Ebay Google Facebook (other columns) MadBillFans.com

1 Dick 507,018 690,414 723,649 . . . . . . . . . . . . . . 675,230

Id Name Google Facebook (other columns) ILoveLarry.com

2 Jane 716,426 643,261 . . . . . . . . . . . . . . 856,767

Hbase Data Model


Hive


SQL

JAVA

Resu

lts


Pig


Pig Latin

SQL or Hive QL


Meanwhile, back at the Death Star….


Oracle Exadata

Database servers64 cores, 576 GB

RAM

Storage Servers112 cores, 100 TB SAS or336 TB SATA plus5 TB SSD


Exadata

Hadoop

$0 $1,000 $2,000 $3,000 $4,000 $5,000 $6,000

$4,911

$750

Exadata vs Hadoop $$/TB (Hardware only)

Economies


18 Sun X4270 M2 servers− 48GB RAM per node (864GB total)− 2x6 Core CPU per node (216 total)− 12x2TB HDD per node (216 spindles,

864 TB)− 40Gb/s Infiniband between nodes− 10Gb/s Ethernet to datacentre

Competitive Pricing

www.oracle.com/us/bigdata/index.html

Oracle Big Data Appliance

http://www.oracle.com/us/bigdata/index.html






Big Data Appliance Software

Cloudera Enterprise

Oracle Enterprise R

Oracle NoSQL

Oracle Big Data Connectors


Oracle’s Storage Hierarchy

ORACLEEXADATA

ORACLEEXALOGIC

ORACLEBIG DATA

APPLIANCE

ORACLE NOSQL

ORACLE LOADER FOR HADOOP

APACHEHADOOP ORACLE

RDBMS

ORACLE WEBLOGIC

ORACLE EXALYTICS

ORACLE ESSBASE

ORACLE TIMES TEN

Latency

Storage Costs


111


Hadoop and RDBMS integration


Scenario #1: Reference data in RDBMS

CUSTOMERS

WEBlOGS

PRODUCTS

HDFS

RDBMS


Scenario #2: Hadoop for off-line analytics

CUSTOMERS

PRODUCTS

RDBMS

SALESHISTORY

HDFS


Scenario #3: MapReduce output to RDBMS

WEBLOGSSUMMARY

RDBMS

DB QUERYTOOL

WEBLOGS

HDFS


Scenario #4: Hadoop as RDBMS “active archive”

SALES 2011

HDFS

RDBMS

QUERYTOOL

SALES 2010

SALES 2009

SALES 2008

SALES 2009

SALES 2008


The Big Data Stack

The Big Data Stack

HDFS

MAP-REDUCE HBASE

PIG

CASCADING

MAHOUT

JAVA APIHIVE

R (ET AL)

JAVA API

DATA SCIENTIST

The Big Data Stack

HDFS

MAP-REDUCE HBASE

PIG

CASCADING

MAHOUT

JAVA API HIVE

R (ET AL)

JAVA API

DATA SCIENTISTBIG DATA ANALAYTIC PLATFORM

Big Data Analytics Platform

BIG DATA ANALYTICS

INDEXING AND SEARCH

VISUALIZATION

RECOMMENDERS

CLUSTERING

CLASSIFICATION

EXPERT SYSTEMS (LIKE WATSON)

OPTIMIZATION

ADVERTISING

BASKET ANALYSIS

SENTIMENT ANALYSIS


In Summary


Hadoop is….


Exadata

Hadoop

$0 $1,000 $2,000 $3,000 $4,000 $5,000 $6,000

$4,911

$750

Exadata vs Hadoop $$/TB (Hardware only)

Economical


Scalable

• 4000 nodes at Yahoo!• >100 PB at Facebook• 10,000 node design

goal for Hadoop 2.0


A platform for AI, CI & analytics


ETL “Free”

Data

Analyse

Aggregate

Normalize

Cleanse

Code

Extract Load TransformData Warehouse

Utilize

Data LoadHadoop

Analyse

Cleanse

Code

Utilize

Schema on Write

Schema on Read


The most concrete technology enabling the Big Data revolution


Hadoop is not….


But future Enterprise Data Architectures will likely incorporate Hadoop side by side with RDBMS

A replacement for RDBMS


Though OLTP systems can be built with Hadoop-compatible NoSQL systems such as HBase and Cassandra

Suitable for OLTP


Hadoop alone only solves the storage challenge of Big Data

A complete solution


Shameless plugs


Toad for Cloud Databases

Work with Hive, Hbase, Oracle, SQL Server, Cassandra, MySQL, MongoDB, BI servers and other NoSQL and SQL datastores


Toad for Cloud Databases• Federated SQL queries across Hive, Hbase, NoSQL, RDBMS

Toad for Cloud Databases


0 5 10 15 20 25 30 350

1,000

2,000

3,000

4,000

5,000

6,000

7,000

50M row, 50GB Oracle table to 16-node Hadoop clusterSQOOP

SQOOP with Quest Connector

Number of mappers

Ela

pse

d T

ime

(ms)

Quest Connector for Oracle and Hadoop

Hi-speed, bi-directional data transfer between Hadoop, Hive and Oracle


Business Intelligence solutions with first class support for Hadoop, Oracle and many other platforms

Toad BI Suite


Redo-logs

Change Data Capture

JMS Queue Hadoop Poster

BatchedHDFS File Copy

Audit / Change Data

HBase RealTime replication

SharePlex® for Hadoop


• Hive Query IDE

• Oracle <-> Hadoop data management

• Basic Hadoop administration

• ETA beta H1 2013

Toad for Hadoop


Summary:

The future belongs to those of us prepared to wear funny hats and glasses

The connected and mobile internet requires and produces “big data” that is qualitatively different from the data we’ve had before− Requiring different types of datastores

Enterprise can leverage big data for competitive advantage− Requiring different types of analytical engines

hadoop, oracle and the industrial revolution of data

Documents

quest software

big data

data input

volume data volumes

gb storage

gb ram

human genomes5

human brain