Treasure Data Cloud Strategy

Masahiro Nakagawa

July Tech Festa: Jul 14, 2013

Sunday, July 14, 13

Who are you?§ Masahiro Nakagawa

• @repeatedly / masa@treasure-data.com

§ Treasure Data, Inc.• Senior Software Engineer, since 2012/11

§ Open Source projects• D Programming Language• MessagePack: D, Python, etc...• Fluentd: Core, Mongo, Logger, etc...• etc...

2

Sunday, July 14, 13

Treasure Data overview

Sunday, July 14, 13

Company Overview§ Silicon Valley-based Company

• All Founders are Japanese• Hironobu Yoshikawa• Kazuki Ohta• Sadayuki Furuhashi

• About 20 people• Over 3.5 million jobs

§ OSS Enthusiasts• MessagePack, Fluentd, etc.

4

Sunday, July 14, 13

Investors

§ Bill Tai§ Othman Laraki - Former VP Growth at Twitter§ James Lindenbaum, Adam Wiggins, Orion Henry -

Heroku Founders§ Anand Babu Periasamy, Hitesh Chellani - Gluster

Founders§ Yukihiro “Matz” Matsumoto - Creator of Ruby§ Dan Scheinman - Director of Arista Networks§ Jerry Yang - Founder of Yahoo!

5

Sunday, July 14, 13

6

Data Volume

Cloud

EnterpriseRDBMSLightweight

RDBMS

DB2

1Bil entryOr 10TB

TraditionalData Warehouse

$10Bmarket

$34Bmarket

Database-as-a-service

Big Data-as-a-Service

On-Premise

© 2012 Forrester Research, Inc. Reproduction Prohibited

Treasure Data = Cloud + Big Data

Sunday, July 14, 13

The Problem with Other Solutions7

CustomerValue

TimeSign-up or PO

On-Premise Solutions

Obsolescenceover time

Treasure Data

Fully integrated Big Data full-stack service with simple interface, low friction initial engagement & continuous

technical upgrade

Need Upgrade

AWS(or hosted Hadoops)EC2

EMR

RedShift

S3 Step-by-step manual integrations

Maintain

NO SpecialistsTOO LONG to get Live

=

Complex Solutions

+

Data Collection

+

Sunday, July 14, 13

8

Big Data Adoption Stages

Intelligence Sophistication

Standard Reports

Ad-hoc Reports

Drill Down Query

Alerts

Statistical Analysis

Predictive Analysis

Optimization

What happened?

Where?

Where exactly?

Error?

Why?

What’s a trend?

What’s the best?

Analytics

Reporting

Sunday, July 14, 13

8

Big Data Adoption Stages

Intelligence Sophistication

Standard Reports

Ad-hoc Reports

Drill Down Query

Alerts

Statistical Analysis

Predictive Analysis

Optimization

What happened?

Where?

Where exactly?

Error?

Why?

What’s a trend?

What’s the best?

Analytics

Reporting

Treasure Data’s FOCUS

(80% of needs)

Sunday, July 14, 13

9

Full Stack Support for Big Data Reporting

Our best-in-class architecture and operations team ensure the integrity and availability of your data.

Data from almost any source can be securely and reliably uploaded using td-agent in streaming or batch mode.

Our SQL, REST, JDBC, ODBC and command-line interfaces support all major query tools and approaches.

You can store gigabytes to petabytes of data efficiently and securely in our cloud-based columnar datastore.

Sunday, July 14, 13

We are...

10

Big Data as a Service

not

Hadoop on Cloud

Sunday, July 14, 13

Columnar Storage+

HadoopMapReduce

600 bil+ records3.5 mil+ jobs

Product11

Data Collection Data Warehouse Data Analysis

Open-SourceLog Collector

2,500+ companies(incl. LinkedIn, etc)

Bulk Loader

CSV / TSVMySQL, Postgres

Oracle, etc.

Web Log

App Log

Sensor

RDBMS

CRM

ERP

Streaming Upload

60billion / month

BI Tools

Tableau, QlickView,Pentaho, Excel, etc.

RESTJDBC / ODBC

SQL(HiveQL)

orPig

Bulk UploadParallel Upload

Value Proposition:“Time-to-Answer” 20bil+, 2 weeks,

UK/Austria3bil+, 3 weeks

Singapore2 weeks,

US

2 weeks, US

3 weeks,Japan

Dashboard

Custom App,RDBMS, FTP, etc.

Result push

Multi-Tenant: Single Code for Everyone - Improving the Platform Faster (e.g. SFDC, Heroku)

Sunday, July 14, 13

12

Our Customers – 80 companies

http://docs.treasure-data.com/categories/success-stories

Sunday, July 14, 13

13

A case: “14 Days” from Signup to Success

1. Europe’s largest mobile ad exchange.

2. Serving >20 billion imps/month for >15,000 mobile apps (Q1 2013)

3. Immediate need of analytics infrastructure: ASAP!

4. With TD, MobFox got into production only in 14 days, by one engineer.

"Time is the most precious asset in our fast-moving business,and Treasure Data saved us a lot of it."

Julian Zehetmayr, CEO & Founder

td-agent = fluentd rpm/deb

Sunday, July 14, 13

14

A case: “Replace” in-house Hadoop to TD

1. Global “Hulu” - Online Video Service with millions of users

2. Video contents are distributed to over 150 languages.

3. Had hard time maintaining Hadoop cluster

4. With TD, Viki deprecated their in-house Hadoop cluster and use engineer for core businesses.

Before

After

“Treasure Data has always given us thorough and timely support peppered with insightful tips to make the best use of their service."

Huy Nguyen, Software Engineer

Sunday, July 14, 13

15

A case: Treasure Data with BI Tool (Tableau)

1. World’s largest android application market

2. Serving >3 billion app downloads for >100 million users

3. Only one engineer managing the data infrastructure

4. With TD, the data engineer can focus on analyzing data with existing BI tool

"I will recommend Treasure Data to my friends in a heartbeat because it benefits all three stakeholders: Operations, Engineering and Business."

Simon Dong, Principal Architect - Data Engineering

Sunday, July 14, 13

16

- Vision -Single Analytics Platform for the World

http://www.chisite.org/initiatives/WGII

Sunday, July 14, 13

Treasure Data’sService Architecture

Sunday, July 14, 13

18

Treasure Data = Collect + Store + Query

Sunday, July 14, 13

19

Architecture Breakdown

Data Collection• Increasing variety of

data sources• No single data schema• Lack of streaming data

collection method• 60% of Big Data project

resource consumed

Data Store/Analytics• Remaining complexity in

both traditional DWH and Hadoop (very slow time to market)

• Challenges in scaling data volume and expanding cost.

Connectivity• Required to ensure

connectivity with existing BI/visualization/apps by JDBC, ODBC and REST.

• Output ot other services, e.g. S3, RDBMS, etc.

Sunday, July 14, 13

Product Philosophy

§ Data first, Schema later• “Schema-on-Read”• Both Batch and Query processing

§ Simple APIs• Easy to use and powerful

§ Easy integration• Log collecting, BI tools and etc...

20

Sunday, July 14, 13

Our technology stack

§ td-agent• ETL part of Treasure Data

§ Plazma• Big data processing infrastructure• Columnar oriented storage• Reliable data handling

§ Multi-tenant scheduler• Robust distributed queue and scheduler

21

Sunday, July 14, 13

§ 60% of BI project resource is consumed here§ Most ‘underestimated’ and ‘unsexy’ but MOST important§ Fluentd: OSS lightweight but robust Log Collector

• http://fluentd.org/

1) Data Collection

22

Sunday, July 14, 13

Apache

App

App

Other data sources

td-agent RDBMS

Treasure Data columnar data

warehouse

Query Processing Cluster

Query API

HIVE, PIG

JDBC, REST

User

td-command

BI apps

23

This!

Sunday, July 14, 13

fluentd.org

Fluentdthe missing log collector

24

Sunday, July 14, 13

Data Processing

Collect Store Process Visualize

Data source

Reporting Monitoring

Sunday, July 14, 13

Store Process

ClouderaHorton WorksTreasure Data

Collect Visualize

TableauExcel

R

easier & shorter time

???

Related Products

Sunday, July 14, 13

In short

§ Open sourced log collector written in Ruby• Easy to use, reliable and well performance

• like streaming event processing

§ Using rubygems ecosystem for plugins

27

It’s like syslogd, butuses JSON for log messages

Sunday, July 14, 13

tail

insert

eventbuffering

127.0.0.1 - - [11/Dec/2012:07:26:27] "GET / ...127.0.0.1 - - [11/Dec/2012:07:26:30] "GET / ...127.0.0.1 - - [11/Dec/2012:07:26:32] "GET / ...127.0.0.1 - - [11/Dec/2012:07:26:40] "GET / ...127.0.0.1 - - [11/Dec/2012:07:27:01] "GET / ...

...

28

Fluentd

Web Server

Example (apache to monogdb)2012-12-11 07:26:27

apache.log

{ "host": "127.0.0.1", "method": "GET", ...}

Sunday, July 14, 13

Application

･･･

Server2

Application

･･･

Server3

Application

･･･

Server1

FluentLog Server

High Latency!must wait for a day...

29

Before Fluentd

Sunday, July 14, 13

Application

･･･

Server2

Application

･･･

Server3

Application

･･･

Server1

Fluentd Fluentd Fluentd

Fluentd Fluentd

In streaming!

30

After Fluentd

Sunday, July 14, 13

Buffer Output

Input

> Forward> HTTP> File tail> dstat> ...

> Forward> File> MongoDB> ...

> File> Memory

31

Pluggable architecture

Engine

Output

> rewrite> ...

Pluggable Pluggable

Sunday, July 14, 13

Nagios

MongoDB

Hadoop

Alerting

Amazon S3

Analysis

Archiving

MySQL

Apache

Frontend

Access logs

syslogd

App logs

System logs

Backend

Databasesbuffer / filter / routing

32

Sunday, July 14, 13

td-agent

§ Open sourced distribution package of Fluentd• ETL part of Treasure Data

• rpm, deb and homebrew

§ Including useful components• ruby, jemalloc, fluentd

• 3rd party gems: td, mongo, webhdfs, etc...

• td plugin is for Treasure Data

§ http://packages.treasure-data.com/

33

Sunday, July 14, 13

§ Remaining complexity in both DWH and Hadoop§ Challenges in scaling data volume and expanding cost§ Plazma: Hadoop eco system and own projects

2) Data Store / Analytics

34

Sunday, July 14, 13

Apache

App

App

Other data sources

td-agent RDBMS

Treasure Data columnar data

warehouse

Query Processing Cluster

Query API

HIVE, PIG

JDBC, REST

User

td-command

BI apps

35

This!

Sunday, July 14, 13

AWS Component Dependencies (1)

§ RDS• Store user information, job status, etc...• Store metadata of our columnar database• Queue worker / Scheduler

§ EC2• API servers (Ruby on Rails 3)• Hadoop clusters• Job workers

• Using Chef to deploy

36

Sunday, July 14, 13

AWS Component Dependencies (2)§ ELB

• Load balancing of API servers• Load balancing of td-agents

§ S3• Columnar storage built on top of S3

• MessagePack columnar format• Realtime / Archive storage

• Our Result feature supports S3 output.

37

No EBS, EMR, SQS and other products !

Sunday, July 14, 13

FrontendQueue

WorkerHadoop

Fluentd

Applications push metrics to Fluentd(via local Fluentd)

Librato Metricsfor realtime analysis

Treasure Data

for historical analysis

Fluentd sums up data minutes(partial aggregation)

Treasure Data Service Processing Flow

38

Hadoop

Sunday, July 14, 13

39

Data Processing Flow

Sunday, July 14, 13

Structure of Columnar Storages

Realtime Storage

merge (every 1 hour)

2013-07-12 00:23:00 912ec802013-07-13 00:01:00 277a2592013-07-14 00:02:00 d52c831

...

23c82b0ba3405d4c15aa85d2190e6d7b1482412ab14f0332b8aee1198a7bc848b2791b8fd603c719e54f0e3d402b17638477c9a7977e7dab

...

SELECT ...

Archive Storage

Data import

40

Sunday, July 14, 13

Query Language

Query Execution

Columnar Data

Object Storage

41

Sunday, July 14, 13

1/4: Compile SQL into MapReduce

SELECT COUNT(DISTINCT ip) FROM tbl;

SQL Statement

HiveSQL - to - MapReduce

42

+TD UDFs

Sunday, July 14, 13

2/4: MapReduce is executed in parallel

SELECT COUNT(DISTINCT ip) FROM tbl;

43

Sunday, July 14, 13

3/4: Columnar Data Access

Read ONLY the Required Part of Data

SELECT COUNT(DISTINCT ip) FROM tbl;

44

Sunday, July 14, 13

4/4: Object-based Storage

45

Sunday, July 14, 13

Apply Schema

{“user”:54, “name”:”test”, “value”:”120”, “host”:”local”}

Schema user:int name:string value:int

SELECT 54 (int)

Raw data（JSON）

“test” (string) 120 (int)

host:int

NULL

46

Sunday, July 14, 13

Multi-Tenancy§ All customers share the Hadoop clusters (Multi Data Centers)§ Resource Sharing (Burst Cores), Rapid Improvement, Ease of Upgrade

47

datacenter A

datacenter B

datacenter C

datacenter D

Local FairScheduler

Local FairScheduler

Local FairScheduler

Local FairScheduler

GlobalScheduler

On-DemandResouce Allocation

Job Submission+ Plan Change

Sunday, July 14, 13

Trial and error on Cloud

§ Rapid development• Change hardware

• New architecture testing• Performance testing

• Change software• Hadoop parameters• etc...

§ Use git and chef for these purposes• Easy to deploy and apply changes• git for change history

48

Sunday, July 14, 13

§ Services• CopperEgg• Librato Metrics• Logentries• NewRelic• PagerDuty• Desk.com• Olark• HipChat

• Alerting

Our Operation Stack: Full Use of SaaS

49

§ Tools• Hosted Chef (Opscode)• Jenkins

• including integration test

44

Sunday, July 14, 13

Sunday, July 14, 13

Sunday, July 14, 13

Sunday, July 14, 13

53

3) Connectivity§ Need to visualize the query result§ Use metrics / graph for interactive comparison§ Result: Export result and use existence tools

45

Sunday, July 14, 13

Apache

App

App

Other data sources

td-agent RDBMS

Treasure Data columnar data

warehouse

Query Processing Cluster

Query API

HIVE, PIG

JDBC, REST

User

td-command

BI apps

54

This!

Sunday, July 14, 13

55

Pull and Push approaches

Query(Pull)

Web App

MySQLTreasure Data

Columnar Storage

QueryProcessingCluster

Query API

REST API

JDBC, ODBC Driver

td-command

BI apps

S3

Result(Push)

…

Sunday, July 14, 13

Support list

56

§ Result• Treasure Data• MySQL• PostgreSQL• Google SpreadSheet• REST API• S3• etc...

§ BI tool• Pentaho• Tableau• JasperSoft• Indicee• Dr. Sum• Metric Insight• etc...

http://docs.treasure-data.com/categories/3rd-party-tools-overview

http://docs.treasure-data.com/categories/result

Sunday, July 14, 13

§ Treasure Data• Cloud based Big-data analytics platform• Provide Machete for Big data reporting

§ Big Data processing• Collect / Store / Analytics / Visualization

§ Consider trade-off• Cloud reinforces idea but not differentiator

• What is the strong point?• Should focus own vision!

Conclusion

57

Our focus!

Sunday, July 14, 13

Big Data for the Rest of Us

www.treasure-data.com | @TreasureData

Sunday, July 14, 13