integrating with hadoop: couchbase connect 2014

Integrating with Hadoop: Import & Export Data with the Couchbase Sqoop Plugin

Shane Johnson | Product Marketing, Couchbase

Matt Ingenthron | Senior Director, Couchbase

©2014 Couchbase, Inc. 2

Who has big data?

Big Data

©2014 Couchbase, Inc. 3

What is it?

Big Data

VS.

VOLUME VELOCITY

VS.

OPERATIONAL ANALYTICAL

©2014 Couchbase, Inc. 5

Where is it?

Big Data

©2014 Couchbase, Inc. 7

Where is it coming from?

Big Data

CONSUMERS

ENERGY

TRANSPORTATION

DATAUSERSENSORGPSTELEMETRYACTIVITYEVENT

MORE MACHINESDATAUSERS

CONTINUOUSSTREAMOFDATA

14

Why Hadoop?

Big Data

PBs OF DATA

1000s OF SERVER

S

MAP REDUCE

16

How?

Big Data

HADOOPSTORMCOUCHBASEKAFKA

COMPLEX EVENT PROCESSING

RTREPOSITORY

PERPETUALSTORE

ANALYTICALDB

BUSINESSINTELLIGENCE

MONITORING

CHAT/VOICESYSTEM

BATCHTRACK

REAL-TIMETRACK

DASHBOARD

COLLECTION

ANALYSIS ANDVISUALIZATION

REST FILTER METRICS

20

What should I remember?

Big Data

NOKETCHUP!

Couchbase Hadoop Connector

©2014 Couchbase, Inc. 23

Sqoop == SQL to Hadoop

About Sqoop

Couchbase had the first certified sqoop connector back in 2012

But, it’s NoSQL?

• Sqoop runs by automatically generating a Hadoop M-R job that runs across the cluster. Input arguments generate code that is turned into a job and executed. In parallel.

Common Patterns

Example: Ad or Recommendation Targeting

©2014 Couchbase, Inc. 26

Example: Content Driven Site

©2014 Couchbase, Inc. 27

About the demonstration…• Couchbase Server 3.0 running locally on Mac OS X• Cloudera Quick Start VM as Hadoop Cluster

• Workload running against cluster• Start a sqoop job to “import” from couchbase

Demo

Alternatives

A similar bridge between Couchbase and Hadoop.

Couchdoop from Avira

• Uses Couchbase Views as a way to identify items for import from Couchbase to Hadoop.• Supports both import and

export--couchbase-view-keys ‘[“click”, “2014-09-16”]; [“click”, “2014-09-17”]’

• Allows for ranges on views

• Used in Avira’s Offers as part of their Antivirus Suite

Screenshot courtesy http://www.avira.com/en/avira-offers

Thanks! Q&A?

Matt Ingenthron – @ingenthr – matt@couchbase.com

Shane Johnson – @shane_dev – shane@couchbase.com

©2014 Couchbase, Inc. 30