integrating with hadoop: couchbase connect 2014
TRANSCRIPT
Integrating with Hadoop: Import & Export Data with the Couchbase Sqoop Plugin
Shane Johnson | Product Marketing, Couchbase
Matt Ingenthron | Senior Director, Couchbase
©2014 Couchbase, Inc. 2
Who has big data?
Big Data
©2014 Couchbase, Inc. 3
What is it?
Big Data
VS.
VOLUME VELOCITY
VS.
OPERATIONAL ANALYTICAL
©2014 Couchbase, Inc. 5
Where is it?
Big Data
©2014 Couchbase, Inc. 7
Where is it coming from?
Big Data
CONSUMERS
ENERGY
TRANSPORTATION
DATAUSERSENSORGPSTELEMETRYACTIVITYEVENT
MORE MACHINESDATAUSERS
CONTINUOUSSTREAMOFDATA
14
Why Hadoop?
Big Data
PBs OF DATA
1000s OF SERVER
S
MAP REDUCE
16
How?
Big Data
HADOOPSTORMCOUCHBASEKAFKA
COMPLEX EVENT PROCESSING
RTREPOSITORY
PERPETUALSTORE
ANALYTICALDB
BUSINESSINTELLIGENCE
MONITORING
CHAT/VOICESYSTEM
BATCHTRACK
REAL-TIMETRACK
DASHBOARD
COLLECTION
ANALYSIS ANDVISUALIZATION
REST FILTER METRICS
20
What should I remember?
Big Data
NOKETCHUP!
Couchbase Hadoop Connector
©2014 Couchbase, Inc. 23
Sqoop == SQL to Hadoop
About Sqoop
Couchbase had the first certified sqoop connector back in 2012
But, it’s NoSQL?
• Sqoop runs by automatically generating a Hadoop M-R job that runs across the cluster. Input arguments generate code that is turned into a job and executed. In parallel.
Common Patterns
Example: Ad or Recommendation Targeting
©2014 Couchbase, Inc. 26
Example: Content Driven Site
©2014 Couchbase, Inc. 27
About the demonstration…• Couchbase Server 3.0 running locally on Mac OS X• Cloudera Quick Start VM as Hadoop Cluster
• Workload running against cluster• Start a sqoop job to “import” from couchbase
Demo
Alternatives
A similar bridge between Couchbase and Hadoop.
Couchdoop from Avira
• Uses Couchbase Views as a way to identify items for import from Couchbase to Hadoop.• Supports both import and
export--couchbase-view-keys ‘[“click”, “2014-09-16”]; [“click”, “2014-09-17”]’
• Allows for ranges on views
• Used in Avira’s Offers as part of their Antivirus Suite
Screenshot courtesy http://www.avira.com/en/avira-offers
Thanks! Q&A?
Matt Ingenthron – @ingenthr – [email protected]
Shane Johnson – @shane_dev – [email protected]
©2014 Couchbase, Inc. 30