hadoop user group eu 2014
DESCRIPTION
TRANSCRIPT
A QUICK INTRODUCTION TO THE CASCADING ECOSYSTEM
Chris K Wensel | Hadoop Summit EU 2014
• Lead developer of the Cascading open-source project
• Founder of Concurrent, Inc.
• Involved with Apache Hadoop since it was called Apache Nutch
!
• Systems Architect, not a Data Scientist
WHO AM I?
2
3
For creating data oriented applications, frameworks, and languages [on Apache Hadoop]
Originally designed to hide complexity of Hadoop and prevent thinking in MapReduce
cascading.org
• Started in 2007
• 2.0 released June 2012
• 2.5 out now
• 3.0 WIP (if you look for it)
• Apache 2.0 Licensed
• Supports all Hadoop distros
SOME STATS
4
5
What’s it used for?
6
• Cascading Java API
• Data normalization and cleansing of search and click-through
logs for use by analytics tools
• Easy to operationalize heavy lifting of data
7
• Cascalog (Clojure)
• Weather pattern modeling to protect growers against loss
• ETL against 20+ datasets daily
• Machine learning to create models
• Purchased by Monsanto for $930M US
8
• Scalding (Scala)
• Machine learning (linear algebra) to improve
• User experience
• Ad quality (matching users and ad effectiveness)
• All revenue applications are running on Cascading/Scalding
• IPO
9
• Estimate suicide risk from what people write online
• Cascading + Cassandra
• You can do more than optimize add yields
• http://www.durkheimproject.org
KEY PROJECTS
10
Lingual Pattern
Cascading
Apache Hadoop
Scalding Cascalog
• Java API (alternative to Hadoop MapReduce)
• Separates business logic from integration
• Testable at every lifecycle stage
• Works with any JVM language
• Many integration adapters
CASCADING
11
Process Planner
Processing API Integration APIScheduler API
Scheduler
Apache Hadoop
Cascading
Data Stores
ScriptingScala, Clojure, JRuby, Jython, Groovy
Enterprise Java
• Functions • Filters • Joins ‣ Inner / Outer / Mixed ‣ Asymmetrical / Symmetrical
• Merge (Union) • Grouping ‣ Secondary Sorting ‣ Unique (Distinct)
• Aggregations ‣ Count, Average, etc ‣ Rolling windows
SOME COMMON PATTERNS
12
filter
filter
function
functionfilterfunctiondata
PipelineSplit Join
Merge
data
Topology
13
word count – Cascading Java API !String docPath = args[ 0 ];!String wcPath = args[ 1 ];!Properties properties = new Properties();!AppProps.setApplicationJarClass( properties, Main.class );!HadoopFlowConnector flowConnector = new HadoopFlowConnector( properties );!!// create source and sink taps!Tap docTap = new Hfs( new TextDelimited( true, "\t" ), docPath );!Tap wcTap = new Hfs( new TextDelimited( true, "\t" ), wcPath );!!// specify a regex to split "document" text lines into token stream!Fields token = new Fields( "token" );!Fields text = new Fields( "text" );!RegexSplitGenerator splitter = new RegexSplitGenerator( token, "[ \\[\\]\\(\\),.]" );!// only returns "token"!Pipe docPipe = new Each( "token", text, splitter, Fields.RESULTS );!// determine the word counts!Pipe wcPipe = new Pipe( "wc", docPipe );!wcPipe = new GroupBy( wcPipe, token );!wcPipe = new Every( wcPipe, Fields.ALL, new Count(), Fields.ALL );!!// connect the taps, pipes, etc., into a flow definition!FlowDef flowDef = FlowDef.flowDef().setName( "wc" )! .addSource( docPipe, docTap )! .addTailSink( wcPipe, wcTap );!// create the Flow!Flow wcFlow = flowConnector.connect( flowDef ); // <<-- Unit of Work!wcFlow.writeDOT( "wc.dot" ); // <<-- On Next Slide!wcFlow.complete(); // <<-- Runs jobs on Cluster
1
3
2
scheduling
processing
integration
configuration
14
map
reduceEvery('wc')[Count[decl:'count']]
Hfs['TextDelimited[[UNKNOWN]->['token', 'count']]']['output/wc']']
GroupBy('wc')[by:['token']]
Each('token')[RegexSplitGenerator[decl:'token'][args:1]]
Hfs['TextDelimited[['doc_id', 'text']->[ALL]]']['data/rain.txt']']
[head]
[tail]
[{2}:'token', 'count'][{1}:'token']
[{2}:'doc_id', 'text'][{2}:'doc_id', 'text']
wc[{1}:'token'][{1}:'token']
[{2}:'token', 'count'][{2}:'token', 'count']
[{1}:'token'][{1}:'token']
wc.dot
A REAL WORLD APP
15
[1/75] map+reduce
[2/75] map+reduce [3/75] map+reduce [4/75] map+reduce[5/75] map+reduce [6/75] map+reduce[7/75] map+reduce [8/75] map+reduce [9/75] map+reduce[10/75] map+reduce [11/75] map+reduce [12/75] map+reduce[13/75] map+reduce [14/75] map+reduce [15/75] map+reduce[16/75] map+reduce [17/75] map+reduce [18/75] map+reduce
[19/75] map+reduce [20/75] map+reduce[21/75] map+reduce [22/75] map+reduce[23/75] map+reduce [24/75] map+reduce[25/75] map+reduce [26/75] map+reduce[27/75] map+reduce [28/75] map+reduce [29/75] map+reduce [30/75] map+reduce[31/75] map+reduce[32/75] map+reduce [33/75] map+reduce [34/75] map+reduce [35/75] map+reduce
[36/75] map+reduce
[37/75] map+reduce
[38/75] map+reduce[39/75] map+reduce [40/75] map+reduce[41/75] map+reduce [42/75] map+reduce[43/75] map+reduce [44/75] map+reduce[45/75] map+reduce [46/75] map+reduce [47/75] map+reduce [48/75] map+reduce[49/75] map+reduce[50/75] map+reduce [51/75] map+reduce [52/75] map+reduce [53/75] map+reduce
[54/75] map+reduce
[55/75] map [56/75] map+reduce [57/75] map[58/75] map[59/75] map
[60/75] map [61/75] map[62/75] map
[63/75] map+reduce[64/75] map+reduce [65/75] map+reduce [66/75] map+reduce[67/75] map+reduce[68/75] map+reduce [69/75] map+reduce[70/75] map+reduce
[71/75] map [72/75] map
[73/75] map+reduce [74/75] map+reduce
[75/75] map+reduce
1 App, 1 Flow, 75 Steps/MRJobs !green = map + reduce purple = map blue = join/merge orange = map split
A graph of jobs, not operations!
16
It’s not just for Java
17
word count – Scalding (Scala) // Sujit Pal!// sujitpal.blogspot.com/2012/08/scalding-for-impatient.html!!
package com.mycompany.impatient!!
import com.twitter.scalding._!!
class Part2(args : Args) extends Job(args) {! val input = Tsv(args("input"), ('docId, 'text))! val output = Tsv(args("output"))! input.read.! flatMap('text -> 'word) {! text : String => text.split("""\s+""")! }.! groupBy('word) { group => group.size }.! write(output)!}!
18
word count – Cascalog (Clojure) ; Paul Lam!; github.com/Quantisan/Impatient!!(ns impatient.core! (:use [cascalog.api]! [cascalog.more-taps :only (hfs-delimited)])! (:require [clojure.string :as s]! [cascalog.ops :as c])! (:gen-class))!!(defmapcatop split [line]! "reads in a line of string and splits it by regex"! (s/split line #"[\[\]\\\(\),.)\s]+"))!!(defn -main [in out & args]! (?<- (hfs-delimited out)! [?word ?count]! ((hfs-delimited in :skip-header? true) _ ?line)! (split ?line :> ?word)! (c/count ?count)))!
• Step by step tutorials on Cascading on GitHub
• Community has ported them to Scalding and Cascalog
!
• http://docs.cascading.org/impatient/
“FOR THE IMPATIENT” SERIES
19
• Foundation of patterns and best practices for building
Languages, Frameworks, and Applications
• Designed to abstract Hadoop away from the business logic
• Other models than MapReduce on the way!
WHY CASCADING?
20
• ANSI Compatible SQL
• JDBC Driver
• Cascading Java API
• SQL Command Shell
• Catalog Manager Tool
• Data Provider API
LINGUAL
21
Query Planner
JDBC API Lingual APIProvider API
Cascading
Apache HadoopLingual
Data Stores
CLI / Shell Enterprise Java
Catalog
22
Cascading API !
FlowDef flowDef = FlowDef.flowDef()! .setName( "sqlflow" )! .addSource( "example.employee", emplTap )! .addSource( "example.sales", salesTap )! .addSink( "results", resultsTap );! !SQLPlanner sqlPlanner = new SQLPlanner()! .setSql( sqlStatement );! !flowDef.addAssemblyPlanner( sqlPlanner );!
!
!
23
JDBC driver public void run() throws ClassNotFoundException, SQLException {! Class.forName( "cascading.lingual.jdbc.Driver" );! Connection connection =! DriverManager.getConnection(! "jdbc:lingual:local;schemas=src/main/resources/data/example" );! Statement statement = connection.createStatement();! ! ResultSet resultSet = statement.executeQuery(! "select *\n"! + "from \"EXAMPLE\".\"SALES_FACT_1997\" as s\n"! + "join \"EXAMPLE\".\"EMPLOYEE\" as e\n"! + "on e.\"EMPID\" = s.\"CUST_ID\"" );! ! // do something! ! resultSet.close();! statement.close();! connection.close();! }
SHELL - !TABLES
24
25
# load the JDBC package!library(RJDBC)! !# set up the driver!drv <- JDBC("cascading.lingual.jdbc.Driver", ! "~/src/concur/lingual/lingual-local/build/libs/lingual-local-1.0.0-wip-dev-jdbc.jar")! !# set up a database connection to a local repository!connection <- dbConnect(drv, ! "jdbc:lingual:local;catalog=~/src/concur/lingual/lingual-examples/tables;schema=EMPLOYEES")! !# query the repository: in this case the MySQL sample database (CSV files)!df <- dbGetQuery(connection, ! "SELECT * FROM EMPLOYEES.EMPLOYEES WHERE FIRST_NAME = 'Gina'")!head(df)! !# use R functions to summarize and visualize part of the data!df$hire_age <- as.integer(as.Date(df$HIRE_DATE) - as.Date(df$BIRTH_DATE)) / 365.25!summary(df$hire_age)!!library(ggplot2)!m <- ggplot(df, aes(x=hire_age))!m <- m + ggtitle("Age at hire, people named Gina")!m + geom_histogram(binwidth=1, aes(y=..density.., fill=..count..)) + geom_density()
26
> summary(df$hire_age) Min. 1st Qu. Median Mean 3rd Qu. Max. 20.86 27.89 31.70 31.61 35.01 43.92
27
“But we use a custom data format”
• Any Cascading Tap and/or Scheme can be used from JDBC
• Use a “fat jar” on local disk or from a Maven repo
‣ cascading-jdbc:cascading-jdbc-oracle-provider:1.0
• The Jar is dynamically loaded into cluster
DATA PROVIDER API
28
29
Amazon Elastic MapReduceJob Job Job Job
SELECT ... FROM file1 JOIN file2 ON file1.id = file2.id ...
Amazon S3
Amazon RedShift
file1 file2
results
• Quickly migrate existing work loads from RDBMS to Hadoop
• Quickly extract data from Hadoop into applications
WHY LINGUAL
30
• Predictive model scoring • Java API and PMML parser • Supports: ‣ (General) Regression ‣ Clustering ‣ Decisions Trees ‣ Random Forest ‣ and ensembles of models
PATTERN
31
PMML Parser Pattern API
Cascading
Apache Hadoop
Pattern
Data Stores
Enterprise Java
32
!
!
FlowDef flowDef = FlowDef.flowDef()! .setName( "classifier" )! .addSource( "input", inputTap )! .addSink( "classify", classifyTap );! !PMMLPlanner pmmlPlanner = new PMMLPlanner()! .setPMMLInput( new File( pmmlModel ) )! .retainOnlyActiveIncomingFields();! !flowDef.addAssemblyPlanner( pmmlPlanner );!!
!
• Standards compliance provides integration with many tools
• Models are independent of data and integration
• Only debugging Cascading, not an ensemble of applications
WHY PATTERN
33
CLOSING THE LOOP
34
Cluster
Pattern
Desktop
Job
PMMLFlow
JDBCFlowimport data
create models
export models
execute models
import resultsJDBC
Flow
PMML
DATA
DATA
test results
Job Job
• Understand how your application maps onto your cluster
• Identify bottlenecks (data, code, or the system)
• Jump to the line of code implicated on a failure
• Plugin available via Maven repo
• Beta UI hosted online
DRIVEN
35
http://cascading.io/driven/
MANAGED WITH DRIVEN
36
37
• New query planner ‣ User definable Assertion and Transformation rules
‣ Sub-Graph Isomorphism Pattern Matching
‣ Cordella, L. P., Foggia, P., Sansone, C., & VENTO, M. (2004). A (sub)graph isomorphism algorithm for matching large graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(10), 1367–1372. doi:10.1109/TPAMI.2004.75
• Hadoop Tez support • And likely other platforms
CASCADING 3.0
38
THERE’S A BOOK!
39
Enterprise Data Workflows with Cascading
- Paco Nathan
O’Reilly, 2013 amazon.com/dp/1449358721
CONTACT
40
@cwensel | @cascading
www.cascading.org
www.concurrentinc.com