hadoop world spark meetup: interactive spark in your browser
Post on 09-Jan-2017
2.125 Views
Preview:
TRANSCRIPT
GOALOF HUE
WEB INTERFACE FOR ANALYZING DATA WITH APACHE HADOOP
SIMPLIFY AND INTEGRATEFREE AND OPEN SOURCE
—> “BIG NOTEBOOK” FOR HADOOP
•Married with full ecosystem • File, Job browsers • Create table wizards • Any language (Hive, Spark...) • Graphing • Export/Import/Sharing
•MulT users • ImpersonaTon
HADOOP WITH SPARKNOTEBOOK
• REST Web server in Scala
• InteracTve Spark Sessions and Batch Jobs
• Type IntrospecTon for VisualizaTon
• Running sessions in YARN local
• Backends: Scala, Python, R
• Open Source: h[ps://github.com/cloudera/hue/tree/master/apps/spark/java
• Play with Curl h[p://gethue.com/how-‐to-‐use-‐the-‐
LIVYSPARK SERVER
LIVY WEB SERVERARCHITECTURE
YARN Master
Spark Client
YARNNode
SparkInterpreter
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
Livy Server
Scalatra
Session Manager
Session
LIVY WEB SERVERARCHITECTURE
Livy Server
YARN Master
Scalatra
Spark Client
Session Manager
Session
YARNNode
SparkInterpreter
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1
LIVY WEB SERVERARCHITECTURE
YARN Master
Spark Client
YARNNode
SparkInterpreter
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1
2
Livy Server
Scalatra
Session Manager
Session
LIVY WEB SERVERARCHITECTURE
YARN Master
Spark Client
YARNNode
SparkInterpreter
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1
2
3
Livy Server
Scalatra
Session Manager
Session
LIVY WEB SERVERARCHITECTURE
YARN Master
Spark Client
YARNNode
SparkInterpreter
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1
2
3
4Livy Server
Scalatra
Session Manager
Session
LIVY WEB SERVERARCHITECTURE
YARN Master
Spark Client
YARNNode
SparkInterpreter
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1
2
3
4
5
Livy Server
Scalatra
Session Manager
Session
LIVY WEB SERVERARCHITECTURE
YARN Master
Spark Client
YARNNode
SparkInterpreter
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1
2
3
4
5
6Livy Server
Scalatra
Session Manager
Session
LIVY WEB SERVERARCHITECTURE
YARN Master
Spark Client
YARNNode
SparkInterpreter
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1 7
2
3
4
5
6Livy Server
Scalatra
Session Manager
Session
SESSION CREATIONAND EXECUTION
% curl -‐XPOST localhost:8998/sessions \ -‐d '{"kind": "spark"}' { "id": 0, "kind": "spark", "log": [...], "state": "idle" }
% curl -‐XPOST localhost:8998/sessions/0/statements -‐d '{"code": "1+1"}' { "id": 0, "output": { "data": { "text/plain": "res0: Int = 2" }, "execution_count": 0, "status": "ok" }, "state": "available" }
@gethue
USER GROUP
hue-‐user@
WEBSITE
h[p://gethue.com
LEARN
h[p://learn.gethue.com
THANKS!
top related