Upload File

solving low latency query over big data with spark sql-(julien pierre, microsoft)

  • Home
  • Data & Analytics
  • Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
24

Upload: spark-summit

Post on 12-Aug-2015

790 views

Category:

Data & Analytics


0 download

Report

TRANSCRIPT

Page 1: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 2: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 3: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 4: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)

Client Data Fluency

Office

Skype

Bing

Modern Data Capability

Instrumentation & Ingestion

Processing & Storage

Reporting & Analytics

Information Management

Mobile-First Analytics Experience

Experimentation

Page 5: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 6: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 7: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)

Data Size

Query Latency

Page 8: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 9: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)

Get results inline in Zeppelin

Need to open the results in Excel

Page 10: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)

0 20 40 60 80 100 120 140 160 180 200

Cosmos

SparkSQL

SparkSQL with Cache

Write and Compile Query Submit and Wait in Job Queue Job Run Time

Page 11: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 12: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 13: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 14: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)

Mesos Cluster/HDFS

Job Manager Zookeeper

Job Frontend Web API

Spark Driver Host Pool

Spark Hive Thrift Server Zeppelin Server

Avocado (Hive Query + Schedule Task)

Rover (Drag & Drop BI tool with Hive Code

Gen)

Zeppelin Web UI

MetastoreDB Hive Loader

Cosmos Storage

Page 15: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)

Partition  1

Partition  2

...

Partition  n

Export  Cosmos  Partition

Partition  1

Partition  2

...

Partition  n

Task  2

HDFS.copyFromLocalFile

...

Task  n

Partition  1

Partition  2

...

Partition  n

saveAsParquetFile

Task  2...

Task  n

Page 16: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)

<Database2>

<Table1><Database1>

<Partition1>

<Table2><Partition2>

MetastoreDB

Hive  Thrift  Server

Hive  Loader

Zeppelin  Server

UserQueryQuery

Page 17: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 18: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 19: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 20: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 21: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 22: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)

Data Ingest

Services

Clients

Transform Compute

Transform Compute

Data Streams

Data Sets

Store

Event Processing

HDFS Data Transportation

Spark Streaming Receiver

Analyst

Zeppelin Notebooks

Avocado

Simple query

Query language

“Analyze”

“Debug”

“Mine”

“Glance”

Data

Unified platform Intelligence Interactive

analytics Data

Products

Better Digital

Experiences

Dual users

“Bing”

“Office”

Page 23: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)
Page 24: Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Microsoft)

Sparrow: Distributed, Low Latency Schedulingpeople.csail.mit.edu/matei/papers/2013/sosp_sparrow.pdf · the Spark [26] scheduler. 3 Sample-Based Scheduling for Parallel Jobs A traditional

Ultra-low Latency Switches XG2000 Series - Fujitsu · Ultra-low Latency Switches XG2000 Series Offers industry-leading low latency in 10GbE Switching (300ns latency) Delivers 400Gbps

LATENCY AWARE ELASTIC STREAMING FOR ESTIMATING ONLINE ... · provisioning and under -provisioning respectively. The sy stem uses Apache spark framework for parallel execution. Fig

High -Fidelity Latency Measurements in Low -Latency Networks

JULIEN DIDIER ROBIN JEAN JEAN-LUC JULIEN BLONDEL POLI

Julien Racca

SPARK SPARK VRT

High concurrency,Low latency analyticsusing Spark/Kudu

Long Tail of Latency - Duke Universitydb.cs.duke.edu/courses/cps214/fall15/Day15.pdf · Day 15. Agenda •Why is Latency Important? •Latency in Data Centers •Reducing Latency

Julien Chable [email protected] Wygwam –

Spark SQL and DataFrames Spark GraphX Spark Mlib Spark ...Spark GraphX! Spark Mlib! Spark Streaming Lightning-fast cluster computing. Chaining transformations 2. ... Covert RDD to

Prépa Phase Exploration · latency Analysis latency Decision latency Action latency time Counter measure takes effect Counter measure approved Analysis completed Insights about the

CYRILLE Julien

Julien ppt

Real-World ADC Interrupt Latency...Preempt-RT Latency Benchmarking of the Cortex-A53 • Software and Hardware setup • Basic Latency Tests • UDP Ping Pong Ethernet Latency •

CPU Sizing vs. Latency Analysis FTS EDR Latency Simulation

Julien Baudry

Sparrow: Distributed, Low Latency Scheduling · •Sparrow must support millions of scheduling decisions per second. Background: Spark •Data processing framework; optimizing for

Streaming Big Data with Spark Streaming, Kafka, Cassandra and … · 2017-08-23 · Asynchronous Data Passing Kafka, Akka, Spark Fast, Low Latency, Data Locality Cassandra, Spark,

Discretized Streams: Fault-Tolerant Streaming …Spark Streaming, based on the Spark engine [43]. The system can process over 60 million records/second on 100 nodes at sub-second latency,

Better with Delta Lake Making Apache Spark™...Making Apache Spark Better with Delta Lake Steven Yu, Senior Solutions Architect 1. Collect Everything ... •Low-latency or manually

Julien Cuny

Making Analytics Viable in Enterprises: Potential routes ... · Apache Spark for distributed computing Edge Analytics Apache Spark Streaming for low latency analytics Managing Industrial

Improving Python and Spark Performance and ... · • Apache member • Apache PMCs: Arrow, Kudu, Incubator, Pig, Parquet Julien Le Dem @J_ Li Jin @icexelloss • Software Engineer

Julien Champagne Et La v...Rale - Julien Champagne

Network Measurement & AAA Œ Overview of My Previous Workcaia.swin.edu.au/talks/CAIA-TALK-040707A.pdf · Metrics: DNS latency, TCP latency, HTTP transaction latency, web page latency,

Spark & Spark SQL

julien sorel

Sparrow Distributed Low-Latency Spark Scheduling Kay Ousterhout, Patrick Wendell, Matei Zaharia, Ion Stoica

Running Spark on Kubernetes: Julien Dumazer t, Co-Founder

Julien Dupré

diapo julien

2013 12 02 Sparrow - EECS at UC Berkeleykeo/talks/sparrow-spark-summit... · Sparrow Distributed Low-Latency Spark Scheduling Kay Ousterhout, Patrick Wendell, Matei Zaharia, Ion Stoica

Julien Lahaut.pdf

Saint Pierre-Julien Eymard - adoperp.fr...Saint Pierre-Julien EymardSaint Pierre-Julien Eymard Le père Pierre-Julien Eymard (1811-1868), fondateur des “Pères du Saint Sacrement”