Solr as a Spark SQL Datasource

Kiran Chitturi,Lucidworks

Solr & Spark

• A few interesting things about Spark

• Overview of SparkSQL and DataFrames

• Solr as a SparkSQL DataSource in depth

• Use Lucene for text analysis in ML pipelines

• Example Use Case: Lucidworks Fusion and Spark

The standard for enterprise search. of Fortune 500

uses Solr.

90%

What’s interesting about Spark?• Wealth of overview / getting started resources on the Web

➢ Start here -> https://spark.apache.org/

➢ Should READ! https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf

• Faster, more modernized alternative to MapReduce

➢ Spark running on Hadoop sorted 100TB in 23 minutes (3x faster than Yahoo’s previous record while using10x less computing power)

• Unified platform for Big Data

➢ Great for iterative algorithms (PageRank, K-Means, Logistic regression) & interactive data mining ➢ Runs on YARN, Mesos, and plays well with HDFS

• Nice API for Java, Scala, Python, R, and sometimes SQL … REPL interface too

• >14,200 Issues in JIRA, 1000+ code contributors, 2.0 coming soon!

Spark Components

Spark Core

Spark SQL

Spark Streaming

MLlib (machine learning)

GraphX (BSP)

Hadoop YARN Mesos Standalone

HDFSExecution

ModelThe Shuffle Caching

components

engine

cluster mgmt

Alluxio (formerly Tachyon)

languages Scala Java Python R

shared memory

Physical Architecture (Standalone)

Spark Master (daemon)

Spark Slave (daemon)

my-spark-job.jar (w/ shaded deps)

My Spark AppSparkContext

(driver)

• Keeps track of live workers

• Web UI on port 8080

• Task Scheduler

• Restart failed tasks

Spark Executor (JVM process)

Tasks

Executor runs in separate process than slave daemon

Spark Worker Node (1...N of these)

Each task works on some partition of a data set to apply a transformation or action

Cache

Losing a master prevents new applications from being executed

Can achieve HA using ZooKeeper and multiple master nodes

Tasks are assigned based on data-locality

When selecting which node to execute a task on, the master takes into account data locality

• RDD Graph • DAG Scheduler • Block tracker • Shuffle tracker

Spark SQL• DataSource API for reading from and writing to external data sources

• DataFrame is an RDD[Row] + schema

• Secret sauce is logical plan optimizer

• SQL or relational operators on DF

• JDBC / ODBC

• UDFs!

• Machine Learning Pipelines

Solr as a Spark SQL Data Source• Read/write data from/to Solr as DataFrame

• Use Solr Schema API to access field-level metadata

• Push predicates down into Solr query constructs, e.g. fq clause

• Deep-paging, shard partitioning, intra-shard splitting, streaming results

// Connect to Solr val opts = Map("zkhost" -> "localhost:9983", "collection" -> "nyc_trips") val solrDF = sqlContext.read.format("solr").options(opts).load

// Register DF as temp table solrDF.registerTempTable("trips")

// Perform SQL queries sqlContext.sql("SELECT avg(tip_amount), avg(fare_amount) FROM trips").show()

Shard 1

Shard 2

socialdata collection

Partition 1 (spark task)

solr.DefaultSource

Partition 2 (spark task)

Spark Driver App

ZooKeeperRead collection metadata (num shards, replica URLs, etc)

TableScanQuery:q=*:*&rows=1000&

distrib=false&cursorMark=*

Results streamed back from Solr

Para

lleliz

e qu

ery

exec

utio

n

sqlContext.load(“solr”,opts)

DataFrame (schema + RDD<Row>)

Map("collection”->“nyc_trips”,“zkHost”->“…”)

SolrRDD: Reading data from Solr into Spark

Shard 1

Shard 2

socialdata collection

Partition 1 (spark task)

solr.DefaultSource

Partition 3 (spark task)

Spark Driver App

ZooKeeperRead collection metadata

(num shards, replica URLs, etc)

TableScanQuery:q=*:*&rows=1000&

distrib=false&cursorMark=*&split_field:[aaTOba}

Para

lleliz

e qu

ery

exec

utio

n

sqlContext.load(“solr”,opts)

DataFrame (schema + RDD<Row>)

Map(“collection”->“nyc_trips”,“zkHost”->“…”)

SolrRDD: Reading data from Solr into Spark

Partition 2 (spark task) &split_field:[baTObz]

Partition 4 (spark task)

&split_field:[aaTOac}

&split_field:[acTOae]

Data-locality Hint

• SolrRDD extends RDD[SolrDocument] (written in Scala)

• Give hint to Spark task scheduler about where data lives

overridedefgetPreferredLocations(split:Partition):Seq[String]={

//returnpreferredhostnameforaSolrpartition }

• Useful when Spark executor and Solr replicas live on same physical host, as we do in Fusion

• Query to a shard has a “preferred” replica; can fallback to other replicas if the preferred goes down (will be in 2.1)

Solr Streaming API for fast reads

• Contributed to spark-solr by Bloomberg team (we PRs)

• Extremely fast “table scans” over large result sets in Solr

• Relies on a column-oriented data structure in Lucene: docValues

• DocValues help speed up faceting and sorting too!

• Coming soon! Push SQL predicates down into Solr’s Parallel SQL engine available in Solr 6.x

Writing to Solr (aka indexing)• Cloud-aware client sends updates to shard leaders in parallel

• Solr Schema API used to create fields on-the-fly using the DataFrame

schema

• Better parallelism than traditional approaches like Solr DIH

val dbOpts = Map( "url" -> "jdbc:postgresql:mydb", "dbtable" -> “schema.table", "partitionColumn" -> "foo", "numPartitions" -> "10")

val jdbcDF = sqlContext.read.format("jdbc").options(dbOpts).load

val solrOpts = Map("zkhost" -> "localhost:9983", "collection" -> "mycoll")

jdbcDF.write.format("solr").options(solrOpts).mode(SaveMode.Overwrite).save

Solr / Lucene Analyzers for Spark ML Pipelines• Spark ML Pipeline provides nice API for defining stages to train / predict ML models

• Crazy idea ~ use battle-hardened Lucene for text analysis in Spark

• Pipelines support import/export (work in progress, more coming in Spark 2.0)

• Can try different text analysis techniques during cross-validation

DF

Spark ML Pipeline

Lucene Analyzer HashingTF

Standard Scaler

Trained Model (SVM)

save

https://lucidworks.com/blog/2016/04/13/spark-solr-lucenetextanalyzer/

Fusion & Spark• spark-solr 2.0.1 released, built into Fusion 2.4

• Users leave evidence of their needs & experience as they use your app

• Fusion closes the feedback loop to improve results based on user “signals” • Train and serve ML Pipeline and mllib based Machine Learning models

• Run custom Scala “script” jobs in the background in Fusion ➢Complex aggregation jobs (see next slide) ➢Unsupervised learning (LDA topic modeling) ➢Re-train supervised models as new training data flows in

Scheduled Fusion job to compute stats for user sessionsvalopts=Map("zkhost"->"localhost:9983”,"collection"->"apachelogs”)varlogEvents=sqlContext.read.format("solr").options(opts).loadlogEvents.registerTempTable("logs”)sqlContext.udf.register("ts2ms",(d:java.sql.Timestamp)=>d.getTime)sqlContext.udf.register("asInt",(b:String)=>b.toInt)valsessions=sqlContext.sql("""|SELECT*,sum(IF(diff_ms>30000,1,0))|OVER(PARTITIONBYclientipORDERBYts)session_id|FROM(SELECT*,ts2ms(ts)-lag(ts2ms(ts))|OVER(PARTITIONBYclientipORDERBYts)asdiff_msFROMlogs)tmp""".stripMargin)sessions.registerTempTable("sessions")

varsessionsAgg=sqlContext.sql("""|SELECTconcat_ws('||',clientip,session_id)asid,|first(clientip)asclientip,|min(ts)assession_start,|max(ts)assession_end,|(ts2ms(max(ts))-ts2ms(min(ts)))assession_len_ms_l,|count(*)astotal_requests_l|FROMsessions|GROUPBYclientip,session_id""".stripMargin)

sessionsAgg.write.format("solr").options(Map("zkhost"->"localhost:9983","collection"->"apachelogs_signals_aggr")).mode(org.apache.spark.sql.SaveMode.Overwrite).save

Getting started with spark-solr

• Import package via maven

./bin/spark-shell --packages "com.lucidworks.solr:spark-solr:2.0.1"

• Build from source git clone https://github.com/LucidWorks/spark-solr cd spark-solr mvn clean package -DskipTests ./bin/spark-shell --jars 2.1.0-SNAPSHOT.jar

Example : Deep paging via shards

// Connect to Solr val opts = Map( "zkhost" -> "localhost:9983", "collection" -> "nyc_trips")

val solrDF = sqlContext.read.format("solr").options(opts).load

// Register DF as temp table solrDF.registerTempTable("trips")

sqlContext.sql("SELECT * FROM trips LIMIT 2").show()

Example : Deep paging with intra shard splitting

// Connect to Solr val opts = Map( "zkhost" -> "localhost:9983", "collection" -> "nyc_trips", "splits" -> "true") val solrDF = sqlContext.read.format("solr").options(opts).load

// Register DF as temp table solrDF.registerTempTable("trips")

sqlContext.sql("SELECT * FROM trips").count()

Example : Streaming API (/export handler)

// Connect to Solr val opts = Map( "zkhost" -> "localhost:9983", "collection" -> "nyc_trips") val solrDF = sqlContext.read.format("solr").options(opts).load

// Register DF as temp table solrDF.registerTempTable("trips")

sqlContext.sql("SELECT avg(tip_amount), avg(fare_amount) FROM trips").show()

Performance test• NYC taxi data (30 months - 91.7M rows) • Dataset loaded in to AWS RDS instance (Postgres)

• 3 EC2 nodes of r3.2x large instances • Solr and Spark instances co-located together

• Collection ‘nyc-taxi’ created with 6 shards, 1 replication • Deployed using solr-scale-tk (https://github.com/LucidWorks/solr-scale-tk)

• Dataset link: https://github.com/toddwschneider/nyc-taxi-data • More details: https://gist.github.com/kiranchitturi/0be62fc13e4ec7f9ae5def53180ed181

• Query - simple aggregation query to calculate averages

• Streaming expressions took 2.3 mins across 6 tasks

• Deep paging took 20 minutes across 120 tasks

Query performance

Index performance• 91.4M rows imported to Solr in 49 minutes

• Docs per second: 31K

• JDBC batch size: 5000

• Indexing batch size: 50000

• Partitions: 200

Wrap-up and Q & A

Download Fusion: http://lucidworks.com/fusion/download/

Feel free to reach out to me with questions:

kiran.chitturi@lucidworks.com / @chitturikiran

Spark Streaming: Nuts & Bolts• Transform a stream of records into small, deterministic batches

✓ Discretized stream: sequence of RDDs ✓ Once you have an RDD, you can use all the other Spark libs (MLlib, etc) ✓ Low-latency micro batches ✓ Time to process a batch must be less than the batch interval time

• Two types of operators: ✓ Transformations (group by, join, etc) ✓ Output (send to some external sink, e.g. Solr)

• Impressive performance! ✓ 4GB/s (40M records/s) on 100 node cluster with less than 1 second latency (note: not indexing rate)

✓ Haven’t found any unbiased, reproducible performance comparisons between Storm / Spark

Spark Streaming Example: Solr as Sink

Twitter

./spark-submit--masterMASTER--classcom.lucidworks.spark.SparkAppspark-solr-1.0.jar\twitter-to-solr-zkHostlocalhost:2181–collectionsocial

Solr

JavaReceiverInputDStream<Status> tweets = TwitterUtils.createStream(jssc, null, filters);

Various transformations / enrichments on each tweet (e.g. sentiment analysis, language detection)

JavaDStream<SolrInputDocument> docs = tweets.map( new Function<Status,SolrInputDocument>() { // Convert a twitter4j Status object into a SolrInputDocument public SolrInputDocument call(Status status) { SolrInputDocument doc = new SolrInputDocument(); … return doc; }});

map()

class TwitterToSolrStreamProcessor extends SparkApp.StreamProcessor

SolrSupport.indexDStreamOfDocs(zkHost, collection, 100, docs);

Slide Legend

Provided by Spark

Custom Java / Scala code

Provided by Lucidworks

Document Matching using Stored Queries

• For each document, determine which of a large set of stored queries matches.

• Useful for alerts, alternative flow paths through a stream, etc

• Index a micro-batch into an embedded (in-memory) Solr instance and then determine which queries match

• Matching framework; you have to decide where to load the stored queries from and what to do when matches are found

• Scale it using Spark … need to scale to many queries, checkout Luwak

Document Matching using Stored Queries

Stored Queries

DocFilterContext

Twitter map()

Slide Legend

Provided by Spark

Custom Java / Scala code

Provided by Lucidworks

JavaReceiverInputDStream<Status> tweets = TwitterUtils.createStream(jssc, null, filters);

JavaDStream<SolrInputDocument> docs = tweets.map( new Function<Status,SolrInputDocument>() { // Convert a twitter4j Status object into a SolrInputDocument public SolrInputDocument call(Status status) { SolrInputDocument doc = new SolrInputDocument(); … return doc; }});

JavaDStream<SolrInputDocument> enriched = SolrSupport.filterDocuments(docFilterContext, …);

Get queries

Index docs into an EmbeddedSolrServer Initialized from configs stored in ZooKeeper

…

ZooKeeper

Key abstraction to allow you to plug-in how to store the queries and what action to take when docs match

RDD Illustrated: Word count

map(word=>(word,1))

Map words into pairs with count of 1(quick,1)

(brown,1)

(fox,1)

(quick,1)

(quick,1)

valfile=spark.textFile("hdfs://...")

HDFS

file RDD from HDFS

quickbrownfoxjumped…

quickbrownierecipe…

quickdryingglue…

……

…

file.flatMap(line=>line.split(""))

Split lines into words

quick

brown

fox

quick

quick

……

reduceByKey(_+_)

Send all keys to same reducer and sum

(quick,1)

(quick,1)

(quick,1)

(quick,3)

Shuffle across

machine boundaries

Executors assigned based on data-locality if possible, narrow transformations occur in same executor Spark keeps track of the transformations made to generate each RDD

Partition 1Partition 2Partition 3

xvalfile=spark.textFile("hdfs://...")valcounts=file.flatMap(line=>line.split("")).map(word=>(word,1)).reduceByKey(_+_)counts.saveAsTextFile("hdfs://...")

Understanding Resilient Distributed Datasets (RDD)• Read-only partitioned collection of records with fault-tolerance

• Created from external system OR using a transformation of another RDD

• RDDs track the lineage of coarse-grained transformations (map, join, filter, etc)

• If a partition is lost, RDDs can be re-computed by re-playing the transformations

• User can choose to persist an RDD (for reusing during interactive data-mining)

• User can control partitioning scheme

Physical Architecture

Spark Master (daemon)

Spark Slave (daemon)

my-spark-job.jar (w/ shaded deps)

My Spark AppSparkContext

(driver)

• Keeps track of live workers

• Web UI on port 8080

• Task Scheduler

• Restart failed tasks

Spark Executor (JVM process)

Tasks

Executor runs in separate process than slave daemon

Spark Worker Node (1...N of these)

Each task works on some partition of a data set to apply a transformation or action

Cache

Losing a master prevents new applications from being executed

Can achieve HA using ZooKeeper and multiple master nodes

Tasks are assigned based on data-locality

When selecting which node to execute a task on, the master takes into account data locality

• RDD Graph • DAG Scheduler • Block tracker • Shuffle tracker

Spark vs. Hadoop’s Map/Reduce

Operators File System Fault-Tolerance Ecosystem

Hadoop Map-Reduce HDFS S3

Replicated blocks

Java API Pig

Hive

Spark filter, map, flatMap, join, groupByKey,

reduce, sortByKey,

count, distinct, top, cogroup,

etc …

HDFS S3

Tachyon

Immutable RDD lineage

Python / Scala / Java API

SparkSQL GraphX MLlib

SparkR