data engineering with solr and spark
TRANSCRIPT
Wifi
•GTVisitor
•hotel/guest
•pass: 76FCE
Data Engineering with Solr and Spark
Grant Ingersoll
@gsingers
CTO, Lucidworks
Lucidworks Fusion Is Search-‐Driven Everything
•Drive next genera=on relevance via Content, Collabora=on and Context
•Harness best in class Open Source: Apache Solr + Spark
•Simplify applica=on development and reduce ongoing maintenance
Fusion is built on three core principles:
Fusion Architecture
REST
API
Worker Worker Cluster Mgr.
Apache Spark
Shards Shards
Apache Solr
HD
FS (O
ptio
nal)
Shared Config Mgmt
Leader Election
Load Balancing
ZK 1
Apache Zookeeper
ZK N
DATABASEWEBFILELOGSHADOOP CLOUD
Connectors
Aler=ng/Messaging
NLP
Pipelines
Blob Storage
Scheduling
Recommenders/Signals
…
Core Services
Admin UI
SECURITY BUILT-‐IN
https://twitter.com/gsingers/status/700459516362625026
Get Started
https://github.com/Lucidworks/fusion-examples/tree/master/great-wide-open-2016
• Why Search for Data Engineering?
• Quick intro to Solr
• Quick intro to Spark
• Solr + Spark
• Relevance 101
• Machine learning with Spark and Solr
• What’s next?
Let’s Do This
Examples throughout!
The Importance of Importance
Search-‐Driven Everything
Customer Service
Customer Insights
Fraud Surveillance
Research Portal
Online Retail Digital Content
• Data Engineering, esp. with text, is a strange and magical world filled with…
– Evil villains – Jesters – Wizards – Unicorns – Heroes!
• In other words, no system will be perfect
Caveat Emptor: Data Engineering EdiLon
• You will spend most of your time in data engineering, search, machine learning and NLP doing “grunt” work nicely labeled as:
– Preprocessing – Feature Selection – Sampling – Validation/testing/etc. – Content extraction – ETL
• Corollary: Start with simple, tried and true algorithms, then iterate
Why do data engineering with Solr and Spark?
Solr Spark
• Data exploration and visualization
• Easy ingestion and feature selection
• Powerful ranking features
• Quick and dirty classification and clustering
• Simple operation and scaling
• Stats and math built in
• Advanced machine learning: MLLib, Mahout, Deep Learning4j
• Fast, large scale iterative algorithms
• General purpose batch/streaming compute engine
Whole collection analysis!
• Lots of integrations with other big data systems
• Apache Lucene
• Grouping and Joins
• Stats, expressions, transformations and more
• Lang. Detection
• Extensible
• Massive Scale/Fault tolerance
Solr Key Features
• Full text search (Info Retr.)
• Facets/Guided Nav galore!
• Lots of data types
• Spelling, auto-complete, highlighting
• Cursors
• More Like This
• De-duplication
Lucene for the Win!
• Vector Space or Probabilistic, it’s your choice!
• Killer FST
• Wicked fast
• Pluggable compression, queries, indexing and more
• Advanced Similarity Models
• Lang. Modeling, Divergence from Random, more
• Easy to plug-in ranking
Solr and Your Tools
• Data ingest:
• JSON, CSV, XML, Rich types (PDF, etc.), custom
• Clients for Python, R, Java, .NET and more
• http://cran.r-project.org/web/packages/solr/index.html, amongst others
• Output formats: JSON, CSV, XML, custom
Basics of Solr Requests
• Querying:
• Simple: term, phrases, boolean, wildcards, weights
• Advanced: query parsers, spatial, etc.
• Facets: term, query, range, pivot, stats
• Highlighting
• Spell checking
Solr Basics
Spark Key Features
• General purpose, high powered cluster computing system
• Modern, faster alternative to MapReduce
• 3x faster w/ 10x less hardware for Terasort
• Great for iterative algorithms
• APIs for Java, Scala, Python and R
• Rich set of add-on libraries for machine learning, graph processing, integrations with SQL and other systems
• Deploys: Standalone, Hadoop YARN, Mesos
Spark Basics
• Resilient Distributed Datasets
• Spark SQL provides a Data Source, which provides a DataFrame
• DataFrames — a DSL for distributed data manipulation
• Seamless integration with other Spark tech: SparkR, Python
Spark Components
Spark Core
Spark SQL
Spark Streaming
MLlib (machine learning)
GraphX (BSP)
Hadoop YARN Mesos Standalone
HDFS Execution
Model The Shuffle Caching
components
engine
cluster mgmt
Tachyon
languages Scala Java Python R
shared memory
Why Spark for Solr?
• Build the index very, very quickly!
• Aggregations
• Boosts, stats, iterative computations
• Offline compute to update index with additional info (e.g. PageRank, popularity)
• Whole corpus analytics, clustering, classification
• Joins with other storage (Cassandra, HDFS, DB, HBase)
Why Solr for Spark?
• Massive simplification of operations!
• Non “dumb” distributed, resilient storage
• Random access with smart queries
• Table scans
• Advanced filtering, feature selection
• Schemaless when you want, predefined when you don’t
• Spatial, columnar, sparse
Spark + Solr in Anger
http://github.com/lucidworks/spark-solr
Map<String, String> options = new HashMap<String, String>();options.put("zkhost", zkHost);options.put("collection”, "tweets");DataFrame df = sqlContext.read().format("solr").options(options).load(); count = df.filter(df.col("type_s").equalTo(“echo")).count();
Spark Shell in a Nutshell
• Common commands
• Solr in Spark: queries, filters and other requests
• See commands.md in the Github repo
But is it relevant?
Tales from the trenches
Look before you leap
• Wing it
• Ask — Caveat Emptor
• Log analysis
• Experimentation: A/B (A/A) testing
Approaches
• Precision/Recall (also, Mean Avg. Precision)
• Mean Reciprocal Rank (MRR)
• Number of {Zero|Embarrassing} Results
• Inter-Annotator Agreement
• Normalized Discounted Cumulative Gain (NDCG)
Common Metrics
Tips and Traps
Algorithms Collective Intelligence Editors/Rules
The mainstay of any approach: leverages Lucene/Solr’s built in similarity engine, function queries and other capabilities to determine importance based on core index
Especially effective for curating the long tail, feedback from users and other systems provide key insights into importance. Can also be used to inform the business about trends and interests.
Should be used sparingly to handle key situations such as promotions and edge cases. Review often. Encourage experimentation instead. Works well for landing pages, boosts and blocks where you know the answers. Not to be confused with curating content.
Big Picture on Relevance
• Similarity Models
Default, BM25F, others
• Function Queries, Reranking, Boosts
• Phrases are almost always a win (edismax does most of this for you)
e.g.: (exact match terms)^100 AND (“termA termB…”~10)^50 AND (termA AND termB…)^10 AND (termA OR termB)
• Mind your analysis
Algorithms
• UI, UI, UI!
• 1000’s of rules
• Second is the first loser
• Local minimum
• Pet peeve queries
• Oprah effect
• Assumptions
It’s a trap!
Level up
• Spark ships with good out of the box machine learning capabilities
• Spark-Solr brings enhanced feature selection tools via Lucene analyzers
• Examples
k-means
word2vec
Find synonyms
Machine Learning at Work
Sneak Peek
• Parallel Execu=on of SQL across SolrCloud
• Real=me Map-‐Reduce (“ish”) Func=onality
• Parallel Rela=onal Algebra
• Builds on streaming capabili=es in 5.x
• JDBC client in the works
Just When You Thought SQL was Dead
Full, Parallelized, SQL Support
• Lots of Func=ons:
• Search, Merge, Group, Unique, Parallel, Select, Reduce, Select, innerJoin, hashJoin, Top, Rollup, Facet, Stats, Update, JDBC, Intersect, Complement, Logit
• Composable Streams
• Query op=miza=on built in
SQL Guts
Example
select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text=’XXXX’ group by str_s
rollup( search(collection1, q=”(text:XXXX)”, qt=”/export”, fl=”str_s,field_i”, partitionKeys=str_s, sort=”str_s asc”, zkHost=”localhost:9989/solr”), over=str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i)
• Provides replica=on between two or more SolrCloud clusters located in two or more data centers
• Uses exis=ng transac=on logs
• Asynchronous indexing
• No Single Point of Failure or boglenecks
• Leader-‐to-‐leader communica=on to ensure updates are only sent once
Never Go Down, or at least Recover Quickly!
Cross Data Center Replication
• Graph Traversal
• Find all tweets men=oning “Solr” by me or people I follow
• Find all drah blog posts about “Parallel SQL” wrigen by a developer
• Find 3-‐star hotels in NYC my friends stayed in last year
• BM25F Default Similarity
• Geo3D search
Make ConnecLons, Get BeXer Results
• Jegy 9.3 and hgp2 (6.x)
• Fully mul=plexed over a single connec=on
• Reduced chance of distributed deadlock
• Backup/Restore API
• Op=miza=ons to distributed search algorithm
• AngularJS-‐based UI
But Wait! There’s More!
2016
OCTOBER 13-16, 2016
BOSTON, MA
Resources
• This code: https://github.com/Lucidworks/fusion-examples/tree/master/great-wide-open-2016
• Company: http://www.lucidworks.com
• Our blog: http://www.lucidworks.com/blog
• Book: http://www.manning.com/ingersoll
• Solr: http://lucene.apache.org/solr
• Fusion: http://www.lucidworks.com/products/fusion
• Twitter: @gsingers
Appendix A: SQL details
Streaming API & Expressions●API ○ Java API to provide programming framework ○ Returns tuples as a JSON stream ○ org.apache.solr.client.solrj.io
●Expressions ○ String Query Language ○ Serialization format ○ Allows non-Java programmers to access Streaming API
DocValues must be enabled for any field to be returned
Streaming Expression Request
curl -‐-‐data-‐urlencode 'stream=search(sample, q="*:*", fl="id,field_i", sort="field_i asc")' http://localhost:8901/solr/sample/stream
Streaming Expression Response
{"responseHeader": {"status": 0, "QTime": 1}, "tuples": { "numFound": -‐1, "start": -‐1, "docs": [ {"id": "doc1", "field_i": 1}, {"id": "doc2", "field_i": 2}, {"EOF": true}] }}
Architecture●MapReduce-ish ○ Borrows Shuffling concept from M/R ●Logical tiers for performing the query ○ SQL tier: translates SQL to streaming expressions for parallel query plan,
selects worker nodes, merges results ○ Worker tier: executes parallel query plan, streams tuples from data tables
back ○ Data Table tier: queries SolrCloud collections, performs initial sort and
partitioning of results for worker nodes
JDBC Client●Parallel SQL includes a “thin” JDBC client
●Expanded to include SQL Clients such as DbVisualizer (SOLR-8502)
●Client only works with Parallel SQL features
Learning MoreJoel Bernstein’s presentation at Lucene Revolution: ●https://www.youtube.com/watch?v=baWQfHWozXc
Apache Solr Reference Guide: ●https://cwiki.apache.org/confluence/display/solr/
Streaming+Expressions ●https://cwiki.apache.org/confluence/display/solr/Parallel
+SQL+Interface
Spark Architecture
Spark Master (daemon)
Spark Slave (daemon)
my-spark-job.jar (w/ shaded deps)
My Spark App SparkContext
(driver) • Keeps track of live workers • Web UI on port 8080 • Task Scheduler • Restart failed tasks
Spark Executor (JVM process)
Tasks
Executor runs in separate process than slave daemon
Spark Worker Node (1...N of these)
Each task works on some partition of a data set to apply a transformation or action
Cache
Losing a master prevents new applications from being executed
Can achieve HA using ZooKeeper
and multiple master nodes
Tasks are assigned based on data-locality
When selecting which node to execute a task on, the master takes into account data locality
• RDD Graph • DAG Scheduler • Block tracker • Shuffle tracker