Batch Indexing & Near Real Time, keeping things fast
Marc SturleseSoftware engineer @ Trovit
About me...
• Marc Sturlese – @sturlese
• Software engineer @Trovit. R&D focused
• Responsible for search and scalability
Agenda
• Who we are
• Batch architecture. Hadoop & Hive
• Near real time architecture. Storm & stuff
• Putting it all together
• Alternatives and Future directions
• Questions
Who we are
Trovit, a search engine for classifieds
Who we are
Batch Layer
• Hadoop based
• Documents are crunched by a pipeline of MR jobs
• Hive to save stats of each phase
Batch LayerPipeline overview
Incoming data
Deployment
Lucene Indexes
Ad Processor Diff Matching Expiration Deduplication Indexing
t – 1
External Data
Hive Stats
Hadoop Cluster
Batch LayerThe good things!
• Index always built from scratch. Small number of big segments
• Multicast deployment allows to send indexes to all slaves at the same time.
• Backups convenient on HDFS
Batch LayerThat was cool but...
• Not even close to real time
• Crunch documents in batch means to wait until all is processed. This can take a few hours
• We want to show the user fresher results!
Near real time LayerStorm and stuff to the rescue
Near real time LayerStorm properties
• Distributed real time computation system
• Fault tolerance
• Horizontal scalability
• Low latency
• Reliability
Near real time LayerStorm in action
Slave
Slave
Solr prod replicas
SlaveXML feed
XML feed
Kafka partition
Kafka partition
Storm topologySources
Kafka spout
Kafka spout
XML spout Doc Manager bolt Indexer bolt
SHUFFLEGROUPING GROUPING
FIELD
Near real time LayerStorm in action
• Spouts just read and send
• Doc Manager Bolt processes and classifies
• Indexer Bolt adds documents to Solr
• Replicated logic with different implementation
• Careful not to overload Solr slaves...
Near real time LayerStorm in action
Near real time LayerStorm in action. But...
Near real time LayerStorm in action. But...
• Now Solr has to handle user queries and storm inserts
• Field grouping on Indexer Bolt for politeness
• Small bulks to reduce insert requests
• Committing on many cores, same host, same time can be painful
Near real time LayerStorm in action - Committing
Indexer Bolt Cars US
Real state UK R1 Cars US R1 Cars US R2 Jobs BR R1 Jobs BR R2 Real state ES R1
Indexer Bolt Jobs BR
ZooKeeper Locker
Slave 1 Slave 2 Slave N
. . .
Near real time LayerStorm in action
• Adding documents now is fast
• Keep number of segments small
• Avoid merges on big segments
• Just add new docs (no deletes or updates)
Mixed ArchitecturePutting it all together
15
Slave
Slave
Solr prod replicas
SlaveXML feed
XML feed
Kafka partition
Kafka partition
Storm topologySources
Hbase doc info
Bulk addExists?
MR Pipeline
zk
Mixed ArchitectureSwapping indexes
• NRT docs might not be contained in the new batch index (even fresher than the “being built” batch index)
• This can lead to inconsistencies...
Mixed ArchitectureSwapping indexes. Time jumps!
Mixed ArchitectureSwapping indexes
HBase
XML feed t
Slave t+1
Slave t
Pipeline t
Pipeline t+1
XML feed t+1
XML feed t+2
NRT indexerBatch indexer
Mixed ArchitectureSwapping indexes
HBase
XML feed t
Slave t+1
Slave t
Pipeline t
Pipeline t+1
XML feed t+1
XML feed t+2
NRT indexerBatch indexer
Mixed ArchitectureSwapping indexes
HBase
XML feed t
Slave t+1
Slave t
Pipeline t
Pipeline t+1
XML feed t+1
XML feed t+2
NRT indexerBatch indexer
NRT t+1
NRT t+2
Mixed ArchitectureSwapping indexes
HBase
XML feed t
Slave t+1
Slave t
Pipeline t
Pipeline t+1
XML feed t+1
XML feed t+2
NRT indexerBatch indexer
NRT t+1
NRT t+2
Mixed ArchitectureSwapping indexes
• NRT indexed docs must be stored in a temporary storage
• Fetch missing docs from the storage and add them before the next deploy
• This avoids time jumps
Mixed ArchitectureStorm and Hadoop
• Near real time inserts, low latency
• Hadoop handles deletes and updates. No rush on those
• No merges on big segments so optimal query response times
• Tolerant to human errors
• Temporary lost of accuracy on the NRT layer
AlternativesSolrCloud - Why not?
• Good for the vast majority of use cases
• Incremental inserts/updates/deletes oriented. Pay segment merges per real time
• Need to deploy full indexes fast (faster that rsync or http replication)
• Now full deploy easier with aliases
Future linesLucene real time feature
• Allows to see docs in the index before they are committed
• Good but not a must right now for the use case
• Very easy to integrate on the current architecture
??
Thanks for your attention!
Marc [email protected]
Lucene/Solr Revolution 2013, San Diego, May 1 2013
CONFERENCE PARTYThe Tipsy Crow: 770 5th AveStarts after Stump The ChumpYour conference badge gets you in the door
TOMORROW Breakfast starts at 7:30Keynotes start at 8:30