batch indexing & near real time, keeping things fast

Batch Indexing & Near Real Time, keeping things fast

Marc SturleseSoftware engineer @ Trovit

Thursday, 2 May 2013

About me...

• Marc Sturlese – @sturlese

• Software engineer @Trovit. R&D focused

• Responsible for search and scalability


Agenda

• Who we are

• Batch architecture. Hadoop & Hive

• Near real time architecture. Storm & stuff

• Putting it all together

• Alternatives and Future directions

• Questions


Who we are

Trovit, a search engine for classifieds


Who we are


Batch Layer

• Hadoop based

• Documents are crunched by a pipeline of MR jobs

• Hive to save stats of each phase


Batch LayerPipeline overview

Incoming data

Deployment

Lucene Indexes

Ad Processor Diff Matching Expiration Deduplication Indexing

t – 1

External Data

Hive Stats

Hadoop Cluster


Batch LayerThe good things!

• Index always built from scratch. Small number of big segments

• Multicast deployment allows to send indexes to all slaves at the same time.

• Backups convenient on HDFS


Batch LayerThat was cool but...

• Not even close to real time

• Crunch documents in batch means to wait until all is processed. This can take a few hours

• We want to show the user fresher results!


Near real time LayerStorm and stuff to the rescue


Near real time LayerStorm properties

• Distributed real time computation system

• Fault tolerance

• Horizontal scalability

• Low latency

• Reliability


Near real time LayerStorm in action

Slave

Slave

Solr prod replicas

SlaveXML feed

XML feed

Kafka partition

Kafka partition

Storm topologySources

Kafka spout

Kafka spout

XML spout Doc Manager bolt Indexer bolt

SHUFFLEGROUPING GROUPING

FIELD



• Spouts just read and send

• Doc Manager Bolt processes and classifies

• Indexer Bolt adds documents to Solr

• Replicated logic with different implementation

• Careful not to overload Solr slaves...


Near real time LayerStorm in action. But...


Near real time LayerStorm in action. But...

• Now Solr has to handle user queries and storm inserts

• Field grouping on Indexer Bolt for politeness

• Small bulks to reduce insert requests

• Committing on many cores, same host, same time can be painful


Near real time LayerStorm in action - Committing

Indexer Bolt Cars US

Real state UK R1 Cars US R1 Cars US R2 Jobs BR R1 Jobs BR R2 Real state ES R1

Indexer Bolt Jobs BR

ZooKeeper Locker

Slave 1 Slave 2 Slave N

. . .



• Adding documents now is fast

• Keep number of segments small

• Avoid merges on big segments

• Just add new docs (no deletes or updates)


Mixed ArchitecturePutting it all together

15

Slave

Slave

Solr prod replicas

SlaveXML feed

XML feed

Kafka partition

Kafka partition

Storm topologySources

Hbase doc info

Bulk addExists?

MR Pipeline

zk


Mixed ArchitectureSwapping indexes

• NRT docs might not be contained in the new batch index (even fresher than the “being built” batch index)

• This can lead to inconsistencies...


Mixed ArchitectureSwapping indexes. Time jumps!



HBase

XML feed t

Slave t+1

Slave t

Pipeline t

Pipeline t+1

XML feed t+1

XML feed t+2

NRT indexerBatch indexer



HBase

XML feed t

Slave t+1

Slave t

Pipeline t

Pipeline t+1

XML feed t+1

XML feed t+2

NRT indexerBatch indexer

NRT t+1

NRT t+2



• NRT indexed docs must be stored in a temporary storage

• Fetch missing docs from the storage and add them before the next deploy

• This avoids time jumps


Mixed ArchitectureStorm and Hadoop

• Near real time inserts, low latency

• Hadoop handles deletes and updates. No rush on those

• No merges on big segments so optimal query response times

• Tolerant to human errors

• Temporary lost of accuracy on the NRT layer


AlternativesSolrCloud - Why not?

• Good for the vast majority of use cases

• Incremental inserts/updates/deletes oriented. Pay segment merges per real time

• Need to deploy full indexes fast (faster that rsync or http replication)

• Now full deploy easier with aliases


Future linesLucene real time feature

• Allows to see docs in the index before they are committed

• Good but not a must right now for the use case

• Very easy to integrate on the current architecture


??Thursday, 2 May 2013

Thanks for your attention!

Marc [email protected]

Lucene/Solr Revolution 2013, San Diego, May 1 2013


mailto:[email protected]

mailto:[email protected]

CONFERENCE PARTYThe Tipsy Crow: 770 5th AveStarts after Stump The ChumpYour conference badge gets you in the door

TOMORROW Breakfast starts at 7:30Keynotes start at 8:30


batch indexing & near real time, keeping things fast

Technology