Upload File

elastic search: lessons learned

prev
next
out of 22
Download Elastic Search: Lessons Learned

Upload: magina143

Post on 14-Apr-2018

222 views

Category:

Documents


0 download

Report

TRANSCRIPT

  • 7/29/2019 Elastic Search: Lessons Learned

    1/22

    ElasticSearch in

    Productionlessonslearned

    Anne Veling, ApacheCon EU, November 6, 2012

  • 7/29/2019 Elastic Search: Lessons Learned

    2/22

    agenda

    ! Introduction! ElasticSearch! Udini! Upcoming Tool! Lessons Learned

  • 7/29/2019 Elastic Search: Lessons Learned

    3/22

    introduction

    ! Anne Veling, @anneveling! Self-employed contractor

    !Software Architect! Agile process management

    ! Performance optimization! Lucene/SOLR/ElasticSearch implementations & training

  • 7/29/2019 Elastic Search: Lessons Learned

    4/22

    ElasticSearch

    ! Apache Lucene! Started in 2010 by Shay Banon! Open Source Apache License! A company was formed in 2012: ElasticSearch

    ! Training, support and development! Careful feature development! vs. build because you can

    @kimchy

  • 7/29/2019 Elastic Search: Lessons Learned

    5/22

    ElasticSearch

    ! Scalable! Distributed, Node Discovery! Automatic sharding! Query distribution

    ! RESTful, HTTP API! With API wrappers for Ruby, Java, Scala, ! JSON in, JSON out

    ! Document Model! Maps book.author.lastName! schemaless -> field type recognition! Keeps source, keeps version number

  • 7/29/2019 Elastic Search: Lessons Learned

    6/22

    ElasticSearch

    ! Integrated faceting! With statistical aggregates (sum/avg/) for free

    !Field types and analyzers! String, numerics, geo, attachment, ! Arrays, subdocuments, nested documents

    ! Integrated sharding! Routing and alias! Cross-index searching / multi-document type

  • 7/29/2019 Elastic Search: Lessons Learned

    7/22

  • 7/29/2019 Elastic Search: Lessons Learned

    8/22

    architecture

    mongo

    DB

    Elastic

    Search

    Udini

    SOLR

    Summon

    PDF pipeline

    FulfillmentProviders

  • 7/29/2019 Elastic Search: Lessons Learned

    9/22

    SOLR at Udini

    ! Connecting to Summon API! 700M SOLR Cluster

    ! In Udini, we serve a subset of 160M full text articles! Including fulfillment mechanisms! PDF and HTML5 viewing and annotation

  • 7/29/2019 Elastic Search: Lessons Learned

    10/22

    ElasticSearch at Udini

    ! Local index to search your articles! Many small user libraries, searching only locally

    ! User-id as sharding key! Include key in all queries

  • 7/29/2019 Elastic Search: Lessons Learned

    11/22

    Exciting new product

    ! Developing for ProQuest! Exciting new research tool for scientific researchers! Creating a large ElasticSearch index for journal article

    canonicalization

    ! Currently in private beta, launching in the coming months

  • 7/29/2019 Elastic Search: Lessons Learned

    12/22

    Lessons Learned

    ! Very fast indexing! Bulk indexing ftw

    ! Set up without replicas (replicas = 0, not 1)! Play with bulk size! Simple write to disk and CURL it in, is very fast! 1M records in 40s

    forfin${BATCH_DIR}/batch-*.json

    doecho"abouttoindex$f"

    curl--silent--show-error--requestPOST

    --data-binary@$flocalhost:9200/_bulk>/dev/null

    echodone

  • 7/29/2019 Elastic Search: Lessons Learned

    13/22

    Lessons learned

    ! Schema(less)?! Automatic field type recognition

    ! Can miss types! Strict about types #duh

    ! Mapping of subfields (doc.title vs doc.publication.title)! Version dependent

    ! In reality! Schema still needed! Mapping changes still non trivial

  • 7/29/2019 Elastic Search: Lessons Learned

    14/22

    Lessons learned

    ! Learn to trust ElasticSearch! Analyzers: do not pretokenize queries yourself

    ! Difference between term and text type queries! tokenized or not

    ! ElasticSearch probably already does what you want it to do! Search for it! Try it

  • 7/29/2019 Elastic Search: Lessons Learned

    15/22

    Lessons learned

    ! Issues with automated testing and node discovery/startup! Start/stop hundreds of times during Jenkins test jobs or

    development boxes

    ! Takes time! Locally sometimes picks up previous versions

    ! Memory issues: ElasticSearch manages a large part of itsmemory outside of the heap! Do not simply increase -Xmx

  • 7/29/2019 Elastic Search: Lessons Learned

    16/22

    Lessons learned

    ! New tools every month! waitForYellowStatus! Aliases, routing allow for clever control

  • 7/29/2019 Elastic Search: Lessons Learned

    17/22

    API

    ! ElasticSearch is new, connection libraries still in infancy,documentation growing

    ! Issues using the Java API in Scala! Happy with Scalastic now

    ! synchronous! asynchronous! bulk prepare https://github.com/bsadeh/scalastic

  • 7/29/2019 Elastic Search: Lessons Learned

    18/22

    #nodb

    ! ElasticSearch used as a full nosql datastore?! Using version and optimistic locking scheme! Could replace MongoDb in our setup

    ! ElasticSearch is actually a store optimized for getting stuffout, not for getting stuff in! With free faceting! Who needs multi-table transactions anyway?

  • 7/29/2019 Elastic Search: Lessons Learned

    19/22

    SOLR vs ElasticSearch

    ! SOLR! Well-known, many tools,

    extensions

    ! Feels clunky to configure! Manual document to

    lucene mapping

    ! Replication and indexingin a cluster non-trivial

    ! Old school ;-)

    ! ElasticSearch! New kid on the block! Very easy to configure!

    Handles document tolucene mapping

    ! Horizontally scalable! Easy replication! But: shard key

    ! New school

  • 7/29/2019 Elastic Search: Lessons Learned

    20/22

    search evolution Custom indexers Inverted index Segment merges Custom analyzers Faceting Configuration of analyzers Faceting, Geospatial Document mapping Sub-document queries Replication JSON document input Faceting, complex queries just work

  • 7/29/2019 Elastic Search: Lessons Learned

    21/22

    conclusions

    ! ElasticSearch benefits! Easy to setup! Very clever architecture

    ! Drawbacks! Very new software, tool support limited

    ! But lots of movement! Change sharding in a full index non-trivial

    ! ElasticSearch! Clever architecture, fast, stable! Does exactly what you need

  • 7/29/2019 Elastic Search: Lessons Learned

    22/22

    thank you

    [email protected]

    @anneveling

    AreyoustillusingSolr?

    Comeon,its2012already;-)


Lessons Learned Events at SLAC · 2012-09-14 · Lessons Learned Events: Summary Comments . Important to identify and communicate lessons learned! Lessons learned and near misses

Lessons learneD

Lessons Learned Remote PM+ Cohort Integrated K+S · Lessons Learned Below are key lessons learned on providing Remote PM+ Training of Helpers. These lessons learned are based off

Lessons learned & not learned at MSU

Lessons Learned

Lessons learned from accident investigation Seminar/IHS - Day 2... · LESSONS LEARNED FROM OTHERS LESSONS LEARNED FROM UAE ... Lessons Learned . ... Dealing with survivors and bereaved

Lessons Learned from “Lessons Learned” · 2 LESSONS LEARNED FROM “LESSONS LEARNED” 2. The NRC set safety goals in its Safety Goal Policy Statement, initiated not long after

Successes, Challenges, & Lessons Learned...Successes, Challenges, & Lessons Learned Overview Introduction Successes Challenges Lessons Learned New Approaches Highway 70 To Phoenix

Lessons Learned and Lessons to Be Learned: Investment Law & … · 2014. 4. 30. · 1 Lessons Learned and Lessons to Be Learned: Investment Law & Development for Developed Countries

Lessons&Learned&in&Deploying&Akka& Persistencedownloads.typesafe.com/website/presentations/lessons-learned-akk… · Lessons&Learned&in&Deploying&Akka& Persistence Duncan&K.&DeVore

FRP GIRDER BRIDGES: LESSONS LEARNED IN SPAIN … Papers/Paulotto.… · FRP GIRDER BRIDGES: LESSONS LEARNED IN S ... compounded by a concrete ... the elastic range during earthquakes

LESSONS LEARNED AND GOOD PRACTICES...Lessons Learned Lessons are learned for sharing and applying to similar projects. Project lessons are learned from experiences, both inspiring

Lessons Learned from Failures Dec 2014 - MTS Houston · 212 on board, 123 perished, 89 survivors ... Lessons Learned Lessons Learned. ... Lessons Learned from Failures Dec 2014.pptx

Lean lessons learned - lean startup methodology and lessons learned from Blossom

Lessons Learned Globally - Campaign for Tobacco-Free Kids...Process lessons learned 2. Content lessons learned See next page for lists of these lessons learned. 7 | Lessons Learned

LEARNED LESSONS:

Lessons Learned:

Lessons Learned Handbook · 2011-10-12 · • Lessons Learned, an adjective, describes anything related to a Lessons Learned procedure. E.g. Lessons Learned process, Lessons Learned

ASSESSMENT & LESSONS LEARNED FROM AFRICAN DIASPORA MARKETPLACEpdf.usaid.gov/pdf_docs/PDACT256.pdf · ASSESSMENT & LESSONS LEARNED FROM AFRICAN ... ASSESSMENT & LESSONS LEARNED FROM