solr performance monitoring with spm
DESCRIPTION
Originally delivered as Lightning Talk at Lucene Eurocon 2011 in Barcelona, this quick presentation shows how to use Sematext's SPM service to monitor Solr, OS, JVM, and more.TRANSCRIPT
Solr Performance Monitoringwith
Scalable Performance Monitoring SaaS
Otis Gospodnetić – Sematext International@otisg ◦ @sematext ◦ sematext.com
sematext.com/spm
Copyright 2011 Sematext Int'l. All rights reserved.2
SPM Overview
SPM = Scalable Performance Monitoring SaaS
Small client-side agent attached to the JVM Collectd for OS, FS, and other metrics Near Real Time Distributed backend: Hadoop, HBase, Flume, etc.
Support: Solr, HBase, etc.
Copyright 2011 Sematext Int'l. All rights reserved.3
SPM Functionality
Look at one or multiple Solr instances, Request Handlers, or Solr Cores at once
See performance of any time period with no loss of resolution
Show/hide extended time periods Compare two time periods Monitor multiple Solr clusters Get Alerts (soon)
Copyright 2011 Sematext Int'l. All rights reserved.4
Solr Metrics
All Solr metrics and more: All cache metrics (hit %,evictions, etc.) Query rate and latency Warmup time Latency breakdown by SearchComponent (soon) Latency breakdown by classes & methods (soon) Index size on disk, # files, # segments, # docs, #
deletions
Copyright 2011 Sematext Int'l. All rights reserved.5
OS Metrics
OS metrics CPU Memory Load Disk IO by device Network IO by device Swap (soon) Disk usage (soon)
Copyright 2011 Sematext Int'l. All rights reserved.6
JVM Metrics
Garbage Collection Major & Minor Counts Time spent in GC
Thread count Memory
Heap memory Non-heap memory Memory pools (soon)
Copyright 2011 Sematext Int'l. All rights reserved.7
Lucene / Solr Index
segment merge
# deleted docs
Copyright 2011 Sematext Int'l. All rights reserved.8
Solr Caches
see the increase!
did you know you had these dips?
Copyright 2011 Sematext Int'l. All rights reserved.9
Search Rate & Latency
Select 1 or more Solr instancesSelect 1 or more Request Handlers
Select any time periodsCompare 2 time periods
Select data granularity
Copyright 2011 Sematext Int'l. All rights reserved.10
Warmup
Warmup time broken by warmup type – caches and searcher
Select 1 or more Solr Cores
Copyright 2011 Sematext Int'l. All rights reserved.11
OS & FS
Copyright 2011 Sematext Int'l. All rights reserved.12
JVM
Copyright 2011 Sematext Int'l. All rights reserved.13
Open Sourcing
2 open-source projects:
github.com/sematext/HBaseWD
github.com/sematext/HBaseHUT See sematext.com/open-source/index.html
Patches for Flume and HBaseblog.sematext.com/tag/flume/
Copyright 2011 Sematext Int'l. All rights reserved.14
About Sematext
Products & Services
Consulting, Development, Tech Support:
Search (Lucene, Solr, ElasticSearch...) Big Data (Hadoop, HBase, Voldemort...) Web Crawling (Nutch, Droids) Machine Learning (Mahout)
Copyright 2011 Sematext Int'l. All rights reserved.15
Sematext Metrics
100% organic: no GMO, no VC 4 years old < 10 people 7 countries 3 timezones 2 continents > 100 customers
Copyright 2011 Sematext Int'l. All rights reserved.16
We're Hiring
Dig Search?
Dig Analytics?
Dig Big Data?
Dig Performance?
Dig working with and in open-source?
We're hiring world-wide!
sematext.com/about/jobs.html
Copyright 2011 Sematext Int'l. All rights reserved.17
sematext.com blog.sematext.com @sematext @otisg [email protected]
sematext.com/spm
Contact