tyingtogetherzabbixand elasticsearch/logstash/kibana(elk ... · tyingtogetherzabbixand...
TRANSCRIPT
![Page 1: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/1.jpg)
Tying together Zabbix andElasticsearch/Logstash/Kibana (ELK) ... and
Grafana, too!
Volker Fröhlich
19 Nov 2015, NLUUG
![Page 2: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/2.jpg)
Problem definition Components Integrating Summary
Who am I?
Volker Fröhlich (volter)Geizhals Preisvergleich Internet Services AG(http://geizhals.at)Zabbix frontend patches, conference, blog, book reviewFedora packager, Openstreetmap contributor
![Page 3: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/3.jpg)
Problem definition Components Integrating Summary
What is this all about?
1 How logs are interesting and difficult
2 Define what we want to achieve3 Explain the setup I am using4 How we can integrate it tighter
![Page 4: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/4.jpg)
Problem definition Components Integrating Summary
What is this all about?
1 How logs are interesting and difficult2 Define what we want to achieve
3 Explain the setup I am using4 How we can integrate it tighter
![Page 5: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/5.jpg)
Problem definition Components Integrating Summary
What is this all about?
1 How logs are interesting and difficult2 Define what we want to achieve3 Explain the setup I am using
4 How we can integrate it tighter
![Page 6: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/6.jpg)
Problem definition Components Integrating Summary
What is this all about?
1 How logs are interesting and difficult2 Define what we want to achieve3 Explain the setup I am using4 How we can integrate it tighter
![Page 7: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/7.jpg)
Problem definition Components Integrating Summary
What logs can contain
Operational messagesPerformance dataEventsError messages, crashesDebugging information
![Page 8: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/8.jpg)
Problem definition Components Integrating Summary
Apache access log
10.0.0.137 - - [06/Nov/2015:01:01:07 +0100]"GET / HTTP/1.1" 200 33771"http://www.geizhals.at/""Mozilla/5.0 (X11; Linux x86_64)AppleWebKit/537.36 (KHTML, like Gecko)Ubuntu Chromium/45.0.2454.101Chrome/45.0.2454.101 Safari/537.36"
Message written to a file directlyCustom timestamp, free-formish strings
![Page 9: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/9.jpg)
Problem definition Components Integrating Summary
Postfix
Nov 7 06:59:40 mailserver postfix/smtpd[29789]:C690912483F1: client=example.com[10.1.1.1]
Nov 7 06:59:59 mailserver postfix/smtp[32571]:C690912483F1: to=<[email protected]>,relay=127.0.0.1[127.0.0.1]:10024, delay=18,delays=0.05/0.03/0/18, dsn=2.0.0,status=sent (250 2.0.0 Ok, id=26552-28,from MTA([127.0.0.1]:10025): 250 2.0.0 Ok:queued as 3155B1248447)
A different timestamp formatSyslog contextSome timing informationQueue ids that connect related messages
![Page 10: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/10.jpg)
Problem definition Components Integrating Summary
Cisco ASA
%ASA-1-105006: (Primary) Link status Upon interface interface_name.
%ASA-7-713204: Adding static route forclient address: IP_address
interface_name and IP_address are placeholders
![Page 11: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/11.jpg)
Problem definition Components Integrating Summary
Apache 2.4 error logs
AH00940: %s: disabled connection for (%s)"AH01408: Zlib: %d bytes of garbage at the
end of "compressed stream."
![Page 12: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/12.jpg)
Problem definition Components Integrating Summary
IP tables
Oct 4 01:14:19 debian kernel: IN=ra0 OUT=MAC=00:17:9a:0a:f6:44:00:08:5c:00:00:01:08:00SRC=200.142.84.36 DST=192.168.1.2 LEN=60TOS=0x00 PREC=0x00 TTL=51 ID=18374 DF PROTO=TCPSPT=46040 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0
Mostly key/value, but not completely!
![Page 13: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/13.jpg)
Problem definition Components Integrating Summary
Jira backtrace
2015-11-07 01:11:00,026 Sending mailitem To=’[email protected]’ Subject=’Some subject’From=’null’ FromName=’null’ Cc=’null’ Bcc=’null’ ReplyTo=’null’ InReplyTo=’null’MimeType=’text/plain’ Encoding=’UTF-8’ Multipart=’null’ MessageId=’null’ ERROR anonymousMail Queue Service [atlassian.mail.queue.MailQueueImpl] Error occurred in sending e-mail:To=’[email protected]’ Subject=’Some subject’ From=’null’ FromName=’null’ Cc=’null’Bcc=’null’ ReplyTo=’null’ InReplyTo=’null’ MimeType=’text/plain’ Encoding=’UTF-8’Multipart=’null’ MessageId=’null’
com.atlassian.mail.MailException: javax.mail.SendFailedException: Invalid Addresses;nested exception is:com.sun.mail.smtp.SMTPAddressFailedException: 550 5.1.6 <[email protected]>:Recipient address rejected: User has moved to somewhere else.For more information call Example at +43 123123 or e-mail [email protected]
at com.atlassian.mail.server.impl.SMTPMailServerImpl.sendWithMessageId(SMTPMailServerImpl.java:213)at com.atlassian.mail.queue.SingleMailQueueItem.send(SingleMailQueueItem.java:44)...
![Page 14: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/14.jpg)
Problem definition Components Integrating Summary
What do we want to achieve?
1 Solve real-world problems
2 Keep it simple3 Collect in one place4 Search and analyze5 React upon things automatically6 Improve our current monitoring system
![Page 15: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/15.jpg)
Problem definition Components Integrating Summary
What do we want to achieve?
1 Solve real-world problems2 Keep it simple
3 Collect in one place4 Search and analyze5 React upon things automatically6 Improve our current monitoring system
![Page 16: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/16.jpg)
Problem definition Components Integrating Summary
What do we want to achieve?
1 Solve real-world problems2 Keep it simple3 Collect in one place
4 Search and analyze5 React upon things automatically6 Improve our current monitoring system
![Page 17: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/17.jpg)
Problem definition Components Integrating Summary
What do we want to achieve?
1 Solve real-world problems2 Keep it simple3 Collect in one place4 Search and analyze
5 React upon things automatically6 Improve our current monitoring system
![Page 18: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/18.jpg)
Problem definition Components Integrating Summary
What do we want to achieve?
1 Solve real-world problems2 Keep it simple3 Collect in one place4 Search and analyze5 React upon things automatically
6 Improve our current monitoring system
![Page 19: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/19.jpg)
Problem definition Components Integrating Summary
What do we want to achieve?
1 Solve real-world problems2 Keep it simple3 Collect in one place4 Search and analyze5 React upon things automatically6 Improve our current monitoring system
![Page 20: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/20.jpg)
Problem definition Components Integrating Summary
What is Zabbix?
Classic monitoring systemRelation database backend for config and dataMostly C and PHPServer, proxy, agentHas complex concepts; Permission modelItem, trigger, event, action, operation, ...Supports trapping mechanismsVersatile, but weak with visualizationSOAP JSON APICan be extended and hacked
![Page 21: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/21.jpg)
Problem definition Components Integrating Summary
Zabbix 3.0 frontend
![Page 22: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/22.jpg)
Problem definition Components Integrating Summary
How can we solve the transport problem?
ChallengesThere are many different sources and devices we should coverWe must not stall operationsWe should not lose a lot of messages
Possible solutionsA transport abstraction layer like fluentdSome special agent and shipping protocolProcess on the host and store remotelySome messaging system (Kafka, ...)Zabbix
![Page 23: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/23.jpg)
Problem definition Components Integrating Summary
Why don’t we use Zabbix’ capabilities?
Needs an agent, an active one even!Is file-based (efficiency, permissions)Can only grab complete lines or one single valueIs not very flexible with date formatsIs exclusively POSIX-regex-basedCan not be graphed, except for those single valuesCan not be searched throughBecomes even less interactive and sufficient when crossinghosts
![Page 24: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/24.jpg)
Problem definition Components Integrating Summary
Why not syslog?
Syslog is ubiquitiousSyslog has limitations90% of them are probably irrelevant for you or can be workedaroundNo new technologies, easy to set upLittle resource consumption, robust
![Page 25: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/25.jpg)
Problem definition Components Integrating Summary
![Page 26: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/26.jpg)
Problem definition Components Integrating Summary
What is Rsyslog?
Journald?Modern syslogd implementationTCP, RELPQueuesSupports various output modulesAlso exists for WindowsStructured logging? CEE-enhancement!Nov 17 12:37:31 x250 volker: @cee:{"key":value,
"key2":"utf-8", "key3":{"subkey":value}}
![Page 27: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/27.jpg)
Problem definition Components Integrating Summary
What is Logstash (LS)?
JRuby-based "processing pipe"File based configuration with if-clausesInput – tcpCodec – json_linesFilter – grok, kv, csv, geoip, ...Output – elasticsearch, zabbixJSON
![Page 28: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/28.jpg)
Problem definition Components Integrating Summary
What is Elasticsearch (ES)?
Java-based document storageBuilt on LuceneMeant to easily scale horizontallyNo pre-configured schema necessaryREST HTTP JSON APIPermissions can be difficult
![Page 29: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/29.jpg)
Problem definition Components Integrating Summary
What is Kibana (4)?
NodeJS-based web frontendOnly data source is ESAllows to search with Lucene queriesExposes some of ES’ capabilitiesAttempts to break request length limitsHas no permission model
![Page 30: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/30.jpg)
Problem definition Components Integrating Summary
Example Kibana dashboard
![Page 31: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/31.jpg)
Problem definition Components Integrating Summary
Graylog2, Heka, Splunk?
Graylog2Java and NodeJSOffers processor and frontendOffers live configuration changes and streamsOffers an API and statsUses ES as the backend
HekaGo and Lua
![Page 32: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/32.jpg)
Problem definition Components Integrating Summary
What is Grafana?
Web time series graphing solutionGo and NodeJSVarious data sources, including ES, from 2.5 ongrafana-zabbix by Alexander ZobninHighly customizable graphsTemplated and scripted dashboardsHas a permission model
![Page 33: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/33.jpg)
Problem definition Components Integrating Summary
Example Grafana dashboard
![Page 34: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/34.jpg)
Problem definition Components Integrating Summary
What can be done?
1 Graphing things together2 Navigating with context3 Tagging logs with Zabbix context4 Sending data from LS5 Polling data from ES6 Sending Zabbix events to LS7 Sending deployment events to LS8 Zabbix daemon logs
![Page 35: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/35.jpg)
Problem definition Components Integrating Summary
Graphing things together
Shortcomings in Zabbix graphing and screensKibana only supports ESGrafana has a plugin for ES and ZabbixNone of the three offers a complete sub-set of anotherIt is not a trivial task to "include" one into another
![Page 36: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/36.jpg)
Problem definition Components Integrating Summary
Zabbix versus Grafana
![Page 37: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/37.jpg)
Problem definition Components Integrating Summary
![Page 38: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/38.jpg)
Problem definition Components Integrating Summary
Navigating with context
No interface can handle all your needsMake it easy to navigate between frontendsUse and extend the Zabbix JS menuUse templated and scripted dashboards in Grafana
![Page 39: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/39.jpg)
Problem definition Components Integrating Summary
Early stage of JS menu navigation
![Page 40: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/40.jpg)
Problem definition Components Integrating Summary
Tagging logs with Zabbix context
Assume that Zabbix host groups are relevantOptionally ignore some of themPeriodically poll host group data from APIUse LS "translate" filter pluginhttp://zabbix.org/wiki/Tagging_logstash
![Page 41: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/41.jpg)
Problem definition Components Integrating Summary
Zabbix host groups added
![Page 42: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/42.jpg)
Problem definition Components Integrating Summary
Sending data from LS
LS output plugin "zabbix"Implements Zabbix sender protocolAllows to submit arbitrary data on arbitrary eventsYou must know the Zabbix host nameYou must know the key of an existing trapper itemNo fallback item?Create a trigger with "multiple problem events" and hysteresis?
![Page 43: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/43.jpg)
Problem definition Components Integrating Summary
Polling data from ES
Query using the ES HTTP APIWrite a script that accepts a reference to a JSON objectSet up an according "Simple script" itemSet up a trigger
![Page 44: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/44.jpg)
Problem definition Components Integrating Summary
Sending Zabbix events to LS
Set up a custom scriptSet up an actionNeither Kibana 4 nor Zabbix can visualize themNone of the systems is offering Gantt charts
![Page 45: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/45.jpg)
Problem definition Components Integrating Summary
Sending deployment events to LS
Free-form information with Zabbix context from UIOr deployment hook elsewhereNeither Kibana 4 nor Zabbix can visualize themhttp://zabbix.org/wiki/Docs/comment_for_logstash
![Page 46: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/46.jpg)
Problem definition Components Integrating Summary
Event markers in Grafana showing Git commits
![Page 47: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/47.jpg)
Problem definition Components Integrating Summary
Zabbix daemon logs
Don’t set a log fileSet up syslog daemon and log rotationCould we have monitored Zabbix logs with Zabbix?Works for all components, except JMX gateway?
![Page 48: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/48.jpg)
Problem definition Components Integrating Summary
Summary and outlook
Great benefitGreat potential for improvementTests, automatismsWill everything become easy soon?Will any single interface be enough?Do we need a meta-interface?
![Page 49: TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK ... · TyingtogetherZabbixand Elasticsearch/Logstash/Kibana(ELK)... and Grafana,too! VolkerFröhlich 19Nov2015,NLUUG](https://reader030.vdocuments.site/reader030/viewer/2022021606/5c7445fc09d3f2a80a8bdefe/html5/thumbnails/49.jpg)
Problem definition Components Integrating Summary
Contact information and readings
volter on Freenode [email protected]
Resources#zabbix, #logstash, #elasticsearch, #kibana, #grafanahttp://www.zabbix.org
https://github.com/alexanderzobnin/grafana-zabbix
http://www.logstashbook.com
https://github.com/coolacid/GettingStartedWithELK
http://geofrogger.net/zabbix_elk_nluug.pdf