osdc 2014: jordan sissel - find happiness in your logs
DESCRIPTION
Got logs? With so much technology powering your business, you need tools to help you identify problems and analyze past behavior. Apache 2.0-licensed Elasticsearch ELK stack is here to help you process, store, and visualize any kind of logging data, in real time, from any source imaginable! Log management seems so boring. Log rotation, retention policy, grep, yuck! What are your servers are doing? Did last night's upgrade break anything? How your users are interacting with your products? Why did the site go down last weekend? Get ready to turn your log pains into awesome visual insights and more! BAM! Elasticsearch ELK! ELK stands for Elasticsearch, Logstash, and Kibana. Each of these three are lovely, open source projects that, together, give you and your business log management superpowers. This talk will primarily be done in three parts: open source and community, technology, and use cases. * The first part will introduce each project and its success as open source software, most notably through supportive and open communities. * The second part will discuss the each project and the problems solved. * The third (and most exciting!) part will highlight a variety of use cases and problem that real humans are using Elasticsearch ELK to solve. Live demos of some use cases will be provided. Attendees will leave the presentation totally full of excitement about this toolset and bursting with fresh ideas about how to tackle their sour logging problems.TRANSCRIPT
![Page 1: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/1.jpg)
Logstashfind happiness in your logs
with Elasticsearch ELK
![Page 2: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/2.jpg)
Open Source
![Page 3: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/3.jpg)
Apache 2.0 License
Open Source
![Page 4: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/4.jpg)
Open and Friendly Community
Open Source
![Page 5: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/5.jpg)
If a new user has a bad time, it’s a bug.
Open Source
![Page 6: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/6.jpg)
Technology
![Page 7: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/7.jpg)
Logstash
![Page 8: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/8.jpg)
Processing & Transport
Logstash
![Page 9: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/9.jpg)
Logstash
FiltersInputs Outputs42 51 54
Processing & Transport
![Page 10: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/10.jpg)
Logstash Processing & Transport
LAMP Example
ApacheMysqlPHP
Syslog
NagiosElasticsearch
Graphite
IRC
Parse: Latency, Bandwidth, and Errors
Load Balancer
![Page 11: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/11.jpg)
Logstash Example Inputs
Files Graphite
SNMPEmail
Syslog
TwitterTCP
RabbitMQ
![Page 12: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/12.jpg)
Logstash Example Filters
Grok Date
FingerprintGeoIPMultiline
User AgentKey-Value
![Page 13: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/13.jpg)
Logstash Example Outputs
Elasticsearch Graphite
XMPPNagios
Email PagerdutyS3
![Page 14: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/14.jpg)
Elasticsearch
![Page 15: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/15.jpg)
Near Real-Time Search & Analysis
Elasticsearch
![Page 16: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/16.jpg)
Scalable
Elasticsearch
![Page 17: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/17.jpg)
REST + JSON API
Elasticsearch
![Page 18: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/18.jpg)
Kibana
![Page 19: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/19.jpg)
Visualization & Exploration
Kibana
![Page 20: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/20.jpg)
![Page 21: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/21.jpg)
Use Cases
![Page 22: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/22.jpg)
Share Logs with Your Tech Support Team
Use Cases
![Page 23: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/23.jpg)
Graph all things!
Use Cases
![Page 24: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/24.jpg)
Exploration by Non-Technical Users
Use Cases
![Page 25: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/25.jpg)
Troubleshooting Latency
?Late
ncy
Time
X
![Page 26: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/26.jpg)
Movie Releases Demo
Use Cases
![Page 27: OSDC 2014: Jordan Sissel - Find Happiness in your Logs](https://reader034.vdocuments.site/reader034/viewer/2022052618/554961abb4c905f24e8b5a67/html5/thumbnails/27.jpg)
Apache Logs Demo
Use Cases