logs aggregation and analysis
TRANSCRIPT
Logs aggregation
and analysis
Agenda
● Problems with logs
● How do we do it at Divante - ELK Stack o ElasticSearch
o Logstash
o Kibana
o Architecture
o Additional tools
● Summary
● Questions
What problems with logs
may we encounter?
Problems with logs
No consistent log format
http://blog.tersmitten.nl/how-to-colorize-your-log-files-with-ccze.html
Problems with logs
• cat
• grep
• awk
• sed
• tail
• regular expressions
Hampered log analysis, increased response time.
Log search, analysis – old school
Problems with logs
Heavy server load, low application performance
• Synchronous, blocking writing
• IO operations burdening the server
• Limited amount of inodes in the file system
• Relatively slow write speed
• Logs cleanup
http://wiki.processmaker.com/index.php/Advanced_Performance_Monitor_Dashboards
Problems with logs
Complex architecture - read and write problem
• Read/write on servers after the load balancer
- NFS application?
https://www.digitalocean.com/community/tutorials/5-common-server-setups-for-your-web-application
Problems with logs
Continuous monitoring
• Continuous monitoring of all application parameters is not easy
• Technical knowledge is required
How do we do it at Divante?
ELK Stack
ELK Stack - what is it?
ELK Stack is a set of tools:
It provides a centralized log management in distributed, high-availability systems.
E as in ElasticSearch
ElasticSearch
● + NoSQL database
● + Full-text search
● + REST Api (json)
● + Based on Apache Lucene
● + Replication, snapshots
● + Official PHP and JavaScript libraries available
● - No transactions
● Requirements: Java
E as in ElasticSearch
Installation and setup
● Java Oracle installation
o sudo apt-get install python-software-properties
o sudo add-apt-repository -y ppa:webupd8team/java
o sudo apt-get update
o sudo apt-get -y install oracle-java8-installer
● ElasticSearch installation
o wget and unzip of the latest distribution available at
https://www.elastic.co/downloads
o Setup in the elasticsearch.yml file
● Running ElasticSearch:
o ./bin/elasticsearch -d
L as in Logstash
● Aggregation of logs from multiple sources
● Normalization (parsing, filtering) of the collected data
● Sending the normalized data to various sources
● Requirements: Java
INPUT FILTER
PARSE OUTPUT
L as in Logstash
Input: (41)
● elasticsearch, imap
● rabbitmq, redis
● sqlite, syslog
● tcp, twitter
● varnishlog, websocket
Output: (55) ● elasticsearch, mongodb,
● redis, email, file, csv
● hipchat, http
● jira, redmine, rabbitmq
● tcp, websocket, zabbix
More at: http://logstash.net/docs/1.4.2/
INPUT FILTER
PARSE OUTPUT
L as in Logstash
Filters: (50)
● checksum
● csv, date
● elasticsearch
● fingerprint
● geoip, grep, grok
● json, json_encode
● ruby, split
● translate
● urldecode
● useragent, xml
More at: http://logstash.net/docs/1.4.2/
INPUT FILTER
PARSE OUTPUT
L as in Logstash
GROK filter
● Parsing and analysis of any text
● Grok is the best way to process even the most unstructured data
● Over 120 samples and possibility to create new ones
[ERROR] - 2015/13/03-11:01:31 - 192.168.0.1 - Some error message
[%{GREEDYDATA:message}] - %{DATESTAMP} - %{IP:client} - %{GREEDYDATA:message}
L as in Logstash
Installation and setup
● Java Oracle and Logstash installation
o wget and unzip of the latest distribution available at
https://www.elastic.co/downloads
● Setup:
input {
file {
type => "syslog"
path => ["/var/log/auth.log", "/var/log/syslog"]
}}
● Running Logstash:
o ./bin/logstash
K as in Kibana
● Data visualization in the form of a web app
● Data search, filters and analysis
● Intuitive interface, not only for the programmers
● Instant sharing and embedding of multiple dashboards
● Easy dashboard adjustment - JSON
● Export of the results
K as in Kibana
https://www.elastic.co/blog/kibana-4-literally
K as in Kibana
Bar chart:
https://www.elastic.co/blog/kibana-4-beta-2-get-now
K as in Kibana
Pie chart:
https://www.elastic.co/blog/kibana-4-for-investigating-pacs-super-pacs-and-your-neighbors
K as in Kibana
Histogram:
http://blog.qbox.io/kibana-4-and-elasticsearch-v-1-4-4-and-1-3-9
K as in Kibana
Data table:
https://www.elastic.co/blog/kibana-4-for-investigating-pacs-super-pacs-and-your-neighbors
K as in Kibana
Geolocation:
https://www.elastic.co/blog/kibana-4-literally
K as in Kibana
Relation chart:
http://demo.packetbeat.com/#/dashboard/elasticsearch/Packetbeat%2520Statistics
K as in Kibana
Installation and setup
● Kibana installation:
o wget and unzip of the latest distribution available at
https://www.elastic.co/downloads
● Setup:
o Kibana is set up on the default ElasticSearch port - 9300
● Running Kibana:
o ./bin/kibana
● Start in a browser:
o http://YOURDOMAIN.com:5601
ELK Stack
ARCHITECTURE
ELK - Architecture
Basic:
What is the potential problem?
ElasticSearch REST API
ELK - Architecture
Performance + scalability Setup with Redis
Providers
Indexer
ELK - Architecture
Performance + scalability Setup with the use of a queuing mechanism
Indexer
Providers
Additional
TOOLS
Tools
ElasticSearch Plugin: Head ElasticSearch monitoring and management panel
Installation:
● elasticsearch/bin/plugin -install mobz/elasticsearch-head
● http://localhost:9200/_plugin/head/
Tools
ElasticSearch Plugin: Morfologik Plugin enabling the use of Polish characters
in ElasticSearch queries
Installation:
● cd elasticsearch
● bin/plugin -install com.github.chytreg/elasticsearch-analysis-
morfologik/2.3.1
Tools
Marvel Application for the continuous monitoring of ElasticSearch
http://blog.qbox.io/elasticsearch-marvel-released
ELK Stack
Summary
● Consistent log structure
● Centralized log system
● Dashboard supporting the search and analysis processes
● Continuous monitoring
● Reduced response time
● Happy customer receives new charts :)
VS
http://blog.rootshell.be/tag/unix/
http://imgbuddy.com/kibana-demo.asp
