Download - Elk with Openstack
ELK with Openstack
Arun prasath SJune 16, 2016
Challenges in log analysis
• Multiple services • Multiple servers behind load balancers• Searching the logs (cat, tail, sed, grep, awk)• Finding logs in particular time in multiple servers• Finding fields (Instance ID, name, IP address) in multiple servers and
correlating them• Log analysis , summary, visualization
ELK user operation demo
• Performing a normal search• Filtering based on time, fields• Viewing document data• Viewing field data statistics• Visualize data• Dashboards
ELK Architecture
Logstash forwarder
Servers
Logstash(Indexer)
Storage and search
Dashboard
Broker
• Temporary buffer between logstash agents and central server• Enhance performance by providing caching buffer for log events• Adds resiliency
• Incase the indexing fails, the events are held in queue instead of getting lost
Logstash
• Runs on JVM• Multiple Input / Multiple output• Centralize logs• Collect• Parse• Store / Forward• 3 components – Input, Filter and Output
Logstash conf - /etc/logstash/conf.d
input {}
filter{}
output{}
Logstash - Input
• Input plugin enables a specific source of events to be read by Logstash.
• Some examples of input• Beats• File• Stdin• Eventlog• More here
Logstash - Filter
• A filter plugin performs intermediary processing on an event. Filters are often applied conditionally depending on the characteristics of the event.
• Some examples are• Csv• Date• Grok• Json• More here
Logstash - Output
• An output plugin sends event data to a particular destination.• Some examples are
• Csv• redis• elasticsearch• File• Jira, Nagios, pagerduty• stdout• More here
Logstash - codec
• A codec plugin changes the data representation of an event• Some examples are
• Collectd - Reads events from the collectd binary protocol using UDP• Graphite - Reads graphite formatted lines• Json - Reads JSON formatted content, creating one event per element in a JSON array• Plain - Reads plaintext with no delimiting between events• rubydebug - Applies the Ruby Awesome Print library to Logstash events
• More here
Example - 1 (stdin input / rubydebug output)
input {stdin {}
}filter{}output{stdout { codec => rubydebug }}
Example – 2 (Basic filter on syslog with stdin input and rubydebug output)
input {stdin {}
}filter{grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]add_field => [ "received_from", "%{host}" ]
}syslog_pri { }date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]}
}output{stdout { codec => rubydebug }}
Example – 3 (Openstack filters with file input and multiple output)
input {file {
path => ['/var/log/nova/nova-api-metadata.log']tags => ['nova', 'oslofmt', 'novametaapi']type => "nova"
}…}filter{…else if "novaapi" in [tags] {
if [module] == "nova.osapi_compute.wsgi.server" {mutate {
gsub => ['logmessage',"\"",""]}grok {
match => { "logmessage" => "\[req\-%{NOTSPACE:requestid} %{NOTSPACE:user_id} %{NOTSPACE:tenant}\] %{NOTSPACE:requesterip} %{NOTSPACE:method} %{NOTSPACE:url_path} %{NOTSPACE:http_ver} status\: %{NUMBER:response} len\: %{NUMBER:bytes} time\: %{NUMBER:seconds}" }
add_field => ["api", "nova"]add_tag => ["apimetrics"]
}…}output {
elasticsearch { hosts => ["localhost:9200"] }stdout { codec => rubydebug }
}
Ref: https://github.com/bingoarunprasath/logstash-openstack-filters
Elasticsearch
• Searching and storing of logs• Built on Apache lucene (https://lucene.apache.org/core/)• Massively distributed• High availability• Developer friendly , RESTful API
Kibana
• Dashboard• Provides various options to search data• Creates bar charts, pie charts and various other data visualizations. • Can create custom dashboard and add the saved visualizations. • Simple data export
Installation notes
• Install Java• Install elasticsearch and Kibana
• Install nginx for reverse proxy and basic AUTH• Install logstash, generate SSH certificates
• Configure one output to elasticsearch• Load Kibana dashboard• Setup filebeat / logstash in agent machines and output to logstash• Ansible role - https://galaxy.ansible.com/bingoarunprasath/elk/• Filters - https://github.com/bingoarunprasath/logstash-openstack-
filters