centralised logging with elk stack

Centralised Logging

with ELK

Credit: Pasakuru76https://flic.kr/p/8K1iYi

simon.hanmer@gmail.com simonhanmer

# whoami• Simon Hanmer– IT Consultant– Sysadmin, Infrastructure architect,

server wrangler.

simon.hanmer@gmail.com simonhanmer

simon.hanmer@gmail.com simonhanmer

Why?• Lots of log files– Server– Applications–Network

• Different formats

• Multiply by many servers

simon.hanmer@gmail.com simonhanmer Credit: Kuhnmihttps://flic.kr/p/jbAnNa

What my brain feels like after trying to parse too many logs

simon.hanmer@gmail.com simonhanmer

Elasticsearch• Indexing and search engine• Near real-time• Distributed, auto-discover clustering– AWS Plugin

simon.hanmer@gmail.com simonhanmer

Logstash• Collects logs• Parses, extracts and formats data• Passes data to Elasticsearch

simon.hanmer@gmail.com simonhanmer

Logstash - examplefilter { if [file] == "/var/log/secure" and ( [syslog_message] =~ /Invalid user/ or [syslog_message] =~ /User root from/ ) { grok { add_tag => [ "LOGIN" ] match => { "syslog_message" => “user %{WORD:username} from %{IP:srcip}" } } }

}

simon.hanmer@gmail.com simonhanmer

Kibana• Web interface to query Elasticsearch• node.js

simon.hanmer@gmail.com simonhanmer

Process flow

simon.hanmer@gmail.com simonhanmer

AWS Architecture

simon.hanmer@gmail.com simonhanmer

Demo

simon.hanmer@gmail.com simonhanmer

What next?• elastic.co

– All components of stack available.