devopsdays amsterdam 2016 workshop
TRANSCRIPT
DevOpsDays 2016Amsterdam
First steps to create your IT Operations data lake with ELK Stack and Graphite
Hands
-on
Works
hop
2
Arnold van Wijnbergen@BSMConsultant / [email protected]
•16 years working experience with automating IT•DevOps Evangelist with great passion about Architecture andAutomation Tooling•Full time Tooling Geek
Automation Monitoring, tasted them all Predictive Analytics Infrastructure-as-Code
#IlovIT ☁
3
INTRODUCE OURSELVES
Who we are and what do we expect
4
CHOOSE YOUR WAY
Dev, Ops or just DevOps
DOD-AMS-WORKSHOP-BANKIT
TOURIST ROUTE
DOD-AMS-WORKSHOP
DANGER ROUTE
JUST ANOTHER DATA LAKE CONCEPT
Metrics, logs, health-states, etc
“ELK Stack + Graphite + Kafka makes a great combination”
ElasticsearchLogstash Kibana
GrafanaGraphiteKafka/ZookeeperJava
Consumer
JavaProducer
BankITLogfile
Topic:Events
Topic:Metrics
Syslog
Main component for managing your element is LogStash
Reference : https://www.elastic.co/guide/en/logstash/current/introduction.html
Logstash Structured way of Working – Processing Pipeline
Reference : https://www.elastic.co/guide/en/logstash/current/pipeline.html
Pipeline processing of an event is mainly existing of three important stages
Reference : https://www.elastic.co/guide/en/logstash/current/filter-plugins.html
(Filter) Processing
Parsefields out
Enrichfields values
CorrelateOn field value matching
Parse unstructured data and make IT readable structured
Enrich & correlate our structured data set
14
Backup slides
Meet Mr LogStashWhat can he do for you ?I Love your data … :-)
Many test options are available to validate your code and expected functional behaviour
Code Validation withCONFIGTEST
Unit Testing with RSPEC
Reference : https://www.elastic.co/blog/logstash-functionality-through-testing
Now combine the forces and create a structured configuration
Configuration folder
Input.conf
Input.conf
Input.conf
Filter.conf
Input.conf
Output.conf
filestdin
mutategrok
stdoutelasticsear
chredis
Reference : https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html