devopsdays amsterdam 2016 workshop

DevOpsDays 2016Amsterdam

First steps to create your IT Operations data lake with ELK Stack and Graphite

Hands

-on

Works

hop

2

Arnold van Wijnbergen@BSMConsultant / Arnold.van.Wijnbergen@gmail.com

•16 years working experience with automating IT•DevOps Evangelist with great passion about Architecture andAutomation Tooling•Full time Tooling Geek

Automation Monitoring, tasted them all Predictive Analytics Infrastructure-as-Code

#IlovIT ☁

3

INTRODUCE OURSELVES

Who we are and what do we expect

4

CHOOSE YOUR WAY

Dev, Ops or just DevOps

DOD-AMS-WORKSHOP-BANKIT

TOURIST ROUTE

DOD-AMS-WORKSHOP

DANGER ROUTE

JUST ANOTHER DATA LAKE CONCEPT

Metrics, logs, health-states, etc

“ELK Stack + Graphite + Kafka makes a great combination”

ElasticsearchLogstash Kibana

GrafanaGraphiteKafka/ZookeeperJava

Consumer

JavaProducer

BankITLogfile

Topic:Events

Topic:Metrics

Syslog

Main component for managing your element is LogStash

Reference : https://www.elastic.co/guide/en/logstash/current/introduction.html

Logstash Structured way of Working – Processing Pipeline

Reference : https://www.elastic.co/guide/en/logstash/current/pipeline.html

Pipeline processing of an event is mainly existing of three important stages

Reference : https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

(Filter) Processing

Parsefields out

Enrichfields values

CorrelateOn field value matching

Parse unstructured data and make IT readable structured

Enrich & correlate our structured data set

13

Success with the workshop

https://goo.gl/97xOX2

14

Backup slides

Meet Mr LogStashWhat can he do for you ?I Love your data … :-)

Many test options are available to validate your code and expected functional behaviour

Code Validation withCONFIGTEST

Unit Testing with RSPEC

Reference : https://www.elastic.co/blog/logstash-functionality-through-testing

Now combine the forces and create a structured configuration

Configuration folder

Input.conf

Input.conf

Input.conf

Filter.conf

Input.conf

Output.conf

filestdin

mutategrok

stdoutelasticsear

chredis

Reference : https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html