using elastic to monitor everything - christoph wurm, elastic - devopsdays tel aviv 2016
TRANSCRIPT
BATTLE PLAN
15:05 – 15:40 SLIDES
15:40 – 15:55 COFFEE
15:55 – 16:55 HANDS ON
IS A PLATFORM
Unstructured Data Structured Data
Search Logging Security MetricsAnalytics
Social SensorLogsDocuments UserActivity Location
Elastic Cloud
Security
X-Pack
KibanaUser Interface
ElasticsearchStore, Index,& Analyze
IngestLogstash Beats
+
Elastic Stack
Alerting
Monitoring
Reporting
Graph
DATA STOREFAST
SCALEABLE
FULL TEXT
Numbers need 60% less memory in 5.0
USER INTERFACEVISUALIZATIONS
DASHBOARDS
REAL TIME
CHARTING LANGUAGE (5.0)
DATA PROCESSING
REAL TIME PIPELINE
200+ PLUGINS
SYSLOGREGEXMULTILINELOOKUPSGEOIP
DATA COLLECTIONAT THE SOURCE
LIGHTWEIGHT
REAL TIME
Preconfigured Dashboards
FilebeatTail files
PacketbeatCapture network packets
MetricbeatSystem (Topbeat), Apache,Docker, HAProxy, MongoDB,MySQL, Nginx, PostgreSQL, Redis, Zookeeper
WinlogbeatWindows event logs
apachebeathttpd server-status
burrowbeatkafka consumer lag
cassandrabeatnodetool cfstats
cloudtrailbeataws events
dockerbeatcontainer stats
elasticbeatelasticsearch status
execbeatshell cmd output
flowbeatsflow
factbeatpuppet facter
httpbeatpoll http(s) endpoints
jmxproxybeattomcat jmx metrics
lmsensorsbeattemp, voltage, fans
mysqlbeatany query -> elasticsearch
packagebeatdpkg, rpm
pingbeaticmp ping w/ rtt
redisbeatredis info
twitterbeattweets by screen namewmibeat
windows monitoring
https://www.elastic.co/guide/en/beats/libbeat/master/community-beats.html
IN PRODUCTION
SECURITY
MONITORING
REPORTING
ALERTING
MACHINE LEARNING
GRAPH
DEPLOYING
Beats
Log Files Metrics
Wire Data
Elasticsearch Kibana
Master / Data Nodes (3)
Ingest Nodes (X)
your{beat}X-Pack X-Pack
SimpleBeats to Ingest Node
Instances (X)
Beats Logstash Elasticsearch Kibana
X-Pack X-Pack
Additional Processing
Nodes (X)
Instances (X)
Master Nodes (3)
Ingest Nodes (X)
Data Nodes – Hot (X)
Data Notes – Warm (X)
Log Files Metrics
Wire Data your{beat}
LDAP
Authentication
AD
Notification
SSO
Deployment at Scale
Kibana
X-Pack
Instances (X)
Beats
Log Files Metrics
Wire Data
Datastore Web APIs
Social Sensors
Kafka
Redis
MessagingQueue
Logstash
Elasticsearch
Nodes (X)
Master Nodes (3)
Ingest Nodes (X)
Data Nodes – Hot (X)
Data Notes – Warm (X)
your{beat}
X-Pack
LDAP
Authentication
AD
Notification
SSO
Beats
Log Files Metrics
Wire Data
Datastore Web APIs
Social Sensors
Kafka
Redis
MessagingQueue
Logstash
ES-Hadoop
Elasticsearch
Kibana
Nodes (X)
Master Nodes (3)
Ingest Nodes (X)
Data Nodes – Hot (X)
Data Notes – Warm (X)
Instances (X)
your{beat}
X-Pack X-Pack
Custom UI
LDAP
Authentication
AD
Notification
SSO
Deployment in the Enterprise
Hadoop Ecosystem
HANDS ON
Metricbeat
Packetbeat
Metricbeat
Packetbeat
Metricbeat
Packetbeat
Metricbeat
PacketbeatTEL AVIV
DUBLIN +
github.com/cwurm/devopsdays-telavivElasticsearch http://tinyurl.com/devopsdays-elasticsearch:9243
Kibana http://tinyurl.com/devopsdays-kibana
Credentials devops / telaviv2016