infrastructure coders logstash

19
logstash Infrastructure Coders Melbourne April 2013 David Lutz @dlutzy

Upload: david-lutz

Post on 11-May-2015

2.146 views

Category:

Technology


5 download

DESCRIPTION

A short talk at Infrastructure Coders Melbourne April 2013 meetup. Covers my first impressions of logstash.

TRANSCRIPT

Page 1: Infrastructure coders logstash

logstash

Infrastructure Coders MelbourneApril 2013David Lutz

@dlutzy

Page 2: Infrastructure coders logstash

What does logstash do?

It does "stuff" with log files.

Page 3: Infrastructure coders logstash

Typical day (or night) in the life of a sysadmin...

Something's wrong.

Check the log files.

How?

Page 4: Infrastructure coders logstash

grep

Page 5: Infrastructure coders logstash

catgrepsedawktailsort

Page 6: Infrastructure coders logstash

and pipes

lots of pipes

Page 7: Infrastructure coders logstash
Page 8: Infrastructure coders logstash

Fine if you have one server. But what if you have 10 or 100 or 1000for i in `seq 1 10` ; do ssh server$i blah blah; done

cluster ssh

Splunk perhaps?

Problems with Splunk...

Page 9: Infrastructure coders logstash

1.eats log files

2. digests data

3. spits it out into other apps

Page 10: Infrastructure coders logstash

inputs● amqp● drupal_dblog● eventlog● exec● file● ganglia● gelf● gemfire● generator● heroku● irc● log4j

● lumberjack● pipe● redis● relp● sqs● stdin● stomp● syslog● tcp● twitter● udp● xmpp● zenoss● zeromq

Page 11: Infrastructure coders logstash

filters● alter● anonymize● checksum● csv● date● dns● environment● gelfify● geoip● grep● grok● grokdiscovery

● grokdiscovery● json● kv● metrics● multiline● mutate● noop● split● syslog_pri● urldecode● xml● zeromq

Page 12: Infrastructure coders logstash

outputs● amqp● boundary● circonus● cloudwatch● datadog● elasticsearch● elasticsearch_http● elasticsearch_river● email● exec● file● ganglia● gelf● gemfire● graphite● graphtastic● http● internal● irc● juggernaut● librato

● loggly● lumberjack● metriccatcher● mongodb● nagios● nagios_nsca● null● opentsdb● pagerduty● pipe● redis● riak● riemann● sns● sqs● statsd● stdout● stomp● syslog● tcp● websocket● xmpp● zabbix● zeromq

Page 13: Infrastructure coders logstash

How to: install logstash

wget http://logstash.objects.dreamhost.com/release/logstash-1.1.9-monolithic.jar

easy!

Page 14: Infrastructure coders logstash

How to: run logstash

java -jar logstash-1.1.9-monolithic.jar agent -f logstash.conf -- web

easy!

Page 15: Infrastructure coders logstash

How to: get some apache logs in

input { tcp { type => "apache" port => 3333 } }

Page 16: Infrastructure coders logstash

How to: get some apache logs in

tail -f /var/log/apache2/access.log | nc localhost 3333

Page 17: Infrastructure coders logstash

How to: digest the logs

filter { grok { type => "apache" pattern => "%{COMBINEDAPACHELOG}" }

date { type => "apache" timestamp => "dd/MMM/yyyy:HH:mm:ss Z" }}

Page 18: Infrastructure coders logstash

How to: output to elasticsearch

output { elasticsearch { embedded => false }}

Page 19: Infrastructure coders logstash

How to: output to elasticsearch and graphite via statsd

output { elasticsearch { embedded => false } statsd { increment => "apache.response.%{response}" }}