application logging with logstash
TRANSCRIPT
![Page 1: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/1.jpg)
Application Logging With Logstash
![Page 2: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/2.jpg)
Ben Waine
• Worked With PHP For 5 Years
• Software Engineer -Sainsbury’s
• Dabbles in devops
![Page 4: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/4.jpg)
System Logs
![Page 5: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/5.jpg)
Application Log
Debug Information - Errors (connections, uncaught exceptions, resource exhaustion)
Narrative Information - Methods Calls, Event Triggers
Business Events - Purchases, Logins, Registrations, Unsubscribes
![Page 6: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/6.jpg)
Keeping Track Of All This....ssh [email protected] -f /var/log/nginx/my-site.access.logtail -f /var/log/my.application.log
ssh [email protected] -f /var/log/mysql/mysql.log
ssh [email protected] -f /var/log/rabbitmq/nodename.log
![Page 7: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/7.jpg)
The Elk Stack
![Page 8: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/8.jpg)
Visualizing Log Data
![Page 9: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/9.jpg)
PHP Logging Tools
1) Monolog2) Everything else....
![Page 10: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/10.jpg)
Basic Logging Examples
1) Monolog: Loggers And Handlers2) Monolog: Tags & Formatters3) Logging business events
![Page 11: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/11.jpg)
use Monolog\Logger;use Monolog\Handler\FingersCrossedHandler;use Monolog\Handler\StreamHandler;
$logEnv = getenv('LOG_LEVEL');$level = empty($logLevel) ? $logEnv : Logger::WARNING;
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG); $fcHandler = new FingersCrossedHandler($strHandler, $level);
$appLog−>pushHandler($fcHandler);$appLog−>debug('LOGGING!');
EG1: Loggers And Handlers
![Page 12: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/12.jpg)
// Set A Log Level$logEnv = getenv('LOG_LEVEL');$level = empty($logLevel) ? $logEnv : Logger::WARNING;
// Create A Logger$appLog = new Logger('AppLog');
![Page 13: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/13.jpg)
$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG);
$fcHandler= new FingersCrossedHandler($strHandler, $level);
// Create Handlers
$appLog−>pushHandler($fcHandler);
$appLog−>debug('Start Logging!');$appLog−>emergency('Something Terrible Happened');
// Push The Handler And Start Logging
![Page 14: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/14.jpg)
EG 2: Tagging Formatting
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/lg.lg', $level);$formatter = new LogstashFormatter("helloapp", "application");
$strHandler−>setFormatter($formatter); $appLog−>pushHandler($strHandler));
$id = $_SERVER('X_VARNISH');$tag = new TagProcessor(['request−id' => $id])
$appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!");
![Page 15: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/15.jpg)
// Create A Logger$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/lg.lg', $level);$formatter = new LogstashFormatter("helloapp", "app");
// Create A Handler & Formatter
// Set Formatter Onto Handler$strHandler−>setFormatter($formatter);
$appLog−>pushHandler($strHandler));
//Push Handler Onto Logger
![Page 16: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/16.jpg)
$id = $_SERVER('X_VARNISH');$tag = new TagProcessor(['request−id' => $id])$appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!");
// Capture A Unique Id, Create A Tag Processor, Push
![Page 17: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/17.jpg)
Log Levels2009 - RFC 5424 - Syslog Protocol
Code / Severity
0 Emergency: system is unusable1 Alert: action must be taken immediately2 Critical: critical conditions3 Error: error conditions4 Warning: warning conditions5 Notice: normal but significant condition6 Informational: informational messages7 Debug: debug-level messages
https://tools.ietf.org/html/rfc5424
![Page 18: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/18.jpg)
Log Levels2013 - PSR03 - PHP Logging Interface Standard
Phrase / Severity
emergency Emergency: system is unusablealert Alert: action must be taken immediatelycritical Critical: critical conditionserror Error: error conditionswarning Warning: warning conditionsnotice Notice: normal but significant conditioninfo Informational: informational messagesdebug Debug: debug-level messages
http://www.php-fig.org/psr/psr-3/
![Page 19: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/19.jpg)
http://imgs.xkcd.com/comics/standards.png
![Page 20: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/20.jpg)
EG 3: Event Logginguse Monolog\Logger;use Symfony\Component\EventDispatcher\EventDispatcher;
$dispatcher = new EventDispatcher();
$dispatcher−>addListener( "business.registration.post", function () use ($busLog) { $busLog−>info("Customer registered"); });
$dispatcher−>dispatch("business.registration.post");
![Page 21: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/21.jpg)
Logstash Architecture
1. Logstash Shipper ships logs to logstash
2. Logstash processes them
3. Logstash Inserts Into Elastic Search
4. Kibana exposes a web interface to Elastic Search data
![Page 22: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/22.jpg)
Logstash Architecture
![Page 23: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/23.jpg)
https://joind.in/talk/view/13369
Why not rate the talk now BEFORE the demo?
![Page 24: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/24.jpg)
Logstash Demo
![Page 25: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/25.jpg)
https://github.com/LoveSoftware/application-logging-with-logstash
![Page 26: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/26.jpg)
![Page 27: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/27.jpg)
![Page 28: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/28.jpg)
![Page 29: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/29.jpg)
![Page 30: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/30.jpg)
![Page 31: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/31.jpg)
Logstash Config
![Page 32: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/32.jpg)
Logstash Collecting{ "network": { "servers": [ "logs.logstashdemo.com:5000" ], "timeout": 15, "ssl ca": "/etc/pki/tls/certs/logstash−forwarder.crt" }, "files": [ { "paths": [ "/var/log/nginx/helloapp.access.log" ], "fields": { "type": "nginx−access" } } ] }
![Page 33: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/33.jpg)
Logstash Processing
input { lumberjack { port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash−forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash−forwarder.key"}
}
Input
![Page 34: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/34.jpg)
Logstash ProcessingFilteringfilter { if [type] == "nginx−access" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } date { match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ] } } }
![Page 35: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/35.jpg)
Logstash ProcessingOutput
output { elasticsearch { host => localhost }}
![Page 36: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/36.jpg)
Groking grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns
http://grokdebug.herokuapp.com/
55.3.244.1 GET /index.html 15824 0.043
%{IP:client}%{WORD:method}%{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}
![Page 37: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/37.jpg)
Logging IdeasRelease MarkerError rates of various applications over timeLatency in various percentiles of each application tierHTTP Responses: 400 series responsesHTTP Responses: 500 series responsesAuto git blame production errorsAuth and Syslogs
![Page 38: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/38.jpg)
Go Forth And Log....BUT
Remember log rotation
Beware running out of space
Beware file logging on NFS
![Page 39: Application Logging With Logstash](https://reader034.vdocuments.site/reader034/viewer/2022042518/55a616671a28abf9328b47f9/html5/thumbnails/39.jpg)
Questions?