Deep Dive into Logging and Error HandlingBy: Miro Dietiker (miro_dietiker)

Eric Peterson (iamEAP)

WSOD & friendsHow to know when your kittens just die()d.

PHP Exception handler

set_exception_handler( callable $exception_handler)http://php.net/manual/en/function.set-exception-handler.php

throw new Exception('Sorry kitty');

● Catches uncaught exceptions● Stacktraces available

○ _drupal_decode_exception()

All the PHP Errors○ E_ERROR○ E_PARSE○ E_NOTICE○ E_CORE_ERROR, E_CORE_WARNING○ E_COMPILE_ERROR, E_COMPILE_WARNING○ E_USER_ERROR, E_USER_WARNING,

E_USER_NOTICE○ E_STRICT○ E_RECOVERABLE_ERROR○ E_DEPRECATED, E_USER_DEPRECATED

http://www.php.net/manual/en/errorfunc.constants.php

PHP Error handler

set_error_handler( callable $error_handler [, int $error_types = E_ALL | E_STRICT ]

)http://php.net/manual/en/function.set-error-handler.php

trigger_error('3 kittens killed');

● Fired on PHP error● Stacktraces available● Can't deal with E_ERROR, E_PARSE, E_CORE_ERROR,

E_CORE_WARNING, E_COMPILE_ERROR, E_COMPILE_WARNING

Devel module

● Improved error handleradmin/config/development/devel○ Krumo backtrace

● Development only!

PHP Shutdown function

register_shutdown_function ( callable $callback);http://php.net/manual/en/function.register-shutdown-function.php

● Captures E_ERROR + friends○ (Non-existing functions, ...)

● Missing in Drupal 7!● No backtrace, sorry!

PHP log

● apache error.log○ E_PARSE

● Backtraces○ With xdebug

Code fails

drupal_set_message('Sad', 'error');

● Remaining invisible errors○ No error thrown○ No watchdog

Past (Log) module

● Log interface, with structured data

● Exception handler○ With backtrace

● hook_watchdog○ With backtrace

● Shutdown handler● Error log grabber

● Alternative storage backends

Conclusion

error_reporting(E_ALL);settings.php: $conf['error_level'] = 2;

● Enable verbosityadmin/config/development/logging

● Get rid off all errors● Check logs in production...

● Shutdown handler for Drupal 8

New RelicShow me the truth

http://newrelic.com/application-monitoring

Architecture

● The PHP plugin

● The agent

● The SaaS UI

https://newrelic.com/docs/php/new-relic-for-php

DrupalPHP plugin

Agent SaaS UI

Apache

Reporting features

● Overall statistics○ Aggregated values○ Execution Time (CPU / Waiting)○ Specific period in time

● Slow traces○ Deep details about a single call

● Error reporting○ Fatal only

Reporting perspectives

● Web transactions (foreground)● Functions● SQL queries● External services● Drupal (modules)● Background tasks

Drupal Modulehttp://drupal.org/project/new_relic_rpm

● Exceptions● Watchdog● Background detection (cron, drush)● Deployments

Deeper integration

Demo

● Let's play...

Logging, Drupal and BeyondEric Peterson (iamEAP)

Goals

1. Understand the fundamentals of logging in a generic sense.

2. Assess Drupal's system and access log implementations and their weaknesses.

3. Become familiar with log management software and the problems it solves.

4. Learn how to solve real-world problems with Splunk.

66.249.64.13 - - [23/Jun/2013:12:07:01 +1000] "GET /robots.txt HTTP/1.1" 200 468 "-" "Googlebot/2.1"

What is logging, why do we do it?

● Logging is the process of recording events in order to provide an audit trail that can be used to understand system activity and diagnose problems.

For example, an apache access log entry:

Traditional log creation

● The IETF maintains the "syslog" standard, including facilities, severities, etc.

● Facilities indicate program type.

● Standard severity levels:○ "Do I Notice When Evenings Come Around Early"

For example, a UNIX tool for logging:

logger -p local7.notice -t teapot "I'm a little teapot."

Traditional log management

● Configure log routing based upon facility, severity. Route to: file, user, host, program.

# Sample syslog.conf

# Send all local7 to file.local7.* /var/log/local7.log

# Send all alerts via email*.alert | my_email_script

# Sample logrotate.conf/var/log/local7.log {

size 1mcreate 700 root rootrotate 4compress

}

● Commands for rotating, archiving, storing, deleting stale logs.

Traditional log analysis

What are the top user agents visiting my site?

awk -F\" '{print $6}' /var/log/httpd/access.log | sort | uniq -c | sort -fr

633 Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)

287 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.

html)

266 Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)

249 Amazon CloudFront

tail -f /var/log/httpd/access.log

Drupal system log creation

● Watchdog API and syslog are very similar.○ Severity levels, able to specify type/application.○ Default enabled "Database Logging" module writes

all data to the watchdog database table.○ hook_watchdog provides flexibility in routing log

messages in custom ways.

watchdog('TYPE', 'MESSAGE: %message_var', array('%message_var => 'MESSAGE VAR VALUE'), WATCHDOG_NOTICE);

Drupal access log creation

● Functionality contained within the Core Statistics module; disabled by default.

● When enabled, writes all accesses to the accesslog database table.

● No API.

Drupal log management

● Drupal cron "rotates" (deletes) stale data; configure the number of records to keep.

Drupal system log analysis

● /admin/reports/dblog

○ Filterable by type or severity.○ Click out to an individual log

event

Drupal access log analysis

● /admin/reports/hits

● Additional "one-off" reports for top visitors, pages, referrers.

Problems with log data analysis

● It can be extremely cumbersome to track down the relevant log entry; let alone the root cause.○ What about searching?○ What about filtering by user or IP or date or path?○ We can't see IP or path on the overview?○ A paged view is terrible when log events flow in copiously.

● https://drupal.org/project/views_watchdog

● Still, Views exposed filters were built for end-user interactions, not deep data analysis.

Problems alerting/reporting on logs

● https://drupal.org/project/logging_alerts

● Still, this only helps in alerting on specific events. What about alerts based on aggregates (e.g. count thresholds, averages, etc)?

Why didn't we catch this problem earlier?

Problems with log rotation/retention

● https://drupal.org/project/dsc (DBlog Selective Cron)

● Configuring this on medium-to-large sites is very cumbersome (what types and severities even exist, let alone are useful to you?)

● Still no archiving.

Didn't we have this same problem last week?

Log performance/scalability problems

● Multi-host sites: accesslog/watchdog quickly force the DB into the biggest bottleneck.

● Shared hosting: latency and/or overloaded DB server can be a performance killer.

● Try an alternate backend:○ https://drupal.org/project/mongodb○ https://drupal.org/project/redis_watchdog

● Write logs to syslog with core Syslog module.

Log management and OI

● There's no clear strategic advantage to using Drupal as your log management platform.

● Why not use something else more fully featured and built specifically for the purpose?

○ Splunk○ Loggly○ Papertrail○ Logstash (a FOSS alternative/complement)

Connecting Drupal + Syslog + Splunk1. Enable syslog (optionally, Syslog Access)

2. Configure facility, application tag, message format

3. Add the following to syslog/rsyslog.conf, restart.

4. Configure Splunk to accept events from your host.

5. Once verified, disable database logging.

Other configs possible (log to file, Splunk's universal forwarder, API calls...

local0.* @@url.to.splunk:12345

Splunk Search Fundamentals

Splunk Fields

Splunk Field Extraction

● In many cases, splunk automatically extracts "interesting fields" based on message syntax

● You can also extract custom fields using regex, or by selecting in the Splunk GUI

Splunk Commands & IO Redirection

● Similar to a command line, you can pipe Splunk search results to commands.

process="drupal" | eval userStatus=(if(uid=0, "Anonymous", "Authenticated"))

process="drupal" | eval userStatus=(if(uid=0, "Anonymous", "Authenticated")) | chart count by userStatus | sort -count

Splunk Visualization Basics

process="drupal" | eval userStatus=(if(uid=0, "Anonymous", "Authenticated")) | chart count by userStatus

Splunk Visualization (cont)

● Analyzing measures over time with timechart

process="drupal-access" | timechart min(timer) as "Min", avg(timer) as "Avg", max(timer) as "Max"

Splunk Visualization (cont)

● 100% stacked area charts

process="drupal-access" | timechart count by cache_status

Splunk Reporting

[XML of the above dashboard]

Splunk Alerting

● Find the top occurring PHP errors● Alert when cron hasn't run recently● Track service outages in 3rd party APIs.● Find all pages/URLs referring to a given page● Detect code sync issues in a multi-host setup● Monitor errors, memory usage, etc. during deployment.● Trace events chronologically for a given IP, user.● DOS or infinite redirect from IE?

A few ideas for inspiration

Conclusion, questions

● Don't use Drupal as your log management platform; there is no advantage.

● Whatever solution you do use, be aware of its analysis and reporting capabilities.

● You can better understand your log data by visualizing it; when reporting, be aware of what you're communicating and how.

● You know your application best.