become a logging expert in 30 minutes

7/27/2019 Become a Logging Expert in 30 Minutes

1/66

Gavin M. Roy

@crad


2/66

Were Hiring!

Senior Architects

API Engineering Team

Android

iOS

Web Front-End

DevOps Engineers Network Engineers


3/66

https://github.com/gmr/pycon2013-logging


4/66

Why should you use logging?


5/66

Common Pattern?try:ifdebug:print'Executingquery:%s'%query

self.cursor.execute(query)exceptpsycopg2.OperationalError,e:

sys.stderr.write('PostgreSQLError:%s'%e[0])sys.exit(1)exceptpsycopg2.IntegrityError,e:sys.stderr.write('PostgreSQLError:%r'%e[0])

returnFalseexceptException,e:sys.stderr.write('UnhandledException:%r'%e[0])sys.exit(1)


6/66

Consistent Output Methodtry:logger.DEBUG('Executingquery:%s',query)cursor.execute(query)

exceptpsycopg2.OperationalErroraserror:logger.CRITICAL('PostgreSQLException:%s',error)

sys.exit(1)exceptpsycopg2.IntegrityErroraserror:

logger.ERROR('PostgreSQLDataError:%s',error)returnFalse

exceptExceptionasexception:logger.CRITICAL('UnhandledException:%s',exception)sys.exit(1)


7/66

Avoid NIH


8/66

Configurable Destinations

Files Rotating Files

Logging

EmailIn-Memory Buffer

Syslog

Console

HTTP Web Service

SocketTCP, UDP

Windows NT Event Log

Watched Files


9/66

Configurable Formatting

Standard string-

formatting options

Many attributes

available

Per destinationconditional formatting


10/66

Filterable Content


11/66

Easy to use API for:

Debugging your application

Providing runtime state information

Triggering alerts upon specific

conditions

Sharing application data andinformation with external services


12/66

Logging LevelsDEBUGINFO

WARNING

ERROR

CRITICAL


13/66

namespaces


14/66

Using for writing Packages

Common, well-documented methods

and tools for configuration

Users can control the level ofinformation they want/need

Reduces the amount of code

Ensure your package is more Pythonic

and plays well with others


15/66

The Logging Lexicon


16/66

Log Records

Used to build the logged messages,

route them and filter them

Hold all attributes about the message


17/66

Loggers The logger is the first actor to

decide if a message should be

distributed to assignedhandlers

Point of entry for all logged

messages

Provides destination fornamespaced logged

messages


18/66

The Object to Use

debug(msg,*args,**kwargs)

info(msg,*args,**kwargs)warning(msg,*args,**kwargs)

error(msg,*args,**kwargs)

exception(msg,*args)*

* Logs at ERROR level and should be called only by exception handlers


19/66

Root Logger


20/66

propagate = True?


21/66

foo.bar.baz

when Propagate is True:

apply to everything under foo


22/66

Double Logging?


23/66

incremental = True?


24/66

Dangerous

BehaviorIf loggers are alreadyconfigured, incremental=Truewill not change pre-existing

configuration for formattersor filters


25/66

Handlers

Deliver messages to their destination

Also filterable by level and filter

Can have multiple per application

Assigned to loggers

Can have multiple per logger as well

Assigned a formatter and any filters


26/66

Formatters

Use for destination specific formatting

One formatter can be assigned to

multiple handlers

Responsible for merging internal values

Uses Python %-style mapping keys to

LogRecord attributes


27/66

Example Formats%(levelname)s %(asctime)s %(module)s.%(funcName)s() (%(lineno)d): %(message)s

INFO2013-03-1617:01:34web.log_request()(1462):200GET/(::1)0.41msINFO2013-03-1617:01:34web.log_request()(1462):200GET/(::1)0.33msINFO2013-03-1617:02:02web.log_request()(1462):200GET/(127.0.0.1)0.39msINFO2013-03-1617:02:03web.log_request()(1462):200GET/(127.0.0.1)0.27ms

%(levelname)s %(name)s.%(funcName)s(): %(message)s

root.log_request():200GET/(127.0.0.1)0.53msroot.log_request():200GET/(127.0.0.1)0.37msroot.log_request():200GET/(127.0.0.1)0.27msroot.log_request():200GET/(127.0.0.1)0.42ms
http://web.log/http://web.log/http://web.log/http://web.log/http://web.log/http://web.log/


28/66

Log Adapters

Build custom adapters

to inject additional

information not available

to formatters by default

Add context, simplify

logger calls

Same signature as a

Logger, passed a loggeron construction

Use like a Logger


29/66

Filters

Applied to Loggers

Base class filters on namespace

Example limiting logging to only items in

the foo.bar.baznamespace:

filter=logging.Filter('content')logger.addFilter(filter)

Can be extended for your own purposes


30/66

Configuration Options

4 configuration options:

Manually in code

Very limiting, code change to alter

configuration

Dictionary based configuration (2.7+)

File based configuration

Socket listener configuration*


31/66

DictConfig

Allows for configuration to be passed in

using a dict format

Great for structured config data, such asconfiguration stored in JSON or YAML

Can be very verbose


32/66

DictConfig Example{'version': 1,

'disable_existing_loggers': True,

'incremental': False,

'filters': [],

'formatters': {'syslog': {'format': '%(levelname)s '

'%(name)s.%(funcName)s(): %(message)s'},

'verbose': {'datefmt': '%Y-%m-%d %H:%M:%S','format': '%(levelname) -10s %(asctime)s '

'%(processName) -35s %(name) -35s '

'%(funcName) -30s: %(message)s'}},

'handlers': {'console': {'class': 'logging.StreamHandler',

'debug_only': False,

'formatter': 'verbose'},

'syslog': {'address': '/var/run/syslog',

'class': 'logging.handlers.SysLogHandler','facility': 'daemon',

'formatter': 'syslog'}},

'loggers': {'django': {'handlers': ['console', 'syslog'],

'level': 'ERROR',

'propagate': False},

'psycopg2': {'handlers': ['console', 'syslog'],

'level': 'INFO','propagate': True}}}


33/66

DictConfig YAML ExampleLogging: formatters:

brief:

format: "%(levelname)s %(name)s.%

(funcName)s(): %(message)s"

filters: []

handlers:

console:class: logging.StreamHandler

formatter: brief

syslog:

class: logging.handlers.SysLogHandler

facility: daemon

address: /dev/log

formatter: brief loggers:

django:

propagate: true

level: WARNING

handlers: [console, syslog]

psycopg2:

propagate: false

level: ERROR

handlers: [console, syslog]

disable_existing_loggers: true

incremental: false

version: 1


34/66

Using a YAML config

import logging.config

import yaml

config = yaml.load(open('config.yaml', 'r'))

logging.config.dictConfig(config)


35/66

FileConfig

ConfigParser formatted configuration

files

Cant use SafeConfigParser due toauto-string interpolation

Somewhat more verbose than a YAML

format


36/66

FileConfig Example[loggers]keys=myapp[logger_myapp]level=INFOhandlers=syslog

propagate=1qualname=myapp[handlers]keys=syslog[handler_syslog]class=handlers.SysLogHandler

formatter=briefargs=('/dev/log',handlers.SysLogHandler.LOG_LOCAL6)[formatters]keys=brief[formatter_brief]format=%(name)s.%(funcName)s():%(message)s


37/66

Logging Config Listener#!/usr/bin/envpythonimportlogging

importlogging.configimporttime

logging.basicConfig(level=logging.INFO)

#Listenondefaultportof9030,isathread

listener=logging.config.listen()listener.start()

logger=logging.getLogger('listener')

start_time=time.time()

whileTrue:

logger.info('Timesincestart:%.2f',time.time()-start_time)try:time.sleep(1)

exceptKeyboardInterrupt:logging.config.stopListening()

break


38/66

Send to Listener

#!/usr/bin/envpython

importlogging.configimportsocket

importstruct

HOST='localhost'

PORT=logging.config.DEFAULT_LOGGING_CONFIG_PORT#9030defsend_conf(config_text):s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)

s.connect((HOST,PORT))s.send(struct.pack('>L',len(config_text)))s.send(config_text)

s.close()

withopen('logging.conf','r')ashandle:send_conf(handle.read())


39/66

logger.cfg[loggers]keys=root,listener[logger_listener]level=INFOhandlers=consolepropagate=1qualname=listener

[logger_root]level=WARNINGhandlers=console,syslogpropagate=1[handlers]keys=console,syslog[handler_console]class=StreamHandlerformatter=brief

args=()stream=(ext://sys.stdout)[handler_syslog]class=handlers.SysLogHandlerformatter=briefargs=('/dev/log',handlers.SysLogHandler.LOG_LOCAL6)[formatters]keys=brief

[formatter_brief]format=%(name)s.%(funcName)s():%(message)s


40/66

OutputBeforecallingsend-config-to-listener.py

INFO:listener:Timesincestart:0.00INFO:listener:Timesincestart:1.00

INFO:listener:Timesincestart:2.00

Aftercallingsend-config-to-listener.py

listener.():Timesincestart:3.00listener.():Timesincestart:3.00

listener.():Timesincestart:4.00listener.():Timesincestart:4.00


41/66

stdlib Handlers

StreamHandler

FileHandler

SyslogHandler

WatchedFileHandler

RotatingFileHandler

TimedRotatingFileHandler

NTEventLogHandler

SocketHandler (TCP)

DatagramHandler (UDP)

SMTPHandler

MemoryHandler

HTTPHandler

NullHandler


42/66

Using SMTPHandler


importloggingfromloggingimporthandlers

importsocketimporttraceback

HOST='localhost'FROM='"APPLICATIONALERT"'TO='you@your-domain'

SUBJECT='NewCriticalEventFrom[APPLICATION]'

logging.basicConfig(level=logging.INFO)handler=handlers.SMTPHandler(HOST,FROM,TO,SUBJECT)

email_logger=logging.getLogger('smtp.example')

email_logger.addHandler(handler)email_logger.setLevel=logging.CRITICAL

logging.info('Rootloggeroutput')try:

email_logger.critical('CriticalEventNotification\n\nTraceback:\n%s',''.join(traceback.format_stack()))

exceptsocket.erroraserror:

logging.critical('CouldnotsendemailviaSMTPHandler:%r',error)


43/66


44/66

Using HTTPHandler


importloggingfromloggingimporthandlers

importsocketimporttraceback

HOST='localhost'PORT=8888


handler=handlers.HTTPHandler('%s:%s'%(HOST,PORT),'/')http_logger=logging.getLogger('http.example')http_logger.addHandler(handler)

http_logger.setLevel=logging.CRITICAL

logging.info('Rootloggeroutput')try:

http_logger.critical('CriticalEventNotification\n\nTraceback:\n%s',''.join(traceback.format_stack()))

exceptsocket.erroraserror:logging.critical('CouldnotdelivermessageviaHTTPHandler:%r',error)


45/66

HTTPHandler Request Data{'args':'(\'File"http_handler-example.py",line21,in\\n''\\\'\\\'.join(traceback.format_stack()))\\n\',)',

'created':'1363542183.32','exc_info':'None',

'exc_text':'None','filename':'http_handler-example.py',

'funcName':'',

'levelname':'CRITICAL','levelno':'50','lineno':'21',

'module':'http_handler-example','msecs':'316.456079483',

'msg':'CriticalEventNotification\n\nTraceback:\n%s','name':'http.example',

'pathname':'http_handler-example.py',

'process':'13948','processName':'MainProcess','relativeCreated':'12.0251178741',

'thread':'140735278571904','threadName':'MainThread'}


46/66

Example Custom Handler

Proof of concept application:

Handler with a rotating log buffer

Tornado web application to view log

entries


47/66


48/66

class WebRequestHandler(web.RequestHandler):

def get(self):

handler = self.application.log_handler

self.write({'entries': handler.buffer})

if self.get_argument('flush', False):

handler.flush()

Tornado Request Handler
http://web.requesthandler/http://web.requesthandler/


49/66

if__name__== '__main__':


routes=[(r'.*',WebRequestHandler)])

application=web.Application(routes)

application.listen(8888)

application.log_handler=BufferedLogHandler(MAX)

root_logger=logging.getLogger()

root_logger.addHandler(application.log_handler)

ioloop.IOLoop.instance().start()

Setup & Start Application


50/66

AMQPHandler?


51/66

Logging

Performance

Fibonacci Sequence

cProfile

Python 2.7.2

Python 3.3.0

Default Handler


52/66

Baseline Profile

def fib(n):

a, b = 0, 1

while a < n:

a, b = b, a+b

fib(1000000)

2.7 3.3


53/66

import logging 2.7

import logging

def fib(n):

a, b = 0, 1while a < n:

a, b = b, a+b

fib(1000000)


54/66

import logging 3.3


55/66

level=INFO, debug 2.7import logging

logger = logging.getLogger(__name__)


def fib(n):

a, b = 0, 1while a < n:

logger.debug(Value: %i, a)

a, b = b, a+b

fib(1000000)


56/66

level=INFO, debug 3.3


57/66

level=DEBUG, debug 2.7import logging

logger = logging.getLogger(__name__)

logging.basicConfig(level=logging.DEBUG)

def fib(n):

a, b = 0, 1while a < n:

logger.debug(Value: %i, a)

a, b = b, a+b

fib(1000000)


58/66


59/66

Logging Performance

0 ms

0.375 ms

0.75 ms

1.125 ms

1.5 ms

import level=INFO level=DEBUG

2.7 3.3

* Chart data at end of deck on slide #62


60/66


61/66

What to do?


62/66

Use Debug*

#*Butcheckifyourappisindebugmodefirstifdebug:

LOGGER.debug('Hasstairsintheirhouse:%r',

is_protected)


63/66


64/66

https://github.com/gmr/pycon2013-logging

Questions?

Follow me on Twitter: @crad


65/66


66/66

Photo Credits

Slide #12 "Back on the Log Train"Claire L. Evans

http://www.flickr.com/photos/10097505@N00/1492818224/

Slide #44 "I'm a lumberjack and I'm ok..."Neil Kremerhttp://www.flickr.com/photos/neilarmstrong2/

become a logging expert in 30 minutes

Documents