hear no evil, see no evil, patch no evil: or, how to monkey-patch safely

Hear no evil, see no evil, patch no evil: Or, how to

monkey-patch safely.

Graham Dumpleton @GrahamDumpleton

PyCon Australia - August 2016

Decorators are useful!

Decorators are easy to implement?

Are you sure?

Typical decorator.

def function_wrapper(wrapped): def _wrapper(*args, **kwargs): return wrapped(*args, **kwargs) return _wrapper

@function_wrapper def function(): pass

This breaks introspection.

__name__ and __doc__ attributes are not

preserved.

Doesn’t @functools.wraps() help?

import functools

def function_wrapper(wrapped): @functools.wraps(wrapped) def _wrapper(*args, **kwargs): return wrapped(*args, **kwargs) return _wrapper

@function_wrapper def function(): pass

No, it doesn’t solve all problems.

Still issues with: introspection, wrapping decorators

implemented using descriptors, and more.

http://blog.dscpl.com.auQuick Link: Decorators and monkey patching.

Complicated details removed ….

Get all the details at:

Please try not to implement decorators

yourself.

What is the solution?

Use ‘wrapt’.

Basic decorator.

import wrapt

@wrapt.decorator def pass_through(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs)

@pass_through def function(): pass

Universal decorator.import wrapt import inspect

@wrapt.decorator def universal(wrapped, instance, args, kwargs): if instance is None: if inspect.isclass(wrapped): # Decorator was applied to a class. return wrapped(*args, **kwargs) else: # Decorator was applied to a function or staticmethod. return wrapped(*args, **kwargs) else: if inspect.isclass(instance): # Decorator was applied to a classmethod. return wrapped(*args, **kwargs) else: # Decorator was applied to an instancemethod. return wrapped(*args, **kwargs)

Bonus feature of wrapt if using multithreading.

Synchronise function calls.

from wrapt import synchronized

@synchronized # lock bound to function1 def function1(): pass

@synchronized # lock bound to function2 def function2(): pass

Methods of classes as well.from wrapt import synchronized

class Class(object):

@synchronized # lock bound to instance of Class def function_im(self): pass

@synchronized # lock bound to Class @classmethod def function_cm(cls): pass

@synchronized # lock bound to function_sm @staticmethod def function_sm(): pass

Synchronise block of code.from wrapt import synchronized

class Object(object): @synchronized def function_im_1(self): pass

def function_im_2(self): with synchronized(self): pass

def function_im_3(self): with synchronized(Object): pass

Don’t trust me when I say you should use wrapt?

Potential candidate for being included in the Python

standard library.

So it must be awesome.

Primary purpose of the wrapt package wasn’t as way to build decorators.

Primary reason for existence of wrapt was to

help with monkey patching.

Decorators rely on similar principles to monkey

patching.

Before decorators.# python 2.4+

@function_wrapper def function(): pass

# python 2.3

def function(): pass function = function_wrapper(function)

Decorators are applied when code is defined.

Monkey patching is performed after the fact,

… and can’t use the decorator syntax

Why monkey patch?

• Fix bugs in code you can’t modify.

• Replace/mock out code for testing.

• Add instrumentation for monitoring.

Monkey patching with wrapt.# example.py

class Example(object): def name(self): return 'name'

# patches.py

import wrapt

def wrapper(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs)

from example import Example

wrapt.wrap_function_wrapper(Example, 'name', wrapper)

Don’t patch it yourself.# patches.py

import wrapt

@wrapt.decorator def wrapper(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs)

from example import Example

# DON’T DO THIS.

Example.name = wrapper(Example.name)

Direct patching of methods breaks in

certain corner cases.

Let wrapt apply the wrapper for you.

Avoiding imports.

# patches.py

import wrapt

@wrapt.patch_function_wrapper('example', 'Example.name') def wrapper(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs)

What about testing, where we do not want permanent patches?

Mock alternative.# example.py

class Storage(object): def lookup(self, key): return 'value' def clear(self): pass

# tests.py

import wrapt

@wrapt.transient_function_wrapper('example', 'Storage.lookup') def validate_storage_lookup(wrapped, instance, args, kwargs): assert len(args) == 1 and not kwargs return wrapped(*args, **kwargs)

@validate_storage_lookup def test_method(): storage = Storage() result = storage.lookup('key') storage.clear()

What if we need to intercept access to single

instance of an object?

Transparent object proxy.

# tests.py

import wrapt

class StorageProxy(wrapt.ObjectProxy): def lookup(self, key): assert isinstance(key, str) return self.__wrapped__.lookup(key)

def test_method(): storage = StorageProxy(Storage()) result = storage.lookup(‘key') storage.clear()

Beware though of ordering problems when

applying monkey patches.

Import from module.# example.py

def function(): pass

# module.py

from example import function

# patches.py

import wrapt

@wrapt.patch_function_wrapper('example', 'function') def wrapper(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs)

Plus importing and patching of modules that the

application doesn’t need.

Post import hooks (PEP 369).

# patches.py

import wrapt

def wrapper(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs)

@wrapt.when_imported('example') def apply_patches(module): wrapt.wrap_function_wrapper(module, 'Example.name', wrapper)

Better, but still requires patches module to be

imported before anything else in the main

application script.

Need a way to trigger monkey patches without

modifying application code.

The autowrapt package.

pip install autowrapt

Bundle patches as module.

from setuptools import setup

PATCHES = [ 'wsgiref.simple_server = wrapt_wsgiref_debugging:apply_patches' ]

setup( name = 'wrapt_wsgiref_debugging', version = '0.1', py_modules = ['wrapt_wsgiref_debugging'], entry_points = {‘wrapt_wsgiref_debugging': PATCHES} )

Patch to time functions.from __future__ import print_function from wrapt import wrap_function_wrapper from timeit import default_timer

def timed_function(wrapped, instance, args, kwargs): start = default_timer() print('start', wrapped.__name__) try: return wrapped(*args, **kwargs) finally: duration = default_timer() - start print('finish %s %.3fms' % ( wrapped.__name__, duration*1000.0))

def apply_patches(module): print('patching', module.__name__)

wrap_function_wrapper(module, 'WSGIRequestHandler.handle', timed_function)

Enabling patches.

$ AUTOWRAPT_BOOTSTRAP=wrapt_wsgiref_debugging $ export AUTOWRAPT_BOOTSTRAP

$ python app.py

patching wsgiref.simple_server start handle 127.0.0.1 - - [14/Jul/2016 10:18:46] "GET / HTTP/1.1" 200 12 finish handle 1.018ms

Packaging of patches means they could technically be

shared via PyPi.

Eg: instrumentation for monitoring.

Reasons to use wrapt.

• Create better decorators.

• Awesome thread synchronisation decorator.

• Safer mechanisms for monkey patching.

Graham.Dumpleton@gmail.com

@GrahamDumpleton

wrapt.readthedocs.io

blog.dscpl.com.au