python packaging and dependency resolution
TRANSCRIPT
Python Packagingand how to improve dependency
resolution
Tatiana Al-Chueyr Martins@tati_alchueyr
PythonDay Pernambuco - 28th September 2013, Recife
tati_alchueyr.__doc__
● computer engineer by UNICAMP
● senior software engineer at Globo.com
● open source enthusiastic● pythonist since 2003
#opensource #python #android #arduino
Packaging overview
Code Package PackageServer
Code
# life.py
def cleanup_house(address): # ....
def walk_dog(dog_name): # …
class Man(object): __doc__ = “”
Package
# world.py
from life import Man
Packaging overview
Code Package PackageServer
Packaging creation
pack upload
Packaging creationpack
# setup.py
(...)
$ python setup.py sdist
Packaging creationupload
# ~/.pypirc[company]username: Andreiapassword: pwdrepository: http://pypi.company.com
$ python setup.py sdist upload -r company
Packaging usage
search &download
use
Packaging usagesearch &download
use
$ easy_install life
# something.py
from life import Man
Package
way to make the code available so developers can use it
Package
setup.py- contains lots of metadata- dependencies- paths
Packages server: Cheese Shop
place where developers can:● find packages● download packages● upload packages
Brief on Python packaging history● distutils
○ no dependecy management○ problems between cross-platforms○ no consistent way to reproduce an installation○ not all metadata was handled
● setuptools: built on top of distutils○ introduces easy_install○ no way to uninstall installed packages○ provides dependencies management○ introduced eggs (similar to zip files)
● distribute: fork of setuptools○ fork of setuptools
● distutils2 (discontinued?)○ standard versioning (major.minor.micro)○ setup.cfg: pulls metadata from setup.py file, without needing to run setup.py○ which operating system requires which dependecy
pysetup: their interations easyinstall and setuptools with disutils- extract stuff from setup.py
Distutils
● Started by Distutils SIG (Greg Ward)● Added to stand lib in Python 1.6 (2000)● solves
○ issues a variety of commands through setup.py (crete tarball, install your project, compiling C extensions of your python code)
● problems○ no dependency management○ problems between OS○ no consistent way to reproduce an installation○ not all metadata was handled
Brief on Python packaging history
PEP 386: changing the version comparison modules
PEP 376: database of installed python distributions
PEP 345: metadata for python software packages 1.2
Chronology of Packaging by Ziade
http://ziade.org/2012/11/17/chronology-of-packaging/
Chronology of Packaging by Ziade
http://ziade.org/2012/11/17/chronology-of-packaging/
Brief on Python packaging history
old busted new hawtnesssetuptools -> distributeeasy_install -> pipsystem python -> virtual-env
Virtualenv
“virtualenv is a tool to create isolated Python environments.”
https://pypi.python.org/pypi/virtualenv
VirtualenvWrapper
“virtualenvwrapper is a set of extensions to Ian Bicking's virtualenv tool” -- Doug Hellmann
https://pypi.python.org/pypi/virtualenvwrapper
$ mkvirtualenv <name>--python=--no-site-packages=--system-site-packages=
$ rmvirtualenv
$VIRTUALENVWRAPPER_HOOK_DIR/initialize
Pip
A tool for installing and managing Python packages.
http://www.pip-installer.org/en/latest/index.html
$ pip search numpy$ pip help$ pip install flask$ pip uninstall django$ pip freeze--no-deps--extra-index-url --index-url--download-cache --proxy --no-installgit / egg / ...pip install -r requirements.txt
Pip
(...)“This allows users to be in control of specifying an environment of packages that are known to work together.”(...)
http://www.pip-installer.org/en/latest/cookbook.html
How Pip deals with dependencyinconsistencies?
Pip install -r requirements.txt
B
A
# requirements.txtBC
# B/setup.pyA==1.0.0
# C/setup.pyA>=2.0.0 C
what version of A is installed?$ pip install -r requirements.txt
Pip install -r requirements.txt
# requirements.txtBC
# B/setup.pyA==1.0.0
# C/setup.pyA>=2.0.0
$ pip freezeA==1.0.0B==1.0.0C==1.00.
B
A
C
Pip install -r requirements.txt
# requirements.txtCB
# B/setup.pyA==1.0.0
# C/setup.pyA>=2.0.0
what happens? error?$ pip install -r requirements.txt
B
A
C
Pip install -r requirements.txt
# requirements.txtCB
# B/setup.pyA==1.0.0
# C/setup.pyA>=2.0.0
$ pip freezeA==2.0.0B==1.0.0C==1.00.
B
A
C
Pip install -r requirements.txt
# requirements.txtCBA==1.5.0
# B/setup.pyA==1.0.0
# C/setup.pyA>=2.0.0
what happens? error?$ pip install -r requirements.txt
B
A
C
Pip install -r requirements.txt
# requirements.txtCBA==1.5.0
# B/setup.pyA==1.0.0
# C/setup.pyA>=2.0.0
$ pip freezeA==1.5.0B==1.0.0C==1.00.
B
A
C
Explanation
Considering pip 1.5.4:● pip doesn’t identify conflicts of interest
between dependency packages● why?
○ pip solves dependencies analyzing them in a list○ it only concerns in solving the dependencies of the
package being analyzed at that moment○ the last package dependencies prevail
provided a package at pypi, how do I know its dependencies?
provided a package at pypi, how do I know its dependencies?
manually looking to them
dependencies of a package
if you install a package, you can use:$ pip show C
To show dependencies, but they don’t contain versions - only packages names
use pipdeptree$ pip freezeA==1.0.0B==1.0.0C==1.0.0
$ pipdeptree Warning!!! Possible confusing dependencies found:* B==1.0.0 -> A [required: ==1.0.0, installed: 1.0.0] C==1.0.0 -> A [required: >=2.0.0, installed: 1.0.0]------------------------------------------------------------------------wsgiref==0.1.2B==1.0.0 - A [required: ==1.0.0, installed: 1.0.0]C==1.0.0 - A [required: >=2.0.0, installed: 1.0.0]
Does the requirements.txt assure your environment will be reproduced always the same?
Does the requirements.txt assure your environment will be reproduced always the same?
not necessarily
requirements.txt
if you want to assert the same behavior in all installations:● don’t use >=, <=, >, <● pin all dependencies (even deps of deps)● pin exactly (==)
some extra notes
Have your own pypi / proxy
old versions might be removed from remote repositories
the repository might be down during a deploy, and can crash your application
Have your own pypi / proxy
Have your own pypi / proxyhost a PyPI mirror (bandersnatch, pep381client)host a PyPI cache (devp)
PyPI server implementations:● resilient (devpi)● AWS S3 PyPI server (pypicloud)● minimalistic PyPI (pypiserver)● PyPI written in Django (chishop, djangopypi)
Many others..!
At globo.com we have both a PyPI server and a PyPI cache proxy.
dumb ways to manage your dependencies….
1. understand the tools you use to manage dependencies2. keep your dependencies up to date, but take care with >= / >3. take care of your cheese-shop
use pipdeptree package!
thanks!
slideshare: @alchueyr
questions?
Tatiana Al-Chueyr Martins@tati_alchueyr
last notehttp://pypi-ranking.info/author