how i learned to stop worrying & love python packaging

28
h [email protected] d @enn_io o W Jannis Leidel

Upload: jannis-leidel

Post on 14-May-2015

7.984 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: How I learned to stop worrying & love Python Packaging

h [email protected]

d @enn_io

oW Jannis Leidel

Page 2: How I learned to stop worrying & love Python Packaging

6How I learned to stop

worrying & love Python Packaging

Page 3: How I learned to stop worrying & love Python Packaging

1) The past of Python packaging2) Common pitfalls and gotchas3) Everyday software development4) The future of Python packaging

6

Page 4: How I learned to stop worrying & love Python Packaging

- Python packageA directory with Python files (module)and a __init__.py file

- ReleaseA version of a specific software, resultof a development cycle, e.g. “Django 1.3”

- DistributionSource or binary form of a release,e.g. zipped tarball or Windows installer

Ambiguous terms

Page 5: How I learned to stop worrying & love Python Packaging

1The past of Python packaging, a brief historical overview

Page 6: How I learned to stop worrying & love Python Packaging

“[..] a standard mechanism for building, distributing, and installing Python modules – or, more realistically, multi-module distributions.”

Distutils history

Greg Ward

- creates distributions

- installs to library directory

- builds C extensions

- processes documentation

Page 7: How I learned to stop worrying & love Python Packaging

- PEP 241 (2001)Metadata in PKG-INFO files added to with package distributions

- PEP 301 (2002)Package Index and Trove classifiers added and register command

- PEP 314 (2003)Metadata 1.1 with license, platform, download URL, dependencies fields

Distutils enhancements

Page 8: How I learned to stop worrying & love Python Packaging

- PEP 345 (2005-today)Metadata 1.2, better dependencies

- PEP 376 (2009-today)Database of Installed Python packages

- PEP 386 (2009-today)Sane version comparison module and better version handling on PyPI

Distutils enhancements

Page 9: How I learned to stop worrying & love Python Packaging

2Common pitfalls and gotchas of packaging

Python software

Page 10: How I learned to stop worrying & love Python Packaging

from  distutils.core  import  setup

setup(        name='MyApp',        version='0.1.0',        description='My  app  that  does  things',        author='John  Doe',        author_email='[email protected]',        license='BSD',        packages=['myapp'],        package_data  =  {                'myapp':  [                        'templates/*.html',                ],        },)

The standard setup.py

Page 11: How I learned to stop worrying & love Python Packaging

Long description

Rendered on PyPI with docutilsreStructuredText renderer

import  codecsdef  readme(filename):        file  =  codecs.open(filename,  'utf-­‐8')                return  unicode(file.read())

setup(        #  ...        long_description  =  readme('README.rst'),)

Page 12: How I learned to stop worrying & love Python Packaging

Lists all additional, non-Python files of a release to be installed from a distribution

setup(        #  ...        package_data  =  {                'myapp':  [                        'static/myapp/*/*',                        'templates/myapp/*',                        'locale/*/LC_MESSAGES/*'                ],    },)

package_data

Page 13: How I learned to stop worrying & love Python Packaging

Template for the “MANIFEST” file thatlists all files to be put in a distribution

include  README.rstrecursive-­‐include  docs  *.txtrecursive-­‐include  myapp/locale  *.mo  *.porecursive-­‐include  myapp/static  *.css  *.pngrecursive-­‐include  myapp/templates  *.html

MANIFEST.in

Page 14: How I learned to stop worrying & love Python Packaging

- 0.18 “Catty And The Major”

- .000001

- “unreleased.uno!cialdev”

Don’t do that.

- Use PEP 386 for formal versions, e.g.“1.0.1” or “0.5a4” etc

- Use release names in any prose documentation and changelogs,e.g. super hero names

Use sane versions, dammit!

Page 15: How I learned to stop worrying & love Python Packaging

3 Using packaging in everyday’s software

development

Page 16: How I learned to stop worrying & love Python Packaging

- Install binary packages globally using your operatings system’s package management, e.g. database adapters, PIL, lxml

- Vendorize packages that require modifications or short term patches

- Install often updated and your project specific packages locally, e.g. Django, pytz, your own app

System package management or not?

Page 17: How I learned to stop worrying & love Python Packaging

- Isolated Python environments

- Create by topic, e.g. by “branch”, “client”, “milestone”, “staging/prod” etc.

- Use virtualenv-wrapper for even more shell helpers

- pip included in every virtual environment

virtualenv

Page 18: How I learned to stop worrying & love Python Packaging

- Sane installation and uninstallation

- freezing and unfreezing of requirements

$  server1:~  pip  freeze  >  dev-­‐deps.txt

$  server2:~  pip  install  -­‐r  dev-­‐deps.txt

- VCS integration for quick and dirty use

- PyPI mirror support (PEP 381), e.g. d.pypi.python.org

pip

Page 19: How I learned to stop worrying & love Python Packaging

- HTTP server with directory index feature, e.g. Apache’s DirectoryIndex

- PyPI clone Chishop

pip  install  -­‐f  http://localhost/dists/

pip  install  -­‐i  http://pypi.corp.local/

- Use ~/.pip/pip.conf

[install]

index-­‐url  =  http://pypi.corp.local/

Simple own package index

Page 20: How I learned to stop worrying & love Python Packaging

- Use ad-hoc HTTP server

$  python  -­‐m  SimpleHTTPServer  8000

- .pydistutils.cfg sdist dir

[sdist]

dist-­‐dir  =  /var/www/dists

- Enables pip download-cache

[global]

download-­‐cache  =  ~/.pip-­‐downloads

Local index

Page 21: How I learned to stop worrying & love Python Packaging

4The future

of packagingwith Distutils

Page 22: How I learned to stop worrying & love Python Packaging

- toolbox with reference implementation for PEP 345, PEP 376, PEP 386

- the standalone pysetup tool thatinstalls and uninstalls distributions(e.g. “pysetup install Django”)

- metadata available outside of Python file in setup.cfg

- already in Python 3 trunk as “packaging”

What is distutils2?

Page 23: How I learned to stop worrying & love Python Packaging

- defined requirements depending on environment

Requires-­‐Dist:  bar;  python_version  ==  '2.4'  or  python_version  ==  '2.5'

Requires-­‐Python:  >=2.5,<3

- Obsoleting other releases

Obsoletes-­‐Dist:  OtherProject  (<3.0)

Other distutils2 features

Page 24: How I learned to stop worrying & love Python Packaging

- “pysetup create” for conversion setup.py to setup.cfg for forward compatibility

- “pysetup generate-setup” for conversion setup.cfg to setup.py for backwards compatibility

Moving from distutils to distutils2

Page 25: How I learned to stop worrying & love Python Packaging

[metadata]name  =  myappversion  =  0.1author  =  John  Doeauthor-­‐email  =  [email protected]  =  My  awesome  appdescription-­‐file  =  README.rsthome-­‐page  =  http://johndoe.com/p/myapp/project-­‐url:  Repository,  http://code.johndoe.com/myapp/classifier  =  Development  Status  ::  3  -­‐  AlphaLicense  ::  OSI  Approved  ::  MIT  License

[files]packages  =  myappextra_files  =  setup.py  READMEresources  =      etc/settings.py.dist  {confdir}/myapp

Example distutils2 setup.cfg

Page 26: How I learned to stop worrying & love Python Packaging

- METADATA

- RECORDa list of installed files (CSV formatting)

- RESOURCESlist of non-python files

- INSTALLERname of the installer

- REQUESTEDexists if distribution wasn’t installed as dependency

Other files that are shipped with distutils2

Page 27: How I learned to stop worrying & love Python Packaging

1) Keep your setup.py files simple2) Follow the standards3) Use virtualenv/pip4) Look out for distutils2/packaging5) In doubt, read the

“Hitchhiker’s Guide to Packaging”

6

Page 28: How I learned to stop worrying & love Python Packaging

h [email protected]

d @enn_io

oW Jannis Leidel

Thanks!

Questions?

Don‘t forget to

while we’re in Amsterdam!