holger krekel: re-inventing packaging and testing with python

Re-inventing Python Packaging &Testing (0.9)Holger Krekel, http://twitter.com/hpk42 (http://twitter.com/hpk42)

Pycon Russia, Ekaterinburg, 25th Feb 2013

(XKCD of course)

file:///home/hpk/p/pycon-russia-keynote/html/index.html#...

1 of 41 03/01/2013 12:04 PM

Very happy to be here!file:///home/hpk/p/pycon-russia-keynote/html/index.html#...

2 of 41 03/01/2013 12:04 PM

My backgroundstarted games programming 1986studied computer science in the 1990tieslots of Assembler, C, C++, distributed programmingcame to Python around 2001co-founded PyPy project in 2003, core dev for many yearsauthor of pytest, tox, execnet libraries


3 of 41 03/01/2013 12:04 PM

And now for something completelydifferent


4 of 41 03/01/2013 12:04 PM

RememberPerl:

There is more than one way to do it.

Python:

There should be one -- and preferably only one -- obvious way to do it.


5 of 41 03/01/2013 12:04 PM

Second version syndrome

Python3 may have beaten Perl6

(http://www.theregister.co.uk/2008/12/04/python_3point0_release/)

(from theregister.co.uk)


6 of 41 03/01/2013 12:04 PM

but Perl is coming back to haunt us ...file:///home/hpk/p/pycon-russia-keynote/html/index.html#...

7 of 41 03/01/2013 12:04 PM

Comprehensive Perl Archive network(CPAN)cpan.pm comes with base Perl installs:

downloads/builds/installs and tests packages and depscpan index has hundreds of public mirrorsuses declarative metadata since 2003can verify cryptographic signing of packages


8 of 41 03/01/2013 12:04 PM

CPAN server / uploadingPAUSE is the upload server for perl packages:

supports/can enforce versioning semanticsdevelopment releases are mirrored but not indexedpolicy for taking over maintenance of abandoned packagesprovides a mirrored un-indexed "home" area for authors/uploaders for patches, littledocs


9 of 41 03/01/2013 12:04 PM

Python Installer situation Feb 2013no download+install tool coming with core Pythontwo ways to download/install packages (pip, easy_install):

pip installs only source distributionseasy_install also binary distributions

slow: dependency discovery requires client-side crawling for packages in alldownload and homepage sitesno reliable way to test before installno cryptographic signature verification


10 of 41 03/01/2013 12:04 PM

upload/PYPI server interaction todaypypi.python.org server code not easily deployable on your laptopno enforced versioning semanticsno policy for taking over maintenance of abandoned packagesbrittle protocol: network and server failures of any download/homepage serverlead to aborted installs or installation of wrong versionsmetadata through code (setup.py) has tons of problems, has been hard so farto move away from.

and as to mirrors ...


11 of 41 03/01/2013 12:04 PM

CPAN Mirror status

(http://mirrors.cpan.org/)

(http://mirrors.cpan.org/ (http://mirrors.cpan.org/) )


12 of 41 03/01/2013 12:04 PM

PyPI Mirror status

(http://www.pypi-mirrors.org)

(http://www.pypi-mirrors.org (http://www.pypi-mirrors.org) )


13 of 41 03/01/2013 12:04 PM

We can stop the comparison already ...

(from www.cavstheblog.com)


14 of 41 03/01/2013 12:04 PM

Three Lessons learnedfile:///home/hpk/p/pycon-russia-keynote/html/index.html#...

15 of 41 03/01/2013 12:04 PM

Comparing with others can quicklymake you unhappy


16 of 41 03/01/2013 12:04 PM

Perl and Python both not living up totheir mantras


17 of 41 03/01/2013 12:04 PM

There is a lot to improve onfile:///home/hpk/p/pycon-russia-keynote/html/index.html#...

18 of 41 03/01/2013 12:04 PM

Python Tools and PEPs to improvesetuptools/distribute projectspip/distlib/pkglib/... projectsPEP386 new version comparison for distutilsPEP345 Packaging Metadata 1.2PEP376 infrastructure for managing distributions locallyPEP405 virtual environmentsPEP426 Packaging Metadata Version 1.3 (finishing)PEP427 Wheel binary package format (APPROVED)


19 of 41 03/01/2013 12:04 PM

beware: the standardization trap

(one of the many great XKCD comics)


20 of 41 03/01/2013 12:04 PM

To avoid the standardization trap:

Don't demand that the world changes first

before your tool/idea can be used

(was also my motto when starting PyPy)


21 of 41 03/01/2013 12:04 PM

Rather: write meta toolsconfigure and invoke existing toolsmake it work for most use casesenable new facilities/standards from there


22 of 41 03/01/2013 12:04 PM

example from the testing arena

(you do know the goat is Python's testing mascott, right?)


23 of 41 03/01/2013 12:04 PM

Python Testing toolsnose: popular unittest-runnerpy.test: popular generic test runnerunittest/2: standard library TestCaseszope.testing: layered-setup unittest-runnertwisted.trial: running twisted/async test casesmany other methods, including "make test"setuptools has "setup.py test"

Would inventing a new test runner help to standardize testing?


24 of 41 03/01/2013 12:04 PM

tox: a "meta" test running toolmission: standardize testing in Python

configure multiple dependency configsinstall packages in virtualenv environmentsrun tests with your tools of choiceact as a frontend to CI servers

See http://tox.testrun.org (http://tox.testrun.org) for details.


25 of 41 03/01/2013 12:04 PM

But that's about testing ...

what about packaging/pypi?


26 of 41 03/01/2013 12:04 PM

I did a little prototyping ...file:///home/hpk/p/pycon-russia-keynote/html/index.html#...

27 of 41 03/01/2013 12:04 PM

shown first here at Pycon Russia :)file:///home/hpk/p/pycon-russia-keynote/html/index.html#...

28 of 41 03/01/2013 12:04 PM

devpi: a meta tool for packagingactivitiesdevpi serves two purposes:

it's a new compatible index and upload serversubcommands for managing release and QA workflows


29 of 41 03/01/2013 12:04 PM

why a new approach to index serving?there is no good existing pypi server to build these features:

multiple indexes and staging (dev->QA->PROD)merging of upstream (pypi.python.org) indexesserver-side crawling of download/homepagesrecording of automated test results with each packageseemlessly allow offline usage (e. g. on a laptop)


30 of 41 03/01/2013 12:04 PM

Open Source principle:

talk is cheap, show working code!


31 of 41 03/01/2013 12:04 PM

$ devpi servermultiple indexes, organized as ~USERNAME/INDEXNAMEeach index has an upstream index whose packages are seemlessly mergedeach index is "crawl-clean" for clientsuploading a package to an index will not push it to upstreamcompatible for pip, easy_install and setup.py upload

enough theory - on to a a little demo :)


32 of 41 03/01/2013 12:04 PM

summary of devpi server demo insightsupstream merging makes a private overlay index simplecrawling at server-side makes it easy and fast for installersdevpi maintains a nginx-servable static view

but:

typing in index server urls, maintaining .pypirc config is burdensome withmultiple indexes ... therefore ...


33 of 41 03/01/2013 12:04 PM

devpi also provides "workflow"subcommandsuse set current pypi index

install download and install one or more packages

upload build and upload packages from a checkout

test download and test a package

push push a package to another index


34 of 41 03/01/2013 12:04 PM

$ devpi testexample:

$ devpi test -e py25,py33 mypkg

download and unpack a package from current indexcall tox for provisioning and performing testsstream test results from a test run (currently pytest only, nose/unittest to follow)


35 of 41 03/01/2013 12:04 PM

$ devpi pushexample:

$ devpi push mypkg https://pypi.python.org

pushes uploaded packages to another indexcurrently specified by "PKGNAME-VER"in the future this could be done to constraints like "all tests on windows need topass"


36 of 41 03/01/2013 12:04 PM

devpi: typical Open Source work flowstart a localhost index server:

$ devpi server

1.

use a user-specific index:

$ devpi use http://localhost:3141/~hpk42/dev/ (http://localhost:3141/~hpk42/dev/)

2.

upload a package from a checkout:

$ devpi upload

3.

perform tests on multiple machines from multiple users:

$ devpi test PKGNAME

4.

test results at http://localhost:3141/~hpk42/dev/tests (http://localhost:3141/~hpk42/dev/tests)

5.

when release is ready:

$ devpi push PKG https://pypi.python.org

6.


37 of 41 03/01/2013 12:04 PM

devpi implementation planto be MIT licensedtest-driven developmentget early adopters and streamline processes according to real-life usagedon't aim for standard lib or PEPs (for now)


38 of 41 03/01/2013 12:04 PM

my priorities

use cases from contracting companies

use cases from my and others open source work


39 of 41 03/01/2013 12:04 PM

Open areas / future workuse VCS for index persistence?plugin system for upload/serve/test/installimplement mirroring of index as static filessupport declarative metadata / Wheel format /...support server-side PGP signing and verificationa public, redundant,global web serviceCI: per-commit automatic integration, testing and packaging


40 of 41 03/01/2013 12:04 PM

devpi going public mid 2013 ...aiming for a <1.0 public release, to involve early adopters and contributors for finalizingprocesses and good defaults.

Approach me here at Pycon Russia 2013

and thanks for listening!

No public channels yet, but am going to announce them via:

http://twitter.com/hpk42 (http://twitter.com/hpk42)http://holgerkrekel.net (http://holgerkrekel.net)


41 of 41 03/01/2013 12:04 PM

holger krekel: re-inventing packaging and testing with python

Technology