Migrating a legacy product to Pyramid

Tom Blockley

Who am I?

• Tom

• Developer for Delib

• Plone for 5 years

• Pyramid for 2 years

• Scrum Master for 18 months

• Before that? Java.

Who are Delib?

• The other Plone company in Bristol

• Est. 2001: Consultation solutions for government

• Plone for 8 years

• First to use Dexterity in production sites

• 3 main products, 2 based on Plone

• >150 clients around the world

• >150,000 responses in 2013

• Peak of 42,000 responses in a single week in 2013

• Biggest ever : 47k Users and 110k Responses in 3 months

What is this?

• Migration of an old product to the Pyramid framework

• Tech choices

• Problems

• Agile

• Translation

• Testing

• Theming

The old

• Conceived 2005

• ZMI product with Kubes

• Completely customisable per-client

• >60 clients, large and small

• Ugly. Really Ugly...

The old

The new

• Conceived May 2013

• Chose Pyramid

• Must be:

• Pretty• Responsive• Accessible• Translatable• Customisable• Tested• ... and done by October 2013

Decisions we made

• Build it using agile methodology

• Use:

• Pyramid• ZODB• Buildout• Diazo• Colander• Deform• repoze.workflow• Pyramid auth• i18n

How did it go?

• Actually only started in July

• 10 week long sprints

• Fast prototyping to test concepts and UX

• Still a lot to do

• BUT the public side of the app nearly finished

Sprint 1 - Ada Lovelace

• 544 lines of python (95 in tests)

• 48% Test coverage

Sprint 2 - Bertrand Russell

• 1693 lines of python (272 in tests)

• 76% Test coverage

Sprint 3 - Charlie Chaplin

• 2299 lines of python (779 in tests)

• 74% Test coverage

Sprint 4 - Don McCullin

• 2419 lines of python (931 in tests)

• 84% Test coverage

Sprint 5 - Edwin Hubble

• 2683 lines of python (1079 in tests)

• 85% Test coverage

Sprint 6 - Francis Crick

• 2867 lines of python (1266 in tests)

• 86% Test coverage

Sprint 7 - Grace Hopper

• 3020 lines of python (1301 in tests)

• 85% Test coverage

Sprint 8 - Harry Houdini

• 3414 lines of python (1416 in tests)

• 86% Test coverage

Sprint 9 - Indiana Jones

• 3752 lines of python (1494 in tests)

• 86% Test coverage

Sprint 10 - Judi Dench

• 4602 lines of python (2027 in tests)

• 87% Test coverage

Technologies used with and in Pyramid

Combined Traversal and Dispatch

• Gives you all the good stuff from traversal

• Without defining your URL structure for you

Combined Traversal and Dispatch

+

+

Combined Traversal and Dispatch

+

Auth framework

• Easy to configure

• Can set it up incrementally

• Hooking up with object classes & __acl__ is a piece of cake

Predicates

• Request Parameter predicates

• Route prefixing

• Stacking predicates

• Custom predicates

• Exception views

• default_views

Colander & Deform

• Does the forms and schema so you don’t have to

• Colander Schema is canonical source for layout

Diazo

• Build basically an HTML API

• Keep basic, semantically correct templates

• Apply anyone’s theme to the templates

• Good introduction to XSL

• Don’t pollute your app code with crazy skin compromises

• Designer & Front end developer could go and do their thing

Things we really liked about Pyramid

• Auth & group finder

• Super flexible predicates

• Combining traversal and dispatch

• Documentation

• Everything runs in one process

• Fast tests

• Minimal boiler plate

• pshell

How did we do against our original goals?

Pretty

• 3 demos in different colour schemes

• Configured with a few lines of LESS

Responsive

• Javascript

• Mobile and Tablet layouts

Accessible

• Well, we’ve not done it yet

• BUT

• We have a set of accessible colour schemes ready to go

• We can fix the accessibility of the markup in Diazo

Translatable

• Every single piece of text on the site has corresponding i18n tag

• No translations yet

• Some of the current theme is too fixed width

Customisable

• Fork it

• Theming for clients is easy

Tested

• Jenkins integration

• No selenium / robot framework yet

• 87% Coverage

Agile

• Fast prototyping

• Populate script

• Make initial implementations as naïve as possible

• We’ve already decided to change the design again

What went wrong?

Pitfalls

• “Protoduction”

• Short term technical debt

• Leaving interns on their own

• Meeting fatigue

• Arbitrary deadlines

Problems

• Incomplete

• Contracted Front End developers

• One tree, many branches

• Rebase is a b*tch

• repoze.workflow

Thanks.