RollbackThe Impossible Dream

by James Turnbull

jamtur01 @ githubkartar @ twitter

jamesturnbull on freenodejames @ puppetlabs.com

About Me

VP Technical Operations at Puppet Labs

Puppet guy

Ruby guy

Talks funny

A show of hands

Who thinks they know what rollback

is?

Last set of hands

YMMV

Definitions

Traditional

Modern

Fact or Fiction?

Accept certain constraints

Constraint #1Apply sufficient

capital

Constraint #2Idempotent

Constraint #3Cascade-less

failure

Constraint #4Resources

A Philosophical Digression

If I know where I amI don’t know how I got there

If I know how I got thereI don’t know where I am

Very few “systems” are

truly deterministic

A Mathematical Digression

On system rollback and totalised fieldsAn algebraic approach to system change

Mark Burgess and Alva Couch20th June 2011

http://cfengine.com/markburgess/papers/totalfield.pdf

So what’s wrong with rollback?

Risk

Learning from mistakes

Complex systems are

… complex

Human error

What is the problem rollback is

trying to solve?

What is the problem YOU are trying to solve?

So how can we mitigate Rollback

shortcomings?

PreventativeDesign

Rollback is (often) an architecture

problem

Increase Resilience

Operational Intelligence

A little bit of DevOps in every byte…

Small, iterative changes

Accept that failure happens

“We can’t test that? Okay we can

roll it back if it breaks…”

Assumption is the mother of all fuckups*

“But the system can’t be {run|upgraded|deployed} like that

because…”

Conclusions

Rollback is possible but not probable

If you have to have “rollback” accept

constraints

You can mitigate the need for it

Thank you!

Questions/Insults?

jamtur01 @ githubkartar @ twitter

jamesturnbull on freenodejames @ puppetlabs.com