continuous delivery and rapid experimentation

Continuous Delivery

The Path to Rapid Experimentation

Paul Coia

Deploy

Experiment

Example

Lessons

Global platform for independent artists and designers to sell

their work more easily, more oen.

10M designs on 40M products

500,000 artists all over the world.

40M pageviews from 7M Users/month

Deploy

Experiment

Example

Lessons

2000 releases 4 years

700 in 2012

Average 3 per day

cap production deploy

1. Continuous Integration success

2. Deploy code via Capistrano

3. Monitor vital signs

Jenkins Continuous Integration Server

New Relic Application Monitoring

New Relic Key Transaction Monitoring

AirbrakeError Reporting

Circonus and StatsdBusiness Metrics Monitoring

StatsdClient.increment("Payment OK")

if enrolled_in?(“new”).variant? // show variantelse // show normalend

Minimal cost of deploying

Rapid pace of change possible

Confidence make lots of changes

Try ideas quicker

Why we like it

Deploy

Experiment

Example

Lessons

1.Observe

2.Form a hypothesis

3.Devise an experiment to test it

4.Draw conclusions

Scientific Method

Quantitive Research - What?

Qualitative Research - Why?

“If the SEARCH box was bigger, then users will be more likely to search”

Split users at point of variation

Isolate experiments

Persist the decision across visits

Exclude crawlers and ‘old’ users

Enrollment

// ON LANDING PAGEif new_visitor() @guinea_pig = trueend

// ON EXPERIMENT PAGEif @guinea_pig && !retrieve_enrolment() experiment = enrol_in_experiment(“new”) persist_enrolment(experiment)end

if enrolled_in?(“new”).variant? // experiment logicelse // normal logicend

RUNNING_EXPERIMENTS ={ ... new: {description: 'New feature', enrolment_percent: 20}, ...}

Google AnalyticsData Collection

‘R’ Statistical LanguageGoogle Analytics API

Respect Statistics

Establish your test duration up front

Wait for the test to complete

Results

Model your Product funnel

Greater volume earlier in the funnel

Higher conversion rate between steps

Add lots of instrumentation

Useful metrics

Homepage

Sign up page

Signed up

1%

Homepage

Sign up page

Signed up

1%

Homepage

Sign up page

Signed up

20%

5%

‘Landing Page’ Optimizers

Client-side changes only

Simple experiments

Trivial deployment

Feature toggle

Experimentation platform

= Trivial feature experimentation

Deploy

Experiment

Example

Lessons

22% less likely to Add to Cart

50% less likely to Checkout

Mobile Visitors

No change to Checkout Completion

Improvement in Checkout Completion

Increase in Add to Cart rate

Further Increase in Add to Cart rate

Multiple lightweight experiments

A handful of weeks

Verified 30% increase in Purchasing

Deploy

Experiment

Example

Lessons

Invest in your deployment tools

Experiment on the small scale

Expect to be wrong

Build on the successes

Design for incremental change

Thank YouPaul Coia

paul@redbubble.com@pjcoia

h!p://www.redbubble.com/jobs