Strategies In Continuous Delivery

Learn the “magic” behind the scenes

Aviran MordoServer Group Manager @ Wix

@aviranmhttp://www.linkedin.com/in/aviran

10:07

About Wix

10:07

Wix in Numbers

• Over 35,000,000 users– Adding over 1,000,000 new users each month

• Static storage is over 150TB of data– Adding over 1TB of files every day

• 3 Data centers + 2 Clouds (Google AE, Amazon)– Around 300 servers

• Over 40,000,000 Server API calls per day• ~400 people work at wix

– Over 100 people in R&D

10:07

Is Doing Multiple Deployments A Day Takes

Guts ?10:07

10:07

From Jan’ 2013 – Jun’ 2013

• 1500• 470• 200

10:07

From Jan’ 2013 – Jun’2013

• 1500 Deployments• 470 A/B Tests• 200 Feature Toggles

Continuous Delivery at Wix

• Abandon “VERSION” paradigm – move feature centric life

• Make small and frequent release as soon as possible

• Automate everything – TDD/CI/CD• Measure Everything

– A/B test every new feature– Monitor real KPIs (business, not CPU)

10:07

Continuous Delivery at Wix

• Empower the developer–The developer is responsible from product idea

to 1M active users–Remove every obstacle in the developer’s path–Big cultural change from waterfall – affects the

whole company

10:07

Test Driven Development

• No new code is pushed to Git without being fully tested

• Before fixing a bug first write a test to reproduce the bug

• Cover legacy (untested) systems with Integration tests

10:07

Test Driven Development

• What people think is the impact on development–TDD slows down development–With TDD we write more code (product + test

code).

• Current Test Count (U-Tests + IT-Tests) – over 10,000

10:07

Test Driven Development

• Actual impact on development–We develop products faster–Removes fear of change–Easier to enter some one else’s project–Do we really need QA? (Yes, they code tests)–Writing a feature is 10-30% slower, 45-90%

less bugs–50% faster to reach production.–Considerably faster time to fix bugs10:07

Guidelines for successful TDD

• Tests should run on project checkout to a random computer.

• Tests that cannot be debugged on a developer machine will never consistently run for any period of time

• Tests should run fast• Tests have to be readable – They are the

project’s specs• Fixture is evil!10:07

Feature Toggles

10:07

Feature Toggles

• Everyone develops on the Trunk

• Every piece of code can get to production at anytime

10:07

Feature Toggle to the rescue

• Unused new code can go to production – no harm done

• Operational new code goes with a guard – use new or old code by feature toggle

10:07

Usage example

Simple “if” statement in your code

10:07

Feature Toggle Strategies (gradual expose users)

• Company employees• Specific users or group of users• Percentage of traffic• By GEO • By Language• By user-agent• User Profile based• By context (site id or some kind of hash on site id)

10:07

Feature Toggle Override

• By specific server– Used to test system load– New database flows/migration– Refactoring that may affect performance and memory usage

• By Url parameter– Enable internal testing– Product acceptance– Faking GEO

• By FT cookie value– Testing– When working with API on a single page application

10:07

Feature Toggles Management

10:07

A/B Tests

10:07

A/B Test

• Every new feature is A/B tested• We open the new feature to a % of

users– Define KPIs to check if the new feature is

better or worse – If it is better, we keep it– If worse, we check why and improve– If we find flaws, the impact is just for %

of our users (kind of Feature Toggle)

10:07

An interesting site effect on product

• How many times did you have the conversion “what is better”?– Put the menu on top / on the side

• Well, how about building both and A/B Testing?

10:07

Marking users with toss value in a cookie

• Anonymous user– Toss is randomly determined– Can not guarantee persistent experience if changing browser

• Registered User– Toss is determined by the user ID– Guarantee toss persistency across browsers– Allows setting additional tossing criteria (for example new users only)– Only use this for sections that a user has to be authenticated

10:07

• Do not mix anonymous and registered tests

• AB test parentage of users with optional filters– New Users Only (Registered users only)– By language – By GEO– By Browser – user-agent – OS– Any other criteria you have on your users

10:07

A/B Test Features

• A/B Test Override– Allows setting a value of a test for validation– Helps support experience what users experiencing

• Override methods– Via URL parameter– Via cookie

• Start/Stop Test• Pause tests• Bots always get “A”10:07

Manage your tests

10:07

Refactoring

10:07

Refactoring

• Before refactoring make sure everything is covered with tests– Legacy code usually covered by IT tests

• Refactor from inside out– Small iterations with tests– Refactor small methods - make sure the tests don’t

break– Deploy often

• Re-write from the outside in– Write from scratch– Code duplication sometimes needed (temporary)– Protected by Feature Toggle

10:07

Code branch

10:07

New Code Old Code

FT Opene

d

Yes No

DB Schema Changes Without Downtime

• Adding columns– Use another table link by primary key– Use blob field for schema flexibility

• Removing fields– Stop using. Do not do any DB schema

changes

10:07

New DB schema

• Plan a lazy migration path controlled by feature toggle

1. Write to old / Read from old2. Write to both / Read from old 3. Write to both / Read from new, fallback to old4. Write to new / Read from new, fallback to

old5. Eagerly migrate data in the background6. Write to new / Read from new

10:07

Gradual Deployment

10:07

• Assume two components

• We shutdown one and install on it the new version. It is not active yet

• Do self test• Activate the new server it is passes self test

• Continue deploying the other servers, a few at a time, checking each one with self test

A 1.1 B 1.1

A 1.1B 1.2

A 1.1A 1.1 B 1.1B 1.1

A 1.1A 1.1B 1.1

B 1.2

A 1.1B 1.2

A 1.1A 1.1B 1.1

B 1.2

A 1.1 B 1.1A 1.1A 1.1 B 1.1B 1.2

Self Test / Post Deployment Test

After each server deployment run a self test before deploying the next server.

• Checking server configuration and topology– Make sure database is accessible (DB connection string)– Is the schema the one I expect– Access required local resources (data files, other config files,

templates, etc’)– Access remote resources– RPC / REST endpoints reachable and operational

• Server will refuse requests unless it passes the self test• Allow a way to skip self test (and continue deployment)10:07

Tools - App-info

10:07

Backward and Forward compatible

• Assume two components

• We release a new version of one

• Now Rollback the other…

10:07

A 1.1B 1.2

A 1.2

B 1.1A 1.1A 1.1B 1.1

B 1.2

A 1.2A 1.1B 1.1B 1.1

A 1.1 B 1.1A 1.1A 1.1 B 1.1B 1.1

A 1.0

A 1.2A 1.1 B 1.2B 1.1

B 1.2 A 1.2

A 1.2A 1.1 B 1.2B 1.1

B 1.0

Production Visibility

10:07

Prepare for a rainy day

How does it work – CD Practices

• Test driven development• Small Development Iterations• Backwards and Forwards compatible• Gradual Deployment & Self-Test• Feature Toggle• A/B Testing• Exception Classification• Production visibility

10:07

Where are we today?

• We have re-written our flash editor product as an HTML 5 editor– In just 4 months

• Introduced Wix 3rd party applications (developers API)– In just 6 weeks

• We are easily replacing significant parts of our infrastructure

• And we are doing ~10 releases a day!

10:07 1/1/2013 1/14/2013 1/28/2013 2/10/2013 2/24/2013 3/10/2013 3/24/2013 4/14/2013 4/29/2013 5/13/2013 5/28/2013 6/9/20130

5

10

15

20

25

30

35

Total

Total

10:07

10:07

Aviran Mordo@aviranm

http://www.linkedin.com/in/aviran

Read more: The Road To Continuous Delivery: http://goo.gl/K6zEK

http://www.slideshare.net/aviranwix/strategies-in-continuous-delivery