Download - Strategies in continuous delivery
Strategies In Continuous Delivery
Learn the “magic” behind the scenes
Aviran MordoServer Group Manager @ Wix
@aviranmhttp://www.linkedin.com/in/aviran
10:07
About Wix
10:07
Wix in Numbers
• Over 35,000,000 users– Adding over 1,000,000 new users each month
• Static storage is over 150TB of data– Adding over 1TB of files every day
• 3 Data centers + 2 Clouds (Google AE, Amazon)– Around 300 servers
• Over 40,000,000 Server API calls per day• ~400 people work at wix
– Over 100 people in R&D
10:07
Is Doing Multiple Deployments A Day Takes
Guts ?10:07
10:07
From Jan’ 2013 – Jun’ 2013
• 1500• 470• 200
10:07
From Jan’ 2013 – Jun’2013
• 1500 Deployments• 470 A/B Tests• 200 Feature Toggles
Continuous Delivery at Wix
• Abandon “VERSION” paradigm – move feature centric life
• Make small and frequent release as soon as possible
• Automate everything – TDD/CI/CD• Measure Everything
– A/B test every new feature– Monitor real KPIs (business, not CPU)
10:07
Continuous Delivery at Wix
• Empower the developer–The developer is responsible from product idea
to 1M active users–Remove every obstacle in the developer’s path–Big cultural change from waterfall – affects the
whole company
10:07
Test Driven Development
• No new code is pushed to Git without being fully tested
• Before fixing a bug first write a test to reproduce the bug
• Cover legacy (untested) systems with Integration tests
10:07
Test Driven Development
• What people think is the impact on development–TDD slows down development–With TDD we write more code (product + test
code).
• Current Test Count (U-Tests + IT-Tests) – over 10,000
10:07
Test Driven Development
• Actual impact on development–We develop products faster–Removes fear of change–Easier to enter some one else’s project–Do we really need QA? (Yes, they code tests)–Writing a feature is 10-30% slower, 45-90%
less bugs–50% faster to reach production.–Considerably faster time to fix bugs10:07
Guidelines for successful TDD
• Tests should run on project checkout to a random computer.
• Tests that cannot be debugged on a developer machine will never consistently run for any period of time
• Tests should run fast• Tests have to be readable – They are the
project’s specs• Fixture is evil!10:07
Feature Toggles
10:07
Feature Toggles
• Everyone develops on the Trunk
• Every piece of code can get to production at anytime
10:07
Feature Toggle to the rescue
• Unused new code can go to production – no harm done
• Operational new code goes with a guard – use new or old code by feature toggle
10:07
Usage example
Simple “if” statement in your code
10:07
Feature Toggle Strategies (gradual expose users)
• Company employees• Specific users or group of users• Percentage of traffic• By GEO • By Language• By user-agent• User Profile based• By context (site id or some kind of hash on site id)
10:07
Feature Toggle Override
• By specific server– Used to test system load– New database flows/migration– Refactoring that may affect performance and memory usage
• By Url parameter– Enable internal testing– Product acceptance– Faking GEO
• By FT cookie value– Testing– When working with API on a single page application
10:07
Feature Toggles Management
10:07
A/B Tests
10:07
A/B Test
• Every new feature is A/B tested• We open the new feature to a % of
users– Define KPIs to check if the new feature is
better or worse – If it is better, we keep it– If worse, we check why and improve– If we find flaws, the impact is just for %
of our users (kind of Feature Toggle)
10:07
An interesting site effect on product
• How many times did you have the conversion “what is better”?– Put the menu on top / on the side
• Well, how about building both and A/B Testing?
10:07
Marking users with toss value in a cookie
• Anonymous user– Toss is randomly determined– Can not guarantee persistent experience if changing browser
• Registered User– Toss is determined by the user ID– Guarantee toss persistency across browsers– Allows setting additional tossing criteria (for example new users only)– Only use this for sections that a user has to be authenticated
10:07
• Do not mix anonymous and registered tests
• AB test parentage of users with optional filters– New Users Only (Registered users only)– By language – By GEO– By Browser – user-agent – OS– Any other criteria you have on your users
10:07
A/B Test Features
• A/B Test Override– Allows setting a value of a test for validation– Helps support experience what users experiencing
• Override methods– Via URL parameter– Via cookie
• Start/Stop Test• Pause tests• Bots always get “A”10:07
Manage your tests
10:07
Refactoring
10:07
Refactoring
• Before refactoring make sure everything is covered with tests– Legacy code usually covered by IT tests
• Refactor from inside out– Small iterations with tests– Refactor small methods - make sure the tests don’t
break– Deploy often
• Re-write from the outside in– Write from scratch– Code duplication sometimes needed (temporary)– Protected by Feature Toggle
10:07
Code branch
10:07
New Code Old Code
FT Opene
d
Yes No
DB Schema Changes Without Downtime
• Adding columns– Use another table link by primary key– Use blob field for schema flexibility
• Removing fields– Stop using. Do not do any DB schema
changes
10:07
New DB schema
• Plan a lazy migration path controlled by feature toggle
1. Write to old / Read from old2. Write to both / Read from old 3. Write to both / Read from new, fallback to old4. Write to new / Read from new, fallback to
old5. Eagerly migrate data in the background6. Write to new / Read from new
10:07
Gradual Deployment
10:07
• Assume two components
• We shutdown one and install on it the new version. It is not active yet
• Do self test• Activate the new server it is passes self test
• Continue deploying the other servers, a few at a time, checking each one with self test
A 1.1 B 1.1
A 1.1B 1.2
A 1.1A 1.1 B 1.1B 1.1
A 1.1A 1.1B 1.1
B 1.2
A 1.1B 1.2
A 1.1A 1.1B 1.1
B 1.2
A 1.1 B 1.1A 1.1A 1.1 B 1.1B 1.2
Self Test / Post Deployment Test
After each server deployment run a self test before deploying the next server.
• Checking server configuration and topology– Make sure database is accessible (DB connection string)– Is the schema the one I expect– Access required local resources (data files, other config files,
templates, etc’)– Access remote resources– RPC / REST endpoints reachable and operational
• Server will refuse requests unless it passes the self test• Allow a way to skip self test (and continue deployment)10:07
Tools - App-info
10:07
Backward and Forward compatible
• Assume two components
• We release a new version of one
• Now Rollback the other…
10:07
A 1.1B 1.2
A 1.2
B 1.1A 1.1A 1.1B 1.1
B 1.2
A 1.2A 1.1B 1.1B 1.1
A 1.1 B 1.1A 1.1A 1.1 B 1.1B 1.1
A 1.0
A 1.2A 1.1 B 1.2B 1.1
B 1.2 A 1.2
A 1.2A 1.1 B 1.2B 1.1
B 1.0
Production Visibility
10:07
Prepare for a rainy day
How does it work – CD Practices
• Test driven development• Small Development Iterations• Backwards and Forwards compatible• Gradual Deployment & Self-Test• Feature Toggle• A/B Testing• Exception Classification• Production visibility
10:07
Where are we today?
• We have re-written our flash editor product as an HTML 5 editor– In just 4 months
• Introduced Wix 3rd party applications (developers API)– In just 6 weeks
• We are easily replacing significant parts of our infrastructure
• And we are doing ~10 releases a day!
10:07 1/1/2013 1/14/2013 1/28/2013 2/10/2013 2/24/2013 3/10/2013 3/24/2013 4/14/2013 4/29/2013 5/13/2013 5/28/2013 6/9/20130
5
10
15
20
25
30
35
Total
Total
10:07
10:07
Aviran Mordo@aviranm
http://www.linkedin.com/in/aviran
Read more: The Road To Continuous Delivery: http://goo.gl/K6zEK
http://www.slideshare.net/aviranwix/strategies-in-continuous-delivery