continuous delivery in practice (public)

Continuous Delivery In Practice

Lessons from Kenshoo’s RTB project

Who, What, Where

Tzach Zohar:● System Architect● tzach.zohar@kenshoo.com

Kenshoo: ● Founded 2006● Online Marketing Technology● ~450 employees● 12 World Wide locations

Continuous Delivery: Why bother?

Faster development, delivery and feedback makes our clients and us happier.

The “us” alone is worth it.

Continuous Delivery: Why bother?

Better suited productResponsiveness

Less wasteHigher quality

Background: RTB Project

● ~1.5 years ● ~4 developers● ~Dozens of paying clients● ~50 servers (AWS)● ~1.5M requests per minute● ~7ms average response time● ~99.9% availability● ~5-10 deployments / week

How?

1.The Obvious

● Single branch● Full, Fast, Reliable coverage● Full deployment automation● Fast feedback● ABCD - Always Be Continuously

Deploying

● Unit: complete functional coverage● Integration: with external systems - thin!● Behavioral: we use Cucumber● Staging: verify actual server upgrade

2. Four-Layer Test Suite

2. Four-Layer Test Suite

Staging: verify compatibility of new build with other components’ production builds

2. Four-Layer Test Suite

3. Keep Builds Stable

Do not overlook a test that “sometimes fails”, trusting build status is crucial

3. Keep Builds Stable

Be Suspicious of:● Random data tests● Asynchronous tests● Integration tests

4. Master Is Always

Shippable

On every commit? Not Quite:Use GitHub Pull Requests

“Merge” == Build and Deploy

credit: tal.salmona@kenshoo.com

5. Rigorous Code Reviews

● Remember “merge” means “deploy”!● Insist on proper coverage● Insist on code cleanliness● Insist on consistent design● Insist!

5. Rigorous Code Reviews

https://github.com/tzachz/github-comment-counter

6. Real-Time Feedback

Detect issues immediately and visually

7. Keep Upgrade in Mind (1)

Use the “Parallel Change” pattern when changing cross-node APIs / Data

1.Write: oldRead: both

2.Write: new Read: both

3.Write: new Read: new

deploy deploy

8. Keep Upgrade in Mind (2)

Verify backward compatibility in tests

9. A/B Testing

Apply new features to a limited user-group Measure business results per-group

(Not by branching)

10. Own It

Constantly check buildsConstantly collect feedbackConstantly check monitorsAnswer the phone at 3am

10. Own It

That’s It.

Appendix A. Partial Tool List

Testing: JUnit, Cucumber, NoseBuild / CI: Jenkins, Gradle, JaCoCo

Code Review: GitHubProvisioning: Puppet

Deployment: Fabric, botoMonitoring: Metrics, Graphite

Appendix B. Are You Ready?

Unit Coverage > 90%?

Good Staging Tests?

Informative Monitors?

Builds Are Kept Green?

No API Breaking Changes?

Rigorous Code Reviews?

Support Has Your Phone Number?

Do You Own it?

Not Ready

No Yes

credit: tal.salmona@kenshoo.com

Thanks. Questions?