Download - Continuous delivery in practice (public)
Continuous Delivery In Practice
Lessons from Kenshoo’s RTB project
Who, What, Where
Tzach Zohar:● System Architect● [email protected]
Kenshoo: ● Founded 2006● Online Marketing Technology● ~450 employees● 12 World Wide locations
Continuous Delivery: Why bother?
Faster development, delivery and feedback makes our clients and us happier.
The “us” alone is worth it.
Continuous Delivery: Why bother?
Better suited productResponsiveness
Less wasteHigher quality
Background: RTB Project
● ~1.5 years ● ~4 developers● ~Dozens of paying clients● ~50 servers (AWS)● ~1.5M requests per minute● ~7ms average response time● ~99.9% availability● ~5-10 deployments / week
How?
1.The Obvious
● Single branch● Full, Fast, Reliable coverage● Full deployment automation● Fast feedback● ABCD - Always Be Continuously
Deploying
● Unit: complete functional coverage● Integration: with external systems - thin!● Behavioral: we use Cucumber● Staging: verify actual server upgrade
2. Four-Layer Test Suite
2. Four-Layer Test Suite
Staging: verify compatibility of new build with other components’ production builds
2. Four-Layer Test Suite
3. Keep Builds Stable
Do not overlook a test that “sometimes fails”, trusting build status is crucial
3. Keep Builds Stable
Be Suspicious of:● Random data tests● Asynchronous tests● Integration tests
4. Master Is Always
Shippable
On every commit? Not Quite:Use GitHub Pull Requests
“Merge” == Build and Deploy
credit: [email protected]
5. Rigorous Code Reviews
● Remember “merge” means “deploy”!● Insist on proper coverage● Insist on code cleanliness● Insist on consistent design● Insist!
5. Rigorous Code Reviews
https://github.com/tzachz/github-comment-counter
6. Real-Time Feedback
Detect issues immediately and visually
7. Keep Upgrade in Mind (1)
Use the “Parallel Change” pattern when changing cross-node APIs / Data
1.Write: oldRead: both
2.Write: new Read: both
3.Write: new Read: new
deploy deploy
8. Keep Upgrade in Mind (2)
Verify backward compatibility in tests
9. A/B Testing
Apply new features to a limited user-group Measure business results per-group
(Not by branching)
10. Own It
Constantly check buildsConstantly collect feedbackConstantly check monitorsAnswer the phone at 3am
10. Own It
That’s It.
Appendix A. Partial Tool List
Testing: JUnit, Cucumber, NoseBuild / CI: Jenkins, Gradle, JaCoCo
Code Review: GitHubProvisioning: Puppet
Deployment: Fabric, botoMonitoring: Metrics, Graphite
Appendix B. Are You Ready?
Unit Coverage > 90%?
Good Staging Tests?
Informative Monitors?
Builds Are Kept Green?
No API Breaking Changes?
Rigorous Code Reviews?
Support Has Your Phone Number?
Do You Own it?
Not Ready
No Yes
credit: [email protected]
Thanks. Questions?
