scaling continuous delivery @ geecon 2014
Post on 11-May-2015
773 Views
Preview:
DESCRIPTION
TRANSCRIPT
ScalingContinuous Delivery
Tomas Riha
Tomas Riha
Architect @ VGT/WirelessCar
Passionate about creativity, change and improvementHorrible at following instructions and performing repetitive tasks
MAJOR Project Liability
mail: triha74@gmail.comtwitter: @TomasRihaSE
blog: continuous-delivery-and-more.blogspot.comslides: www.slideshare.net/TomasRiha
Three Years ago
New Telematics Delivery Platform
Multiple Stakeholders
Continuous Regression Testing
How hard can it be?
Regression Test
Unit Testing
Component Testing
System Testing
Rollback/Compatibility Testing
Failover Testing
Performance Testing
At first it was super easy!
Small team of just product owner architect and scrum master.
Huge productivity, natural test driven development.
Fast return on investment
All by the book.... we were doingContinuous Delivery!
Then we tried to scale it...
... and we failed in every possible way.
We lost our test driven developmentWe lost the individual responsibility
We more or less became a automated waterfall unable to delivery daily
We lost our ability to Continuously Improve the process
Three components of Continuous Delivery
Process & ImplementationDefinition of the Software Delivery process and its lead time optimization
Product IntegrityThe Architecture & Test Strategy to optimize product integrity and lead time.
People & OrganizationBehavioural change to the individuals and the organization needed to optimize
lead time and increase product integrity.
Process & Implementation“Dedicated test servers”
First Continuous Delivery implementation relied on standard corporate test environment with one system test server, one integration test server and one
pre prod server per production delivery
Only one commit tested at the time.
Build Release Deploy Test
DB
Test Server
Process & Implementation“Single threaded process”
Stacking jobs became a problem at 5 code committers, almost killed us at 20
Feedback time == Stack Depth * Pipe Exec Time
Variable feedback time results in less responsible developers
We were somewhat helped by component based architecture
Process & Implementation“First effort to scale”
Forced to use our internal Operations as infrastructure provider, no cloud and very rudimentary virtualization
Built a Server Pool based on Jenkins slaves and “sticky pipes”
Primitive but multi threaded pipes
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Process & Implementation“Crowded Environment”
Visibility was bad due to hundreds of jobs on same build system.
Teams blocking each other due to still insufficient server resources.
Traceability hard due to all server clutter.
Changes to one teams Test Environment affected the other teams.
Overall performance was horrible at peak with Jenkins totally dieing
Process & Implementation“Continuous Delivery of Continuous Delivery”
Live what you teach!
Our ability to Continuously Improve was seriously slowed by lack of Continuous Delivery
Deployment Scripts didn't have test automation.
No test environment for Continuous Delivery Process
Build pipes where manually maintained.
Jenkins servers where manually maintained.
Process & Implementation“Autonomous Environments for Autonomous Teams”
Started to do Continuous Delivery of Continuous Delivery
Automated provisioning of Continuous Delivery Environment
Created template based pipeline generator
Built Autonomous Continuous Delivery environments for each team
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Process & Implementation“Portability is Scalability”
Portability in the Runtime Environments
Dev
Test
Load Test
UATPROD
INTTest
Complexity
Scale
Process & Implementation“Portability is Scalability”
If the Continuous Delivery process can run without a Build Server then it can scale to X number of build servers
By ensuring that anything above the portability line can run local or remote to the orchestration we ensure portability
Orchestration
ProvisionBuild Deploy Test Release
Test EnvironmentSCM ArtifactRepoPortability Line
Process & Implementation“Portability is Scalability”
All environments provisioned and deployed in the same way
Orchestration
ProvisionBuild Deploy Test Commit
Orchestration
ProvisionBuild Deploy Test Release
Orchestration
Provision Deploy Test Accept
Orchestration
Provision Deploy Test
Local Dev Continuous Delivery Pipe
UAT Prod
Process & Implementation“Portability is Scalability”
Dev Test Load Test UATINT
Test
Provisioning Interface
Vagrant Impl Cloud Provider Impl PDF Document Impl
Topology Spec Env Spec
LegacyPROD
With the help of Cloud Technology and Vagrant we manage to create test and dev environments with the same complexity as our production
environments.
Process & Implementation“Portability is Scalability”
To make the Continuous Delivery Process portable we also need to move all reporting and monitoring out of the test servers and the build servers
Orchestration
ProvisionBuild Deploy Test Release
Test EnvironmentSCM ArtifactRepo
Logging & Metrics Services
Build Data Repository
Process & Implementation“Portability is Scalability”
Orchestration Worker
Log, Metrics, Build Data Repositories
WorkerWorker
Build Env
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Team Environment Pre CloudOne Server Per Pipe Shared Database for all pipes
Team Environment in CloudOne Environment Per PipeIncluding Load Balance, App Servers & DB
Separating Data from Process Implementation improves the traceability increases dramatically.
Portability in Test Environments increases Quality and Scalability by removing bottlenecks in shared infrastructure.
WorkerWorkerWorker
Test Env
Worker
Worker
Worker
WorkerWorkerTest Env
WorkerTest EnvTest Env
Process & Implementation“Provisioning, Install and Dependency Management”
If one or more environments is provisioned manually it becomes a bottleneck
Application artifacts need dependency management to infrastructure.
Infrastructure has to be provisioned and installed with the application.
Databases have to be upgraded and migrated as part of the automated process.
An automated process doesn't pause for DBAs or SysAdmins to do manual work.
People & Organization“The Tools Team”
Initially we did development of the Continuous Delivery process as skunkworkz.
Quickly realized this was a bottleneck.
Added a CM to our team with disastrous result.
Finally created a Tools Team.
People & Organization“The Tools Team is a Bottleneck”
Tools Team a huge bottleneck, simplest issues took ages to fix.
Managed the Continuous Delivery process manually.
Only ones that understood the process.
Became responsible for every teams ability to interface with the process.
Only had time to do reactive maintenance.
Short on resources.
Became yet another IT department.
People & Organization“Build Police, please dont!”
Only Tools Team and few others understood the process.
Tried to help the few others by adding a rotational “Build Police” Role, as bad idea as it sounds.
People & Organization“The Tools Team matures”
Continuous Delivery as a Service
Tools Team responsible for the process implementation, its interfaces and the infrastructure it runs on.
The Development Teams responsible for integrating with these interfaces, configuring the pipe generator and maintaining green state of pipes.
Tools Team supports and helps if there are bugs in the mechanisms.
Tools Team consists of a core team of Developers, Testers and Operations specialists but is extended with part time resources from Development
Teams.
People & Organization“Lack of Consensus”
Disagreements on shared responsibility for the release.
Disagreements on what test automation is and how it should be used.
Disagreement on who implements tests and how.
A Continuous Delivery Engine without the consensus of the people that use it is just an overpriced CI System.
Product Integrity “Build Quality In”
SYSTEM
GUI
Product Integrity “Build Quality In”
Internal Service
GUI
Internal Service
Internal Service
Service API
Product Integrity “Build Quality In”
Internal Service
Internal Service
Internal Service
Each Service Tested as Black Box in isolationand with high detail level
Product Integrity“Build Quality In”
GUI
Service API Mock Impl
GUI Test are much faster and robust if they test the GUI in isolation. High detail level.
Product Integrity“Build Quality In”
Internal Service
GUI
Internal Service
Internal Service
Service API
Verifying Use Case acceptance criterias. Low level of detail
Product Integrity“Build Quality In” Optimizes Lead Time
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Assemble Release Deploy Test
Component Pipe
1000s of tests run in parallel testing
System Pipe
10s Use Case Requirements Verified
Product Integrity“Build Quality In”
An Architecture & Test Strategy that goes hand in hand increases product integrity and optimizes the lead time and allows the Continuous Delivery
process to scale
People & Organization“Empower the Team!”
Increase Team responsibility.
Team is responsible for everything within the Continuous Delivery process
Analyzing RequirementsDefining Requirement Verification
Automating Requirement VerificationImplementing the System
Shipping the SystemSupporting the System
Create Team consensus on what its responsibility and delivery is.
People & Organization“Empower the Team!”
To help empower teams we have
Autonomous Team Environments where teams are responsible for their own Continuous Delivery increased the teams understanding of the process.
Teams that needed extra help sent one or more individuals to work part time on Tools Team.
Architecture & Test Strategy that works well with Continuous Delivery
Cross functional communities created to help increase consensus on Test Driven Development, Continuous Delivery and Architecture.
People & Organization“Test Driven Development Community”
Developers need to take more responsibility.
Code has to work all the time.
Test Driven Development requires developers to participate in specification of requirement verifications.
Test Automation is Code, Developers need to code much more tests.
Continuous Regression Testing means Developers get instant feedback and have to act on it.
People & Organization“Test Driven Development Community”
Test Profession changes.
Two main type of Test activities Test Automation and Exploratory Testing.
Test Automation is Code.
Let developers test.
Test Driven Development is Proactive.
Exploratory testing is NOT manual regression testing.
People & Organization“Continuous Delivery Community”
Agreement on Practices of Continuous Delivery
Agreement on Practices of Provisioning & Deployment
Competence Development
Infrastructure as Code
Testing Infrastructure as Code
Operations Specialists need to work with Developer tools and languages
People & Organization“Organizational Support”
Continuous Delivery is organizational change.
Scaling Continuous Delivery is as much creating a scalable agile organization as it is creating a scalable implementation.
Organization has to fully support Continuous Delivery through re-organization.
Organization has to fully support and invest in infrastructure, architecture, automation, process and test.
Organization needs to start measuring key values, such as cycle time, early.
Summary
Process & ImplementationPortability is Scalability.
Product IntegrityThe Architecture & Test Strategy go hand in hand.
People & OrganizationEmpowered Teams and Consensus.
Thats it!
Feedback & Any questions you forgot to ask?
http://continuous-delivery-and-more.blogspot.se@TomasRihaSE
or bythe Coffee stand!
And the slides are athttp://www.slideshare.net/TomasRiha
top related