continuous delivery and continuous agile by andy singleton - agile maine day 2016
TRANSCRIPT
![Page 1: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/1.jpg)
Continuous Deliveryand Continuous Agile
From Andy Singleton, http://continuousagile.com
www.assembla.com
![Page 2: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/2.jpg)
The secret weapon that Silicon Valley is using todisrupt and destroy competitors
• Retailer X deploys changes to their monolithic online ordering app once every six weeks. Ops holds for three weeks to make sure the complete system is stable.
• Amazon has thousands of services and more than 1000 service teams. They release something about once every 11.6 seconds. In the time that Retailer X takes to try one new release, Amazon has made 100,000 changes.
• Amazon hosting competitor: “It’s an emergency”.• 6 Apex Predators in the top 10 public companies worth
$2.4 trillion: Google, Amazon, Facebook, Microsoft, GE, Apple
![Page 3: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/3.jpg)
Survey on Continuous Delivery
46% think their competitors
have adopted Continuous
Delivery
![Page 4: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/4.jpg)
Problems solved by Continuous
• Stressful releases
• Speed to market• Competitiveness• Quality
• Scaling – building large systems
![Page 5: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/5.jpg)
Not Scrum
• Continuous improvement versus CadenceUnblock your teams
• Web services versus monolithsSmaller, faster testing, faster releases
• Technical practices, not team practicesHUGE productivity gains from automation
• Tech leads versus self-managing teamsOne leader, ready to build services
• MAXOS versus SAFeScale different
![Page 6: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/6.jpg)
改善1. リリースを頻繁にする2. 改善する
/ 月割りのリリース
![Page 7: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/7.jpg)
Kaizen
1. Release more frequently2. Improve
![Page 8: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/8.jpg)
Continuous Agile Agenda
1) DevOps - Continuous delivery of software – Build/Test/Deploy
2) Continuous, metric-driven product management
3) MAXOS enterprise management
4) Making the Move
![Page 9: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/9.jpg)
Continuous Deliveryor “DevOps”
![Page 10: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/10.jpg)
Scrum SprintValidate and
Sort DeliverablesSelect a
Sprint Plan
Completed = observed
velocity
Expand stories and estimate
tasks
CollectIdeas
Close Sprint
Into next sprint
Work on tasks
![Page 11: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/11.jpg)
Scrumban Iteration
Validate and Sort Deliverables
Pull DeliverablesWhen Ready
CollectIdeas
StabilizeRelease
Into next release
Feature Freeze
Triage
![Page 12: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/12.jpg)
Idealized ScrumBan Release
Copyright 2012 Perforce Software, Inc. - Assembla, Inc.
![Page 13: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/13.jpg)
Kanban / ContinuousValidate and
Sort DeliverablesPull Deliverables
When ReadyCollectIdeas
Continuous Release
Stay within WIP limit
Release Features Continuously
![Page 14: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/14.jpg)
Program Launch
Releases
Plan Program Test Doc Deploy
SkipAutomate& Blend Lag Automate
Pull
CI & CD
End up with
Waterfall to Continuous
Measure
![Page 15: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/15.jpg)
Code Contribution PatternsManage code if possible. People are hard to manage and can’t be automated. They want to contribute.• Centralized continuous delivery
– No branches, finds and fixes problems as early as possible• Distributed continuous delivery
– Release every change with its own branch and test• Temporary branches
– Combines benefits of centralized and distributed• MAXOS
– Use centralized continuous integration to manage a massively scalable IT system
![Page 16: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/16.jpg)
Centralized CI/CDContributor Commits – “as early as possible” to find problems
ContinuousIntegration tests
Fail - alarm
Release CandidateTest System Release
QA Testing
Pass
![Page 17: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/17.jpg)
Continuous delivery at Edmunds.comwith PERFORCE
![Page 18: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/18.jpg)
The big question: How to test?
• We release software in batches so that we can test it.
• In continuous delivery, we might get as little at 10 minutes to test a release candidate
![Page 19: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/19.jpg)
Test Layering
Monitor your released software: Errors, Usage volume, usage patterns, user feedback
QA System with Human test consultants
Code review: Both a manual test, and a place to ask for test scripts.
Continuous integration: Run automated tests before using human review time
Unit tests in the development environment
Switch new features and architectureStart h
ere to
add layers
Start here to
release a change
More frequent releases can increase quality
![Page 20: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/20.jpg)
9 (sparse) Layers at Edmunds
![Page 21: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/21.jpg)
![Page 22: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/22.jpg)
Feature Switch and Unveil
HiddenProgrammer sees a change locally. Change is tested in the main version but not seen.
Test Story Owner and testers see the change on test systems.
BetaInsiders see it and use it. Story Owner can show it to selected users for feedback or A/B testing.
UNVEIL! The big event. Communicate with all users. Measure reaction.
One code version
No special test builds
No long-running branches
![Page 23: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/23.jpg)
Role: Developer
• Developers have more power and responsibility.• Developers have more responsibility for testing.• Developers (not QA or PM) decide when to release.
This is a strong finding.• Incentives are correct. Developer might have to
come back from Friday night beers to fix a problem. This provides a motivation to make good decisions and automate testing.
• Features can be released but hidden. Product Managers and Marketers will unveil when they are ready. Unblock!
Programmers approve releases, not QA
![Page 24: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/24.jpg)
Role: IT VendorOld Agreement• Fixed Price• Fixed deliverables• Large probability of late delivery
New Agreement• Fixed price• Fixed Time• Variable deliverables
– Small guaranteed deliverable– Weekly meetings to agree on improvements
Advantages• No late delivery• Deliverable is better
than original specification
• Start sooner and finish sooner
![Page 25: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/25.jpg)
Role: Product Manager/Owner
• Batch -> Continuous
• Requirements -> User Experience
• Strategy -> Measurement– Usage measurements are so important, so
underutilized– Double your productivity
![Page 26: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/26.jpg)
Batch Agile Continuous
Explains the resurgence of vertical integration(eg Apple, Tesla). Requires CI
![Page 27: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/27.jpg)
Making one good change is not a team sport
Product Owner -> Story Owner
![Page 28: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/28.jpg)
What you realize when you begin to be very data-driven is a very large percentage of our ideas are bad.
Microsoft Product CTO Adam Pisoni
![Page 29: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/29.jpg)
Measure everything• Measurements are power. They are impossible to ignore. They
win arguments.
• Double your development capacity for zero extra cost! Users ignore 50% of all new features. If you can find and ignore these features, you double your capacity.
• Measure everything possible about usage. Easy with an online service.
• Collect measurements from installed products and optionally transmit them.
• Use your marketing department. They measure.
![Page 30: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/30.jpg)
Make Product, not Quality
ContinuousRelease
InventoryRelease
Improve Quality
Improve Product Usage
![Page 31: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/31.jpg)
Matrix of Services(MAXOS)
Breaking the scale barrier
![Page 32: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/32.jpg)
The Services MegatrendDesktop Web App Cloud
Services
App
DB
Service
Service
Service
![Page 33: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/33.jpg)
Scale it like Google
• 15,000 developers, 5,000 projects, one current version of the code (2013). They can go from an idea to a release in 48 hours
• Vast Internet system divided into thousands of "services"• Most programming done by teams of 3-4• Centralized process with single version of the test system – run
100 million test cases daily• Before developers release a change, they test it with the most
recent version of all the other services. If a test script finds conflicts, it tells developers who to contact to resolve them
![Page 34: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/34.jpg)
Matrix of Services - MAXOS
PrioritizedBacklog
CurrentWork
Each team releases
when ready
Hundredsof releases
per day
Service team Productionservice
Service team Productionservice
Service team Productionservice
Feedback on speed, errors, usage, and requests
Test as one system
Integrationtest env
Integrationtest env
Integrationtest env
![Page 35: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/35.jpg)
Coordinate without big meetingsContinuous Integration between latest dev version of each service
• Continuous integration helps teams coordinate.
• See dependencies between “producers” and “consumers”
• Errors and conflicts show related team contact info
• Meetings and changes negotiated between two teams, not many
PrioritizedBacklog
CurrentWork
Service team
Service team
Service team
Integrationtest env
Integrationtest env
Integrationtest env
Machines can replace layers of management
![Page 36: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/36.jpg)
Teams are largely self-managing
PrioritizedBacklog
CurrentWork
Service team Integrationtest env
Up to 50% of workfrom backlog
At least 50% of work is self-plannedProblems get fixed quickly
Productionservice
Productionservice
Productionservice
Feedback: quality, reliability, speed, user support
Productionservice
ProductionServer
Sense, respond, self manageminimize planning
![Page 37: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/37.jpg)
Hubspot – Great at Mid-scale• Transformed a monolithic app to 200 services over one year• 3-person programming teams. Each of 20 teams is
responsible for about 10 services• Dev teams responsible for design, programming, testing,
release, monitoring, and responding to production problems. No full-time QA. Shared PM and UX. 4 Ops guys for 2000 servers.
• Lot’s of tooling and dashboards to help teams deploy, manage, and monitor their services
• Feedback from customer support also grouped by team
![Page 38: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/38.jpg)
Team Building
PrioritizedBacklog
CurrentWork
Each team releases
when ready
Service team Productionservice
Service team Productionservice
Add capacity fast by building aroundSingle-function programmer/tech leads
Integrationtest env
Integrationtest env
Teams are not permanently multifunctional
![Page 39: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/39.jpg)
Ways to Scale
Scrum + SAFe• Add more hierarchy• Complex multifunction
teams• Hold big meetings and
teleconferences• Block everyone into
one cadence• Coordinate big releases
Top Tech CompaniesAutomate management,
as well as testing and deployment.
Dev-lead teamsCommunicate peer to
peerUnblock teams to move as
fast a possibleRelease more frequently
![Page 40: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/40.jpg)
SAFe versus MAXOS
![Page 41: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/41.jpg)
How to move to CD and Services
• Measure• Release more frequently• Assign blame and other organizational and
authority hacks• Automate• Organize the fast layer• Reverse Conway’s Law• Manage products, not programs
![Page 42: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/42.jpg)
Measure
People respond directly to measurements. They are very difficult to ignore. They are also MORE IMPORTANT than internal process discussions.• Usage and customer value• Errors• Fix times• Release frequency
![Page 43: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/43.jpg)
Release more frequently
1. Release more frequently2. Improve
![Page 44: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/44.jpg)
Authority Hacks
Training, persuasion, browbeating, and “culture change” will not work. They don’t change the game. You first change the game. Then, people figure out how to play the game and win.• Need automated tests? Mandatory code review• Need higher quality and shorter test cycles? Force
programmers to push the “release button”, not QA• Stop your boss from launching big waterfall
projects? Give him monthly budgeting, not annual.
![Page 45: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/45.jpg)
Code Review gets you tests
AutomateIf the machine works, people will use it
![Page 46: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/46.jpg)
Core ITannual budget
Reliability & security mission
WebAPI Layer
Mobile SaaS / Cloud
Marketing
Fast (Awesome Mode) ITmonthly budget
Mission to respond to opportunities
![Page 47: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/47.jpg)
Service team Productionservice
Service team Productionservice
Productionservice
Integrationtest env
Integrationtest env
Integrationtest env
Fast IT
Core IT (stable service)
![Page 48: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/48.jpg)
Reverse Conway’s Law
Boss
Dept A Dept B
PartA
PartB
Service A1
Service A2
Service B1
Service B2
Team A1
Team A2 Team B2
Team B1
![Page 49: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/49.jpg)
Manage Products, not Programs
• Strong product managers deliver value to users as directly as possible
• The Matrix of Service teams respond to product managers. The front end teams provide the product. They act as “consumers” to pull what they need from back-end teams
• Service teams support multiple products, dissolving the program hierarchy
![Page 50: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/50.jpg)
Product Matrix
Boss
Dept A Dept B
PartA
PartB
Service A1
Service A2
Service B1
Service B2
Team A1
Team A2 Team B2
Team B1
![Page 51: Continuous Delivery and Continuous Agile by Andy Singleton - Agile Maine Day 2016](https://reader036.vdocuments.site/reader036/viewer/2022062503/588483181a28ab6d1a8b49e5/html5/thumbnails/51.jpg)
People first, or Technology first?
• Your comments