maxos - the new scalable continuous agile
DESCRIPTION
This talk got a good reception at Agile2014. I introduced MAXOS (Matrix of Services) as very different alternative to the old Scrum and SAFe way of organizing software development. MAXOS is a pattern used by top technology companies like Google, Amazon, Hubspot, and Edmunds. It uses continuous delivery from many different "service teams" in parallel, and it replaces a lot of management coordination with continuous integration. I highlighted in yellow a series of "debatable points" that are surprising to more traditional agile developers.TRANSCRIPT
![Page 1: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/1.jpg)
MAXOS Scalable Continuous Agile
with debatable points
From Andy Singleton, http://continuousagile.com
www.assembla.com
![Page 2: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/2.jpg)
Microsoft Journey
• Scale: Releases were buggy at scale• Stress: Releases were stressful• Online: SaaS, devices and services, cloud first• Competition: The competition was releasing
more frequently and moving faster
• Now, about 80% of the way to “second generation agile”
![Page 3: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/3.jpg)
Survey on Continuous Delivery
46% think their competitors
have adopted Continuous
Delivery
![Page 4: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/4.jpg)
Ways to Scale
Scrum + SAFe• Add more hierarchy• Hold big meetings
and teleconferences• Block everyone into
one cadence• Coordinate big
releases
Top Tech CompaniesAutomate management,
as well as testing and deployment.
Communicate peer to peerUnblock! teams to move
as fast a possibleRelease early and often.
Separate release from launch
![Page 5: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/5.jpg)
2 Ways to Be More Productive
![Page 6: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/6.jpg)
1) Do the right thing
Users ignore at least 50% of the new stuff you try. If you can measure usage or value and figure out what to ignore in development, you can increase development productivity by 100% for zero extra cost.
Measurement is very important.
More frequent releases = more measurements
![Page 7: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/7.jpg)
2) Use more machines
![Page 8: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/8.jpg)
Better organization is not high on the list of ways to get more productive. Does not change much over centuries.
Are the methodology booths in the exhibit hall are being pushed out by vendors that make build, test, and deploy tools?
“Technical practices”
Productivity comes from more/bigger machines
![Page 9: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/9.jpg)
Continuous Delivery Basics
• Code contribution patterns• Test layering• Automation• Developer responsibility• Feature switches
![Page 10: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/10.jpg)
Code Contribution Patterns
Manage code if possible. People are hard to manage and can’t be automated. They want to contribute.• Centralized continuous delivery
– No branches, finds and fixes problems as early as possible• Distributed continuous delivery
– Release every change with its own branch and test• Temporary branches
– Combines benefits of centralized and distributed• MAXOS
– Use centralized continuous integration to manage a massively scalable IT system
![Page 11: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/11.jpg)
Centralized CI/CDContributor Commits – “as early as possible” to find problems
ContinuousIntegration tests
Fail - alarm
Release CandidateTest System Release
QA Testing
Pass
![Page 12: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/12.jpg)
Continuous delivery at Edmunds.com
![Page 13: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/13.jpg)
Distributed Continuous DeliveryContributor Commits
Branchor Fork
Deployed version
Peer review merge requests
Other contributionsmerged and released“as late as possible”
QA Consults
Pass FinalAuto Test?
Merge backCurrent
Deploy
![Page 14: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/14.jpg)
Distributed: Multiple Test Systems
Contributor 1
Contributor 2
ProductionRevision
ReleaseAnytime
CI SystemQA Team
Test System 1
Test System 2
![Page 15: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/15.jpg)
Assembla spins up test servers
![Page 16: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/16.jpg)
MAXOS Service Team
![Page 17: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/17.jpg)
Test Layering
Monitor your released software: Errors, Usage volume, usage patterns, user feedback
QA System with Human test consultants
Code review: Both a manual test, and a place to ask for test scripts.
Continuous integration: Run automated tests before using human review time
Unit tests in the development environment
Switch new features and architectureStart h
ere to
add layers
Start here to
release a change
More frequent releases can increase quality
![Page 18: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/18.jpg)
9 (sparse) Layers at Edmunds
![Page 19: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/19.jpg)
Feature Switch and Unveil
HiddenProgrammer sees a change locally. Change is tested in the main version but not seen.
Test Story Owner and testers see the change on test systems.
BetaInsiders see it and use it. Story Owner can show it to selected users for feedback or A/B testing.
UNVEIL! The big event. Communicate with all users. Measure reaction.
One code version
No special test builds
No long-running branches
![Page 20: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/20.jpg)
Go Both Ways
Veloc i ty
Qu
alit
y
Increase Quality (more layers, longer beta)
Increase Velocity (less layers, faster unveil)
![Page 21: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/21.jpg)
Role: Developer
• Developers have more power and responsibility.• Developers have more responsibility for testing.• Developers (not QA or PM) decide when to release.
This is a strong finding.• Incentives are correct. Developer might have to
come back from Friday night beers to fix a problem. This provides a motivation to make good decisions and automate testing.
• Features can be released but hidden. Product Managers and Marketers will unveil when they are ready. Unblock!
Programmers approve releases, not QA
![Page 22: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/22.jpg)
Person, Dog, or Bot writes codeComputers will write code
![Page 23: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/23.jpg)
Continuous Agile
![Page 24: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/24.jpg)
Lean process
1. Release more frequently2. Improve
![Page 25: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/25.jpg)
![Page 26: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/26.jpg)
![Page 27: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/27.jpg)
![Page 28: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/28.jpg)
Role: Product Manager/Owner
• Batch -> Continuous
• Requirements -> User Experience
• Strategy -> Measurement– Usage measurements are so important, so
underutilized– Double your productivity
![Page 29: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/29.jpg)
Product Owner -> Story Owner
![Page 30: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/30.jpg)
Program Launch
Releases
Plan Program Test Doc Deploy
SkipAutomate& Blend Lag Automate
Pull
CI & CD
End up with
Separate Release from Launch
Measure
![Page 31: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/31.jpg)
Matrix of ServicesBreaking the scale barrier
![Page 32: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/32.jpg)
The Services MegatrendDesktop Web App Cloud
Services
App
DB
Service
Service
Service
![Page 33: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/33.jpg)
Scale it like Google
• 15,000 developers, 5,000 projects, one current version of the code (2013). They can go from an idea to a release in 48 hours
• Vast Internet system divided into thousands of "services"
• Most programming done by teams of 3-4
• Centralized process with single version of the test system – run 100 million test cases daily
• Before developers release a change, they test it with the most recent version of all the other services. If a test script finds conflicts, it tells developers who to contact to resolve them
![Page 34: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/34.jpg)
Matrix of Services - MAXOS
PrioritizedBacklog
CurrentWork
Each team releases
when ready
Hundredsof releases
per day
Service team Productionservice
Service team Productionservice
Service team Productionservice
Feedback on speed, errors, usage, and requests
Test as one system
Integrationtest env
Integrationtest env
Integrationtest env
![Page 35: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/35.jpg)
Coordinate without big meetingsContinuous Integration between latest dev version of each service
• Continuous integration helps teams coordinate.
• See dependencies between “producers” and “consumers”
• Errors and conflicts show related team contact info
• Meetings and changes negotiated between two teams, not many
PrioritizedBacklog
CurrentWork
Service team
Service team
Service team
Integrationtest env
Integrationtest env
Integrationtest env
Machines can replace layers of management
![Page 36: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/36.jpg)
Teams are largely self-managing
PrioritizedBacklog
CurrentWork
Service team Integrationtest env
Up to 50% of workfrom backlog
At least 50% of work is self-plannedProblems get fixed quickly
Productionservice
Productionservice
Productionservice
Feedback: quality, reliability, speed, user support
Productionservice
ProductionServer
Sense, respond, self manageminimize planning
![Page 37: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/37.jpg)
Hubspot – Great at Mid-scale
• Transformed a monolithic app to 200 services over one year
• 3-person programming teams. Each of 20 teams is responsible for about 10 services
• Dev teams responsible for design, programming, testing, release, monitoring, and responding to production problems. No full-time QA. Shared PM and UX. 4 Ops guys for 2000 servers.
• Lot’s of tooling and dashboards to help teams deploy, manage, and monitor their services
• Feedback from customer support also grouped by team
• Transformed a monolithic app to 200 services over one year
• 3-person programming teams. Each of 20 teams is responsible for about 10 services
• Dev teams responsible for design, programming, testing, release, monitoring, and responding to production problems. No full-time QA. Shared PM and UX. 4 Ops guys for 2000 servers.
• Lot’s of tooling and dashboards to help teams deploy, manage, and monitor their services
• Feedback from customer support also grouped by team
![Page 38: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/38.jpg)
Scaling
PrioritizedBacklog
CurrentWork
Each team releases
when ready
Service team Productionservice
Service team Productionservice
Add capacity fast by building aroundSingle-function programmer/tech leads
Integrationtest env
Integrationtest env
Teams are not permanently multifunctional
![Page 39: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/39.jpg)
Core ITannual budget
Reliability & security mission
Web API LayerMobile SaaS / Cloud
Marketing
Fast ITmonthly budget
Mission to respond to opportunities
![Page 40: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/40.jpg)
Service team Productionservice
Service team Productionservice
Productionservice
Integrationtest env
Integrationtest env
Integrationtest env
Fast IT (continuous)
Core IT (stable service)
![Page 41: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/41.jpg)
Service team Productionservice
Service team Productionservice
Productionservice
Integrationtest env
Integrationtest env
Integrationtest env
Different culture, different company, or no company
United by Testbeds
Culture does not needto be consistent
![Page 42: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/42.jpg)
Scaling Methods
![Page 43: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/43.jpg)
Brooks and Mythical Man Month– Hypothesis: Scaling
problems comes from n^2 communication explosion
– Solution: Cells and hierarchy to contain communication
Internet reality– Scaling problems come
from dependencies– Solution: more sharing,
more communicationMythical Man Month is wrong40 years of awesome insights, but
![Page 44: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/44.jpg)
SAFe (Copyright Dean Leffingwell)
![Page 45: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/45.jpg)
Ways to Scale
Scrum + SAFe• Add more hierarchy• Complex multifunction
teams• Hold big meetings and
teleconferences• Block everyone into
one cadence• Coordinate big releases
Top Tech CompaniesAutomate management,
as well as testing and deployment.
Dev-lead teamsCommunicate peer to
peerUnblock! teams to move
as fast a possibleRelease more frequently
![Page 46: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/46.jpg)
Competing with MAXOSThe secret weapon that Silicon Valley is using to
disrupt and destroy competitors
• Retailer X deploys changes to their monolithic online ordering app once every six weeks. Ops holds for three weeks to make sure the complete system is stable.
• Amazon has thousands of services and more than 1000 service teams. They release something about once every 11.6 seconds. In the time that Retailer X takes to try one new release, Amazon has made 100,000 changes.
• Amazon hosting competitor: “It’s an emergency”.
![Page 47: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/47.jpg)
SAFe and MAXOS
![Page 48: MAXOS - the new Scalable Continuous Agile](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b6e1034a795943588b458e/html5/thumbnails/48.jpg)
Debatable Points• More frequent releases can increase quality• Culture does not need to be consistent• Teams are not permanently multifunctional• Programmers approve releases, not QA• Productivity comes from bigger/more machines• Machines will write code• Sense, respond, self-manage, minimize planning• Mythical Man Month is wrong• Machines and continuous integration can replace layers of
management• Scrum and SAFe cannot compete with MAXOS