High Speed Software and Business Innovation

From Andy Singleton, http://continuousagile.com

www.assembla.com

Survey on Continuous Delivery

46% think their competitors

have adopted Continuous

Delivery

Why?

• Fixes specific problems

– Stressful or late releases

– You are building a big system with many parts

– You are running a lean product development or marketing process with frequent measurements

– Competitors are gaining with a faster cycle

• Opportunity to conquer the competition through faster delivery and fit

3 Ways to Be More Productive

1)Practice2)Do the right thing3)Use more machines

1) Practice

We can get more efficient at anything with practice and optimization

This effect explains almost all of the productivity difference between Waterfall and Agile. In the agile process, we do more cycles, so we get better at each step.

Any periodic release cycle causes a lot of stress around releases. Continuous delivery removes this stress by doing releases more frequently and optimizing down to a button push.

2) Do the right thing

Users ignore at least 50% of the new stuff you show them. If you can measure usage or value and figure out what to ignore in development, you can increase development productivity by 100% for zero extra cost.

Measurement is very important.

More frequent releases = more measurements

3) Use more machines

Continuous Delivery Basics

• Lean process improvement

• Code contribution patterns

• Continuous Integration

• Test layering

• Code review

• Automation

• Developer responsibility

• Feature switches

Lean process

1. Release more frequently

2. Improve

This can be easier than a batch process. Do less

Centralized Continuous IntegrationContributor Commits – “as early as possible” to find problems

ContinuousIntegration tests

Fail - alarm

Release CandidateTest System

Release

QA Testing

Pass

Test Layering

Monitor your released software: Errors, Usage volume, usage patterns, user feedback

QA System with Human test consultants

Code review: Both a manual test, and a place to ask for test scripts.

Continuous integration: Run automated tests before using human review time

Unit tests in the development environment

Switch new features and architecture

More frequent releases can increase quality

Go Both Ways

Veloc i ty

Qu

ali

ty

Increase Quality (more layers, longer beta)

Increase Velocity (less layers, faster unveil)

9 (sparse) Layers at Edmunds

Feature Switch and Unveil

HiddenProgrammer sees a change locally. Change is tested in the main version but not seen.

TestStory Owner and testers see the change on test systems.

BetaInsiders see it and use it. Story Owner can show it to selected users for feedback or A/B testing.

UNVEIL!The big event. Communicate with all users. Measure reaction.

On

e co

de

versio

nN

o sp

ecial te

st bu

ilds

No

lon

g-run

nin

g bran

che

s

Role: Product Manager/Owner

• Batch -> Continuous

• Requirements -> User Experience

• Strategy -> Measurement

– Usage measurements are so important, so underutilized

– Double your productivity

Batch Agile Continuous

Explains the resurgence of vertical integration(eg Apple, Tesla). Requires CI

Making one good change is not a team sport

Product Owner -> Story Owner

Matrix of Services(MAXOS)

Breaking the scale barrier

The Services Megatrend

Desktop Web App Cloud Services

App

DB

Service

Service

Service

Scale it like Google

• 15,000 developers, 5,000 projects, one current version of the code (2013). They can go from an idea to a release in 48 hours

• Vast Internet system divided into thousands of "services"

• Most programming done by teams of 3-4

• Centralized process with single version of the test system –run 100 million test cases daily

• Before developers release a change, they test it with the most recent version of all the other services. If a test script finds conflicts, it tells developers who to contact to resolve them

Matrix of Services - MAXOS

PrioritizedBacklog

CurrentWork

Each team releases

when ready

Hundredsof releases

per day

Service teamProduction

service

Service teamProduction

service

Service teamProduction

service

Feedback on speed, errors, usage, and requests

Test as one system

Integrationtest env

Integrationtest env

Integrationtest env

Coordinate without big meetings

Continuous Integration between latest dev version of each service

• Continuous integration helps teams coordinate.

• See dependencies between “producers” and “consumers”

• Errors and conflicts show related team contact info

• Meetings and changes negotiated between two teams, not many

PrioritizedBacklog

CurrentWork

Service team

Service team

Service team

Integrationtest env

Integrationtest env

Integrationtest env

Machines can replace layers of management

Teams are largely self-managing

PrioritizedBacklog

CurrentWork

Service teamIntegration

test env

Up to 50% of workfrom backlog

At least 50% of work is self-planned

Problems get fixed quickly

Productionservice

Productionservice

Productionservice

Feedback: quality, reliability, speed, user support

Productionservice

ProductionServer

Sense, respond, self manageminimize planning

Continuous Buzzwords

• Lean- Continually improve concrete measures• Continuous Delivery (CD) – build, test, and

release each code change• Microservices – Small programs that talk to each

other. Combined with REST communication, a new and more adaptable incarnation of SOA

• DevOps – developers test, deploy and monitor services

• Cloud – huge on-demand datacenters• MAXOS – all of this, plus some scaling and

management tricks

“50% of all large IT projects fail”– Capers Jones

“9 women can’t make a baby in one month”– Fred Brooks

Mythical Man Month is now wrong40 years of awesome insights, but

SAFe (Copyright Dean Leffingwell)

Ways to Scale

Scrum + SAFe• Add more hierarchy• Complex

multifunction teams• Hold big meetings and

teleconferences• Block everyone into

one cadence• Coordinate big

releases

Top Tech Companies Automate management,

as well as testing and deployment.

Dev-lead teams

Talk peer to peer

Unblock! teams to move as fast a possible

Release more frequently

Competing with MAXOSThe secret weapon that Silicon Valley is using to

disrupt and destroy competitors

• Retailer X deploys changes to their monolithic online ordering app once every six weeks. Ops holds for three weeks to make sure the complete system is stable.

• Amazon has thousands of services and more than 1000 service teams. They release something about once every 11.6 seconds. In the time that Retailer X takes to try one new release, Amazon has made 100,000 changes.

• Amazon hosting competitor: “It’s an emergency”.

SAFe versus MAXOS

Core ITannual budget

Reliability & security mission

API Layer

Marketing

Fast ITmonthly budget

Mission to respond to opportunities

Service teamProduction

service

Service teamProduction

service

Productionservice

Integrationtest env

Integrationtest env

Integrationtest env

Fast IT (continuous)

Core IT (stable service)

Software eats your company in 3 steps

1. Product2. Cloud3. Ecosystem

1) Products and services contain more software every year.

Continuous delivery accelerates this trend. Software is not only Faster Cheaper Better (FCB) in many current uses, but also improving faster than either hardware, or humans

2) Cloud

Link products and data together into a cloud of services.

Happening now at Apple, Cisco, GE, GM and many other product silo companies

Integrationtest env

Integrationtest env

Integrationtest env

Services

Logistics

Life Cycle

Customer Experience

Marketing

3) Ecosystem

Nobody is safe. You are linked in an ecosystem

• Banking

• Health Care

• Software

• Government

• Internet of Things

Company AProduction

service

Company BProduction

service

Productionservice

Integrationtest env

Integrationtest env

Integrationtest env

United by Testbeds

Steps to the Future

Start with cycles or batches

1. Continuous delivery inside one team

2. Lean, metric-driven product management

3. MAXOS large scale continuous delivery

4. Coordinate ecosystems with Industry Testbeds