ibm innovate - adoption of continuous delivery at scale at a large telco v0 3
TRANSCRIPT
© 2014 IBM Corporation
1586 - Adoption of Continuous Delivery at Scale at a Large Telco
Tony Christensen
General Manager &
CD Program Director, Telstra
Mirco Hering
Senior Manager, Accenture
Agenda
• What you will get from the session
• Enterprise Situation
• Continuous Delivery is and is not
• The organisational context
• Continuous Delivery Program
• CD Organisational and Technical Practices
• Change Management Considerations
• Tooling Architecture and Operational Model
• Lessons Learned
Presentation Overview What will you get out of this session?
• An understanding of what Continuous Delivery is and what it is not in our context
• Understand why Continuous Delivery is relevant to address many of our pain points
• Understand the ultimate goal envisaged by Telstra
• Understand that Continuous Delivery requires fundamental changes to how we plan,
prioritize, breakdown work as well as build, support, test and release software
• How we approached the adoption of Continuous Delivery
• What we learned throughout our adoption journey
3
Animation from Laptop
This will be available once on location in Orlando.
4
• Telstra spends more on testing, deploying and releasing software
than designing and building it
• A high proportion of the production incidents are a result of
human errors in the manual release of software
• The solution delivery process is often slow and arduous
• It is necessary to evolve away from the current manual ways of
working to lean/automated ways of working whilst maintaining
and improving production stability
Enterprise Situation
Telstra’s Continuous Delivery Program
Continuous Delivery Competitive Market Necessity
5
“We must become significantly more
productive and efficient in our core
businesses, for us to be competitive in a
post NBN all IP world. A strategic
necessity! This is not a technical vanity
project."
Telstra CIO - Patrick Eltridge 2014
• Telstra aims to reduce the
time, cost, and risk of
delivering business
desired outcomes from the
time they are conceived to
the time they are realised.
• This “Idea to Value”
process must be
continuous, reliable,
predictable & visible.
• To achieve this Telstra is
progressively
reengineering its ITS
operational model to
implement Continuous
Delivery
6
What CD is at Telstra ....
The ability to deploy large complex, quality assured,
integrated software releases at the click of a button
Every part of the software delivery process is
automated, tested and logged for traceability
Provides visibility of the progress and the quality of
software to ITS Operations through all environments
from development to production and places them in
control of the final release to production
CD provides greater adherence to the Change Control
governance processes
CD is designed to ensure software is always in a ‘clean
and ready state’ to release when the business chooses
to do so
What CD is NOT....
Continuous Deployment
Providing developers access to production systems
A decrease in the quality of software
An increase in risk of production incidents
A way to avoid appropriate governance processes
We also want to delight our customers
with new features, products and services
along with rapid fixes to identified issues
when they do occur.
We care deeply about our customer‘s
experience; and operational stability is
key to maintaining a great experience
and NPS advocacy rating.
valued equally
Continuous Delivery makes this possible
Continuous Delivery is and is not
Agile @ Scale
7
The way the work works How Does Continuous Delivery fit the way we work
8
Strategy
Portfolios
Prioritisation (WSJF)
Telstra Development
Process
Outcomes
Sliced into
Prioritised
Once prioritised
Emerging
Architecture
Business & Technical
Roadmaps inform Strategy
Influences
$
Idea Value
$
Continuous Reliable Visible Predictable
Continuous Delivery
Backlog Backlog groomed
frequently by
priority items
Next most valuable priority
item selected
Approval provided for business cases
through Stage Gate process by OMC, PMC
& IMC, where capabilities are assessed
with their alignment at portfolio level
Benefits
Realisation
Post Implementation Assessing post implementation the
benefits realisation of delivered capability:
• Has the intent remained intact
• Feasibility for cost
• Capability value delivered
• What did we learn
Capabilities
Benefits Realisation Feeds into
Rolling Wave Planning to
inform and evolve ITS Strategy
Work progresses
through prioritisation
based on financial,
backlog and workforce
constraints
9
Continuous Delivery Program Vision
$
Idea Value
$
Continuous Reliable Visible Predictable
Idea to Value Process
One team
Holistic
funding
Smaller Value
Increments
Service Mgmt
System Resilience
Automation
capability
Develop /
Augment Skills
Clear enterprise
roadmaps
Time Cost Risk
Reduce Desired business
outcomes
Win-win
commercial
constructs
Align delivery
to Business
readiness
Effective
measures & ROI
Aim to reduce time, cost
and risk of delivering
business desired
outcomes from idea to
realised value
The “idea to value”
process must be
continuous, reliable,
predictable & visible
For continuous
delivery, we
believe we
must
establish/
evolve the
following
Business
Process
Architecture
Continuous Delivery The Program Journey – Strategy Pivots
10
True North(TN) Program CD Program 1.0 Proposed
CD 2.0 approach
Working within the release
& test organisation on test
automation & release
process
Approach
• Trying to solve the problems from
within release and test
organisation through test
automation and governance for
an alternate path to production.
Scope
Test Automation
Introduction of alternate path to
production
Learnings
• Cost in later part of the SDLC are
caused early on in the lifecycle
• Functional test automation
requires long term ROI
Coordinated delivery of CD
Technical Practices across
3 delivery teams
Approach
• Coordinate the implementation of
technical practices
• Partner with release and test
organisation changing processes
Scope
• Coordination of automation of
package, deployment and unit
testing
• Coordination of optimisation of #
of required integration test cases
Learnings
• Pace of change does not meet
expectations)
• Separated CD team from delivery
is sub-optimal
Partnering with Delivery
teams and creating CD path
to Production
Approach
• Partnering with delivery teams
• Create alternative path to
production though a CD
deployment pipeline
Scope
• Joint accountability of delivery
end-to-end
• Implementation of automation
required to optimise delivery
Phasing of implementation:
1. Simple independent application
within Telstra
2. Simple integrated
application/cluster with one
vendor
3. Complex Integrated Cluster
across vendors)
Shifting
our
focus to
delivery
teams
From
Coordin
ating to
Partneri
ng
11
Strategy Pivots illustrated
Tru
e
No
rth
Test Automation
Fast Lane
CD
1.0
Optimised workforce
Test Planning
CD
1.0
Automated
• Build
• Packaging
• Deployment
Config Mgt
CD 2.0
Enterprise DevOps function
• Governance
• Process
• Tooling
Dedicated Funding
Alternate Path to Production Continuous Reliable Visible Predictable
CD Organisational Practices
Organisational Dependencies Program & Team
13
Mid-Range
Planning
Demand
Management
Commercial
and Funding
Construct
Complete Training Key Mindset: Appetite
for Risk
Complete SAFe
Transition
Develop Common
Understanding and
Buy-in Across Telstra
Key Mindset: Quality
is Everyone’s
Responsibility
Change
Management
Prioritise WSJF
Features
Use SOWs to Support
CD
Use Multi-Sourced
Funding Allocation
Model
Ensure Funding
Supports Train as well
as Teams
Create Visibility of
Demand
Separate Planning &
Deployment Cadence
Develop Rolling Wave
Planning Cadence
Manage Feature
Dependencies
Design Slice to
Independent Valuable
Features
Create Single Feature
Backlog for Train
Communicate
Changes
Adopt Business Case
Model Supporting
Iterative Funding
Incorporate Business
Rollout Strategy in Design
Slicing & Feature
Prioritisation
Lean
Leadership Understand CD / Agile
(Management)
Develop High Trust:
-Management / Teams
-Business / IT
-Telstra / Vendors
Practice Lean
Leadership
Program Level
Teams
Dedicate UX Design
People: Leaner
Approach
Dedicate & Align
Architectural, Release &
Test Management Teams
Establish Persistent
System Teams
Establish Persistent
Operations Teams
Product
Ownership
Access Real
Customers who Pay
Bills
Establish Clear
Product Ownership
Hierarchy
Core Team
Construct
Include Multiple
Vendors per Team
(As Required)
Dedicate Iteration
Manager per Team
Establish Core Set of
Persistent,
Empowered Teams
Establish Persistent
DevOps Teams
Establish Cross-
Functional Agile DBT
Teams
Align People to ‘Long
Lived’ Feature Teams
Align Feature Teams
Around Product Wherever
Possible
Key Mindset: Flexibility
(Fix Quickly) versus
Robustness (Extreme/
Excessive Testing)
Align to Value
Streams
Define Functions &
Capabilities
Define Functions &
Capabilities
Strategically Manage
Persistent versus
Flexible Capacity
Organisational Dependencies Program & Team
14
Continuous
Improvement
Collaboration
Enablement
Reporting
External
Alignment
Normalise Sizing &
Track Train Velocity Forecast Capacity Track Cost & Value
Measure Continuous
Improvement
Extend Delivery
Lifecycle Visibility to
Business Rollout
Allocate Time to
Innovate / Experiment
Ensure Release Train
is a Learning
Organisation
Use Lifecycle Tooling Co-Locate Team Collaborate Remotely
Create Rolling Wave
Feature Roadmap
Collaborate with Non-
CD / Non-Agile Teams
to Enable Progress
Ensure Ability to
Accept Frequent
Deployments
Ensure Service
Management &
Operations Construct
Supports CD
Track Train Level
Metrics
Align to Business
Rollout Strategy
Organisational Dependencies Portfolio
15
Functions and
Capabilities
Business
Demand
Management
Complete Training Key Mindset: Appetite
for Risk
Complete SAFe
Transition
Develop Common
Understanding and
Buy-in Across Telstra
Align to Other Change
Programs, e.g. Design
Thinking, OCC2,
Customer In
Change
Management
Define Governance –
Milestones, Value,
Metrics, Accountability
Decentralise
Decision Making
Establish Portfolio
Roadmap - Business
Define Rolling 12
Month Business Epics
Define Rolling 12
Month Architecture
Epics
Establish Rolling Wave
Portfolio Planning
Cadence with BUs & ITS
Track Flow of Value
Define Functions &
Capabilities: Epic Owner,
Enterprise Architect,
Business Engagement
Communicate
Changes
Lean
Leadership Train Leaders
Practice One Voice
Leadership – “Walk
the Talk”
Foster Culture of
Learning (Continuous
Improvement)
Create Ability to
Distribute Work to
Trains
Create Ability to
Distribute Work to Non-
Trains, e.g. Waterfall
Teams
Strategic
Planning
Use Portfolio Kanban
System
Prioritise
Organisational
Backlog
Architecture
Demand
Management
Establish Mature
Recognition &
Management of
Technical Debt
Establish Enterprise
Roadmap -
Architecture
Educate Business on
Slicing Work
Distribution of
Work
Ensure Architecture
Epics Support Business
& System Changes
Educate Architecture
on Slicing Work
Establish Product
Management
Hierarchy
Manage Organisational
Structure Impacts -
System Team in Release
Train, etc.
Manage Portfolio
Dependencies
Align Release Trains
to Value Streams
Identify Portfolio
Investment Themes
Define Portfolio
Strategy & Vision
Associate Work to
Investment Themes /
Strategies
Investment
Themes Map Portfolio Vision
to Corporate Vision
Develop High Trust:
-Management / Teams
-Business / IT
-Telstra / Vendors
Practice Telstra
Purpose and Values
Follow Guiding
Principles
Establish Culture for
Stopping Work
Key Mindset: Flexibility
(Fix Quickly) versus
Robustness (Extreme/
Excessive Testing)
Identify Value
Streams
Organisational Dependencies Portfolio
16
Agree on Lifecycle
Tooling / System to
Record Portfolio Plan
Establishment
Activities Use Smart Sourcing
Strategy Manage Property &
Logistics
Identify Current
Blueprints and
Associate to
Investment Themes
Funding
Construct Use Persistent
Funding Model
Create Lightweight
Business Cases
17
CD Technical Practices - How do we represent Technical Maturity in our program
18
How do we represent Technical Maturity in our program
Continuous Delivery is
at the very far right!
CD Technical Capabilities – Dependency Tree
Feature Toggling
Configuration
Management
Unit Testing
Build Practices
Deployment
Practices
Functional Testing
Virtualisation
Software Quality
Package Promotion
Reporting &
Visibility
•CD team have defined a
set of technical
dependencies
(capabilities) required on
the path to achieving
Continuous Delivery
•Foundation capabilities
are at the left. These are
the minimum required.
•The most mature
capabilities are at the
right.
•Yellow are enterprise
dependencies
• Each box comes with
definition and metrics
Continuous Delivery is
at the very far right!
CD Technical Capabilities – Dependency Tree
Feature Toggling
Configuration
Management
Unit Testing
Build Practices
Deployment
Practices
Functional Testing
Virtualisation
Software Quality
Package Promotion
Reporting &
Visibility
•CD team have defined a
set of technical
dependencies
(capabilities) required on
the path to achieving
Continuous Delivery
•Foundation capabilities
are at the left. These are
the minimum required.
•The most mature
capabilities are at the
right.
•Yellow are enterprise
dependencies
• Each box comes with
definition and metrics
21
Change Management
Depending on the context (maturity, capability,…) each Application team
will work differently with the Enterprise Tooling backbone
Our adoption approach with the central tool backbone
22
CD pipeline
operations
CD pipeline
engineering
Developer
Support
Env and Tooling
Management
Process and
Governance
Serv
ice L
evel
Complete E2E
Service
Deployment from
Asset Repository
Interface with
EST backbone
Certification of
compliance
Support for
building
decentralised
capability
Interface with
EST backbone
Certification of
compliance
Telstra SOE
toolbox
Enterprise DevOps – Scope Architecture
23
SCM
AR
De
live
ry
Pkg
Pkg
Dev
AT SIT Prod
Initial Scope Boundary
Pkg Pkg
VSR
De
live
ry
VS
VS
VS
VS
VS
Enterprise
DevOps
SDCs &
Delivery
Partners
P P
P P
24
Release Pipeline
Release Management
Deliv
ery
Part
ner
1
Deliv
ery
Part
ner
2
Stubbed Integration
SIT/FIT/Other
Integrated Env
Provision Environment
Run Tests
Deploy Package
Completely
Integrated
Dev Int Env
Dev Int Env
CAT
Provision Environment
Run Tests
Deploy Package
Completely
Integrated
FDR
Provision Environment
Run Tests
Deploy Package
Completely
Integrated
Prod
Provision Environment
Run Tests
Deploy Package
Completely
Integrated
Enterprise DevOps – End-State Pipeline
25
9. Lessons Learned
Thank You!
Your Feedback is Important!
Access the Innovate agenda tool to complete your
session surveys from your smartphone, laptop or
conference kiosk.