reduce release cycle time: nine months to a week - nice!
DESCRIPTION
Picture this scene from three years ago: Employing the corporately mandated processes, a software engineering team is delivering system updates about once every nine months. When their senior user suddenly demands the next delivery in twenty-two weeks-half the current cycle duration-the team realize that they must quickly change development practices. Mathew Bissett describes how Her Majesty's Government did precisely that-and much, much more. First, they reduced delivery cycles from unpredictable dates every nine months to predictable releases every six weeks. Then, they cut releases cycle time to once every week. By identifying and mitigating risks early in the work intake process, enforcing quality gates, executing multiple test levels concurrently-and more-they dramatically increased throughput with the same or better quality. Today, these new processes provide their teams the best balance of structure versus agility. Join Mathew to see if what works for Her Majesty's Government might just work for you and your company.TRANSCRIPT
BW5 Concurrent Session 11/7/2012 2:15 PM
"Reduce Release Cycle Time: Nine Months to a Week Is Jolly Good!"
Presented by:
Mathew Bissett UK Government
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Mathew Bissett UK Government
Mathew Bissett has been working for Her Majesty’s Government for more than six years, having been recruited straight from University. Deployed into the integration and test team of his department’s main engineering group, Mathew demonstrated strong leadership qualities to quickly progress from test analyst through test team lead to test management. Currently the test manager responsible for the integration and testing of his area’s flagship system, he has driven through delivery process improvements to enable weekly deliveries. Mathew has delivered many talks within his government department and has presented at a British Computer Society Special Interest Group in Software Testing conference.
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 1
1
Reduce Release Cycle TimeFrom Nine Months to a Week – Nice!
M h Bi
03/09/2012BSC East 2012 v0.13
Mathew BissettTest ManagerHer Majesty’s [email protected]
Introduction
03/09/2012BSC East 2012 v0.13
2
• I joined Her Majesty’s Government in late 2006 from University
• I worked my way up to Test Management within our Integration & Test Team
b ( )
© UK Crown Copyright 2012
• Have been (E2E) Test Manager for my Department’s flagship product since early 2011
• I also run a bit
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 2
Show of Hands
03/09/2012BSC East 2012 v0.13
3
• Would you call yourself a:▫ Developer▫ Tester▫ Project Manager▫ Business Analyst
h
© UK Crown Copyright 2012
▫ Other
BSC East 2012 v0.13
4
03/09/2012
Context SettingDelivery CyclesConclusions & Future
© UK Crown Copyright 2012
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 3
Context Setting
BSC East 2012 v0.13
5
03/09/2012
© UK Crown Copyright 2012
Processed Data Flow
Context Setting
BSC East 2012 v0.13
6
03/09/2012
CCom
mand &
Control
© UK Crown Copyright 2012
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 4
Context Setting
BSC East 2012 v0.13
7
03/09/2012
© UK Crown Copyright 2012
System Under Test
Context Setting
BSC East 2012 v0.13
8
03/09/2012
• “[Testing’s] role changes from verifying to prevention by concurrent engineering with both acceptance- and unit- test-driven development –and that blurs the distinction between test, requirements analysis, and design, so th t t ti i l t ti ”
© UK Crown Copyright 2012
that…testing is no longer testing.”[LV10]
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 5
BSC East 2012 v0.13
9
03/09/2012
Context SettingDelivery CyclesConclusions & Future
© UK Crown Copyright 2012
BSC East 2012 v0.13
10
03/09/2012
9 Months
© UK Crown Copyright 2012
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 6
9 Months - Process
BSC East 2012 v0.13
11
03/09/2012
• Based on RUP & CMMI▫ Heavyweight documentation up front
• Implemented more like Waterfall▫ Defined by serial workflows
• User Acceptance left until the end
© UK Crown Copyright 2012
▫ Once product was nearing completion
9 Months - Process
BSC East 2012 v0.13
12
D eve lopm en t In teg ra tion I& T R e fe rence O perations
03/09/2012
O peva l
Q ua lity R eview(C he cklis t)
Ins ta ll R e lease
B uild V erifica tio nT est
D e ve lop cod eA p p ly (D E V ) lab e l
T est
B u ild ap p lica tio n
T est B u ild
Q uality R eview(C he cklis t)
Ins ta ll R e lease
B uild V erifica tio nT est
S ystem Te st
A ccep tance T est
1
1
1
2
2
Delive
ry Proc
ess
Delive
ry Proc
ess
Delive
ry Proc
ess
T est2
© UK Crown Copyright 2012
K e yD ev/I& T P ro cess
TT O P ro cess
C reate R e le aseA pp ly (IN T ) lab e l
Y e sN o N o
1
D efau lt P ath
Fau lt P ath
E labo ra tion /C o nstructio nTrans is tio n
Ite ra tive ly
T estin g C om ple te T estin g C om ple te
Q ua lity R evie w(C h e cklis t)
3
C rea te R e lease3
F it fo rR e lease
F it fo rR e lease Y e s
C reate R e le aseA p p ly (R E L ) la b el
2
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 7
9 Months – Process
03/09/2012BSC East 2012 v0.13
13
Planning
© UK Crown Copyright 2012
TestingDeveloping
9 Months - Pros
BSC East 2012 v0.13
14
03/09/2012
• Time-boxed process▫ Iterations were a fixed duration
• Code freezes▫ After which all functionality should have been
delivered
© UK Crown Copyright 2012
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 8
9 Months - Cons
BSC East 2012 v0.13
15
03/09/2012
• Feature-bound delivery▫ More iterations were added until the product was
fit for delivery - resulting in endless Transition phases
• Subsystems were produced independently▫ It was not possible to test major integration risks
© UK Crown Copyright 2012
p j guntil late in the delivery cycle
▫ Manual regression testing meant that integrating and re-testing fixes was very costly
• Users not consulted until the product neared completion
BSC East 2012 v0.13
16
03/09/2012
6 Weeks – The Heartbeat
© UK Crown Copyright 2012
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 9
9 Months – 6 Weeks
BSC East 2012 v0.13
17
03/09/2012
• Intermediary ‘Stepping Stones’ process▫ Heavy Systems Engineering involvement▫ Defined integration points
• Challenge lead by Senior User▫ Changed risk appetite in exchange for a delivery
i hi k
© UK Crown Copyright 2012
within 22 weeks▫ Successfully met challenge after 20 weeks but
wasted final 2 weeks
6 Weeks - Process
BSC East 2012 v0.13
18
03/09/2012
• Backlog driven & ‘light-weight RUP’ based▫ Customer lead prioritisation▫ Planned for Product rather than Subsystems
• Delivery scope was advertised up front on a Wiki▫ Based on customer priority
© UK Crown Copyright 2012
• Introduced concurrent test phases▫ Start of automated regression testing▫ Continuous Subsystem Integration▫ User Trials undertaken in parallel to development
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 10
6 Weeks - Process
BSC East 2012 v0.13
19
03/09/2012
© UK Crown Copyright 2012
6 Weeks – Process
03/09/2012BSC East 2012 v0.13
20
Planning
© UK Crown Copyright 2012
TestingDeveloping
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 11
6 Weeks - Pros
BSC East 2012 v0.13
21
03/09/2012
• Time-boxed process▫ Delivery cycles were fixed to 6 week
• Integration risks assessed faster▫ Invested in Subsystem and Product automated
regression testingi l h l d
© UK Crown Copyright 2012
• User Trials helped User Acceptance▫ Prior-proven business benefit
6 Weeks - Cons
BSC East 2012 v0.13
22
03/09/2012
• Feature-bound delivery▫ Great pressure to deliver desired functionality
• Requirements/Designs were often not produced▫ Lack of upfront Systems Engineering effort
• Subsystems still produced independently
© UK Crown Copyright 2012
▫ Usually only managed one or two integration points
▫ Often made to work by I&T in the final week
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 12
BSC East 2012 v0.13
23
03/09/2012
1 Week – The Pulse
© UK Crown Copyright 2012
6 Weeks – 1 Week
BSC East 2012 v0.13
24
03/09/2012
• Challenge lead by Integration & Test Team▫ Needed to assess integration risks earlier▫ Enabled by previous investment in automation▫ Intention was to remove the pressure from the
end of the delivery cycle
© UK Crown Copyright 2012
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 13
1 Week - Process
BSC East 2012 v0.13
25
03/09/2012
• Customer Backlog and Integration Logic driven▫ Technical lead prioritisation
• Delivery scope was more flexible▫ Reduced the planning horizon to ~3 weeks
• Removed team independence by introducing CI
© UK Crown Copyright 2012
▫ Subsystem integration responsibility moved into Development Teams
▫ Enforced by Quality Gates
1 Week - Process
BSC East 2012 v0.13
26
03/09/2012
Weekly Regression
Weekly E2E & UATDaily CIE Regression
TestReportsCDD’s
Build Logic
© UK Crown Copyright 2012
Weekly DQ Regression
Builds& Docs
Builds
VDDs
TestReports
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 14
1 Week - Process
BSC East 2012 v0.13
27
03/09/2012
WednesdayWednesdayWednesdayWednesday ThursdayThursday FridayFriday Sat/SunSat/Sun MondayMonday TuesdayTuesday ThursdayThursday FridayFriday Sat/SunSat/Sun MondayMonday TuesdayTuesday
E2EE2E
CIECIE
TRR TRR TRR TRR TRR TRR TRR TRR TRR TRR
CIE CIE CIE CIE CIE CIE CIE CIE CIE
E2E Agility
E2E Regression
TRR
E2E Agility
E2E Regression
TRR
CIE
© UK Crown Copyright 2012
Transition Transition to to DeploymentDeployment
DQ Regression DQ Regression
Build
TRR
VDDs
Test Reports
Build
VDD
Test Reports
TRR
GOPRM PRM GO
1 Week - Process
BSC East 2012 v0.13
28
WedWedWedWed ThuThu FriFri Sat/SunSat/Sun MonMon TueTue ThuThu FriFri Sat/SunSat/Sun MonMon TueTueMonMon TueTue WedWed ThuThu
03/09/2012
PlanningPlanning
E2EE2E
CIECIE
22 22 22 22
CIECIE CIECIE CIECIE
E2E AgilityE2E Agility
33
11
44
CIECIE CIECIE
33
Test RptTest Rpt22 22
- Pulse Planning Meeting (PPM)- Pulse Planning Meeting (PPM)11
C
D
A
B B
C
G
HE E
© UK Crown Copyright 2012
Supplier #1Supplier #1
Supplier #2Supplier #2
E2E RegressionE2E Regression
DQ RegressionDQ Regression
- Pulse Planning Meeting (PPM)- Daily Test Readiness Review (TRR) (1200hrs)- Product Handover Meeting (1230hrs)- Pulse Go/No Go meeting (1630hrs)
- Create RTC Tickets- Update RTC Ticket- Build and Test- Test results analysis and report to TRR- Possible new builds following TRR- Continuous analysis of test results and report defects- Produce Test Report (1400hrs)- Review Problem Tickets (1500hrs)
- Pulse Planning Meeting (PPM)- Daily Test Readiness Review (TRR) (1200hrs)- Product Handover Meeting (1230hrs)- Pulse Go/No Go meeting (1630hrs)
- Create RTC Tickets- Update RTC Ticket- Build and Test- Test results analysis and report to TRR- Possible new builds following TRR- Continuous analysis of test results and report defects- Produce Test Report (1400hrs)- Review Problem Tickets (1500hrs)
11
22
33
44
ABCDEFG
F
F
H
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 15
1 Week – Process
03/09/2012BSC East 2012 v0.13
29
Planning
© UK Crown Copyright 2012
TestingDeveloping
1 Week - Pros
BSC East 2012 v0.13
30
03/09/2012
• Non-feature bound time-boxed process▫ Delivery cycles were fixed to 1 week and delivered
only what was ready▫ Regular release candidates based on Product
integration logic• Risk identified up front
© UK Crown Copyright 2012
p▫ Testing defined in the take-on-work process▫ Enforced by quality gates between delivery phases
• User Acceptance became an input into delivery quality gate
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 16
1 Week – Cons
BSC East 2012 v0.13
31
03/09/2012
• Software is constantly changing▫ Manual continuous integration is expensive▫ Manual configuration management is hard across
many integration environments▫ Delivery process required improvement to meet 1
k d li i
© UK Crown Copyright 2012
week deliveries
BSC East 2012 v0.13
32
03/09/2012
Context SettingDelivery CyclesConclusions & Future
© UK Crown Copyright 2012
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 17
Conclusions
BSC East 2012 v0.13
33
03/09/2012
• Identify Risks Early• Concurrency• Planning Horizon
© UK Crown Copyright 2012
Identify Risks Early
BSC East 2012 v0.13
34
03/09/2012
• Introduce risk identification into your take on work process
• Have many risk assessment opportunities and assess as soon as practical▫ Share mitigation of risk between teams
© UK Crown Copyright 2012
• Introduce Quality Gates and enforce them
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 18
Identify Risks Early
BSC East 2012 v0.13
35
03/09/2012
Phase Who Environment Exit CriteriaTest Tool Development Automation Team Tool Development Env. Test tools available
Development Testing Development Team Development Env. x% code coverage
System Testing Test Team System Test Env. x% feature coverage
Feature Trials Project Team Trial Environment Initial User validationInitial E2E verification
Quality Gate #1
© UK Crown Copyright 2012
System Integration Testing
Test Team System IntegrationEnvironment
Verify integration to delivery baseline
Quality Gate #2
End-to-End Testing Test Team End-to-End Test Environment
End-to-End data flow verification successful
User Acceptance Testing
Test Team End-to-End Test Environment
Final User validation successful
Concurrency
BSC East 2012 v0.13
36
03/09/2012
• Introduce concurrency where practical▫ Which test phases can be performed in parallel?▫ Use different test environments for different
purposes• Give yourself more opportunities to succeed
l l did
© UK Crown Copyright 2012
▫ Regular release candidates• Enforce process when pulling through bugfixes
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 19
Concurrency
BSC East 2012 v0.13
37
03/09/2012
18.5
101214161820
Time to Market
Weekly Delivery
© UK Crown Copyright 2012
1
6
2.5
0
7
2
02468
P12 P13 P14 P15 P16 P17 HB21+
y y6-Weekly Delivery
Concurrency
BSC East 2012 v0.13
38
03/09/2012
11.5
8 5
13.5
9.5
12.5 12.511
23.5
14.5
17.5 17.5
8
2018.5
10
15
20
25Delivery Capacity
© UK Crown Copyright 2012
5
8.5 8
0
5
10
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 20
Planning Horizon
BSC East 2012 v0.13
39
03/09/2012
• Only plan as far as practical▫ Let the technical community define the best path
to meet your customers’ requirements▫ Reduce your planning horizon to save re-work
• Advertise your planning horizon to your
© UK Crown Copyright 2012
customers▫ Negotiate flexibility with your customers
• Be flexible to unexpected change▫ Embrace the opportunity to deliver fixes quickly
BSC East 2012 v0.13
40
03/09/2012
© UK Crown Copyright 2012Known Predicted Make Believe
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 21
Future
BSC East 2012 v0.13
41
03/09/2012
• Automated CI & Build tool• Replace E2E Environment with OpEval
Environment• Empower Users to drive & test system and
enable direct feedback loops
© UK Crown Copyright 2012
Future
BSC East 2012 v0.13
42
03/09/2012
Volumetric
RegressionFuRe
Op Evaluation
OpRelease
Build & Test
Queue
Dev Component TestDev Component
Test Package Sys TestComponent Int. Test
Dev Component TestDev Component
Test Package Sys TestComponent Int. Test
Dev Component Test Package Sys Test Static
ReviewComponent Int. Test
© UK Crown Copyright 2012
Dev Component Test
Static Review
unctional egression
Supplier Capability Team
Platform Team CI Tools Delivery
Team
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 22
BSC East 2012 v0.13
43
03/09/2012
Context SettingDelivery CyclesConclusions & Future
© UK Crown Copyright 2012
References
BSC East 2012 v0.13
44
03/09/2012
• Bibliography▫ Ambler10a
Ambler, S., 2010. Agile Testing and Quality Strategies: Discipline Over Rhetoric
▫ Ambler10bAmbler S 2010 Introduction to Disciplined Agile
© UK Crown Copyright 2012
Ambler, S., 2010. Introduction to Disciplined Agile Delivery
▫ CG09Crispin, L., Gregory, J., 2009. Agile Testing: A Practical Guide for Testers and Agile Teams
Reduce Release Cycle Time From Nine Months to a Week ‐ Nice!
Better Software Conference East 2012 v0.13 23
References
BSC East 2012 v0.13
45
03/09/2012
• Bibliography▫ Leffingwell10
Leffingwell, D., 2010. Scaling Software Agility: Rearchitecting Enterprise Class systems – An Agile Enterprise Trifecta
▫ LV10
© UK Crown Copyright 2012
▫ LV10Larman, C., Vodde, B., 2010. Practices for Scaling Lean & Agile Development: Large, Multisite, and Offshore Product Development with Large-Scale Scrum
BSC East 2012 v0.13
46
03/09/2012
© UK Crown Copyright 2012
Mathew BissettTest Manager
Her Majesty’s [email protected]