How toSuspend Testingand Still Succeed

– A True Story

Graham ThomasIndependent Software

Testing Consultant

1

ABSTRACTThis is the story of a testing programme that was destined to fail, but ultimately succeeded. We will look at what went wrong, explain why testing had to be suspended, and discuss

how with no real hope of recovery the team managed to set and meet their resumption requirements, and ultimately complete their testing on time.

We will discuss where things started to go wrong, how this was identified, and what

measures were taken to ensure a successful resolution. We will also look at the detail of the challenges that the testing team, and the program,

were daily presented with when testing was suspended. And tell how innovation, ingenuity and perseverance, against all the odds, won the day.

This is a real ‘war story’, from the testing front line, with valuable hard won experience, and is told in the very real hope that it will benefit all who hear it.

2

AGENDA

Background & Set the Scene

Testing Strategy

Initial Progress

Suspension

Resumption

Further Demands

Conclusions

3

BACKGROUND The Program was scoped as delivery of the infrastructure

and processing of the new Faster Payments solution for one of the member banks

Faster Payments is a new (mid 2008) component of the UK’s Financial Services Infrastructureo First new payments system in UK for 20 yearso Centrally Regulatedo High Volume and High Availabilityo Near Real Time financial transactions between member banks for

phone, internet and standing order transactions.o Daily (5/7) settlement

4

BACKGROUND 2 Arrived as Program Test Manager just as;

o Phase 1 development was deliveringo The first phase of external Scheme testing commenced (Self Certification)o Development plans for subsequent releases were firming upo The Program Testing Strategy was issued for review

Role was to pull together existing testing threads of:o Unit Testing (offshore, external suppliers, and internal)o Systems Testingo Systems Integration Testingo External Scheme Testingo OATo UATo Business Procedure Testing

Providing oversight, control, and stakeholder management for testing

5

TESTING STRATEGY First task was to review the Draft Testing Strategy

Met with current development and testing approach Aligned with outsourcing partner Strengthened testing organisation with representation at Program level Weak in Systems Integration Weak test measurement Weak in presentation

• Document too long 50+pp• and not audience friendly, too much text, few diagrams

Outcome of review:o Testing Strategy ‘fit for purpose’o Systems Integration Risk accepted at Program level

• Mitigation too expensive and not viable within timeframe

6

INITIAL PROGRESS Systems Integration completed successfully (as scoped)

Initially testing hampered by slippage in development delivery;o Single phase delivery into testing became multiple drops of codeo Testing planned on a Just in Time basis

OAT planning delayed and execution moved back

Slow progress in UAT execution due to issues with the test automation infrastructure supplied by the solution vendor

Non-functional proving of infrastructure took 250% longer than planned and had a qualified exit (i.e. not complete)

7

SUSPENSION Take a view from the room?

o How many people have actually experienced testing being suspended?

o What if any where the suspension criteria?

o What was the impact?

o Who got the blame?

8

WHAT THE STANDARD SAYS IEEE829

o 9.2.7 (LTP Section 2.7) Suspension criteria and resumption requirementsSpecify the criteria to be used to suspend all or a portion of the testing activity on the test items associated with this plan. Specify the testing activities that must be repeated when testing is resumed.

Two Website References:o Know when to pause in a series of tests. o If the number or type of defects reaches a point where the follow on testing has no

value, it makes no sense to continue the test; you are just wasting resources. o Specify what constitutes stoppage for a test or series of tests and what is the acceptable

level of defects that will allow the testing to proceed past the defects.o Testing after a truly fatal error will generate conditions that may be identified as defects

but are in fact ghost errors caused by the earlier defects that were ignored.

A Document Reference:o Suspension of test execution will occur if the number or severity of incidents raised is so

great as to make further execution valuelesso Specify the testing activities that must be repeated, when testing is resumed

9

WHY WE STOPPED TESTING We had realised that we did not have enough test resource to

complete the task so had brought in external resource to augment UAT

At current rate of progress in UAT 8 weeks into a 12 week schedule it was going to take over a year to complete!

Many of issues (defects) were to do with automation infrastructure and product configurationo The Systems Integration Risk had matured!

Unless we did something radical we were going to have to declare to the Scheme that we were failingo This would mean losing our implementation slot, and cause significant

internal cost overruns and knock-on impacts on other programs of work

10

SO NOW WHAT? We had to swiftly identify what was wrong

o A quick series of workshops with all stakeholders• Developers, Testers, Suppliers, Customers, and Business

Once we had an idea of what was wrong we had to speedily plan a resolutiono A targeted Systems Integration Test, carried out in the UAT environments

• Our Systems Integration risk had maturedo Product configuration

• We had a default supplier provided configuration which needed to be revised to reflect the current software and business model

o Test infrastructure optimisation• Multi threaded and concurrent testing configuration

We had to execute the resolutiono Allocate highest non-production priorityo Extended working hours, Provision of support, reservation of key resource to supporto Close knit group working between all teams

And the program still had to deliver on timeo We had stopped, were going to lose time resolving our problems, but still had to deliver on the same

date!

11

RESUMPTION We also had to determine what would allow us to resume testing

o New UAT manager wanted all UAT tests run successfully before UAT could commence!

o Requested a set of viable Resumption Requirements that met the reasons why we suspended testing• System Integrated• System Configured• Test Automation met desired throughput rates• That we developed a set of re-usable System Integration Tests that could be

re-run• No Short-cuts, position or views – We were not going to kid ourselves

I tried in vain to get the organisation to use the term Resumption Requirements, but everyone kept on sayingResumption Criteria!

12

RE-PLANNING

It wouldn’t be a software development without re-planning!

We re-planned UAT Executiono Combined Future development releases

• They weren’t separate business releases• Saved re-testing time lost with resumption activities

Utilised contingency Accepted a scheme request to move the implementation

date back gaining 3 weeks Moved Business Procedure Testing out of UAT Implemented predictive test measures

13

RESTARTING TESTING Amazingly the 4 week suspension was completed on time

o We fixed outstanding integration issueso We configured the producto We resolved the problems with test automationo And created a repeatable set of System Integration tests

o And in 5 weeks UAT had re-planned from a 3 phase to a 1 phase test cycle.

It took a week to deploy the changeso Don’t forget there may be a lag in resuming execution

And within 3 days of test execution we had already completed 60% of the revised UAT scheduleo I don’t want to make it sound that easy, because some tests required multiple

cycles to be run, however, suddenly the end was in sight and achievable

14

FURTHER DEMANDS So all is looking good, testing has just finished a 4 week

suspension, and current throughput rates show that UAT will complete on time.

However this is not an easy programo It is the largest, costliest and most complex development that the

organisation has undertaken for nearly a decade, and it is in an area of new technologies that the organisation is not familiar with

More testing is identified for UATo We have 6 times as much to do as they thought!o OAT planning completed and shows that 9 months are required not the 3

that the program haso And External Scheme testing is about to commence, and the timescales

have been cut by over 50%!

15

ORGANISE FOR SUCCESS

16

We had been playing football like 11 year olds o We had all been chasing after the ball – Testing Suspension

We instituted a daily War Room involvingo Decision Makerso The doerso Program Office to record and account

We met daily at 13:00, or more frequently if required to manage issues

We made decisions and gave direction

THE WAR ROOM 1 physical dedicated Meeting Room

o Equipped with projectoro Reports printed out and placed on the

wallso Measures and metrics for all to seeo Dedicated conference number

Strict rules as to attendance and protocol

Handled all program issues, not just testing

Worked as a collective, with individual responsibility

17

THE END GAME We used all of the tools at our disposal

o Took a firm view on prioritisation of defectso Took a clear view of scope for current releaseo Compromised for success

• To fail to deliver would cost the organisation £1m per month, excluding reputational and other damage

o Exercised tight control on changeo Engaged with Business, IT and Change throughouto Used all contingency even within deployment windowo Limited go-live exposure with low initial volumeso Provided full implementation hand-holding

18

POSTSCRIPTWhat did Success look like?

The program delivered on-time There were zero defects in the first 3 months The development group gained the confidence to know that it

could deliver the remaining phases of the program

The bad practice lessons were not repeated for future phases

As a result of the suspension, senior management questioned the original Testing Strategy,o The Testing Strategy was retrospectively re-writteno And signed off shortly after the system went live!

19

SUMMARY Suspension and Resumption is more complicated than the standard leads us

to believe

You will probably not be able to precisely specify suspension criteria prior to the suspension of testing. You will only know these at or around the time you suspend testing. (The standard is not too helpful here!)

Resumption Requirements will be directly dependent upon the reasons for suspension and will have to be written and agreed very shortly after suspension. (You are not just repeating previous testing as per the standard.)

Suspending testing does not necessarily buy you more time to test, and in fact may actually give you less

Suspending testing is a very powerful tool. Don’t underestimate the amount of management attention you will receive, but do be aware of the benefits that fully met resumption requirements can bring

20

CONTACT DETAILS

Graham Thomas

Independent SoftwareTesting Consultant

graham@badgerscroft.com +44 7973 387 853 www.badgerscroft.com

21