Test Management Challenges

Topics

• Drivers• Value Contribution • Focus Areas

– Information provision– Estimation– Supplier management– End to end view of testing– Offshore– Automation– Environments– Non-functional testing– Staff assessment and selection

Drivers

• Business view of information technology’s contribution– Frustration– Lack of confidence– Not in line with business objectives

• Greater intolerance of inappropriate levels of quality– Can be too high quality as well as too poor, depends on

business goals– Delivering the wrong system on time is also poor quality

Drivers - 2• Higher number of parties involved with their own agenda’s

– Internal development teams– Product suppliers– Outsourced partner– Systems integrator– Niche suppliers

• Different types of development e.g. Agile– Beware as lots of different “flavours” – Test driven development is a good thing and not really new– Some Agile proponents believe no integration or acceptance testing

is required• Complex architecture e.g. SOA “Service Oriented

Architecture”– High degree of retesting required– Not well understood in the market generally yet– Design wise makes sense if technical infrastructure can support it

Drivers - 3

• Different geographic locations and cultures– Additional management challenge– Method of working needs adjustment to fully benefit– Account of different cultures in planning and estimating

• Multiple skill types required within the team– An effective testing team has ALL the skills within IT life

cycle– Test managers usually manage progress during stormy

weather– Non-functional testing requires development, database,

infrastructure, design skills– Automation requires development skills– Requirements verification requires analysis skills– Plus all the testing skills!

Value Contribution

• Information provision– During the strategy, discover what information is

valuable this time around– It’s the stake holders view that matters

• Finding smarter ways to work– Maybe automation– Effective reporting techniques– Workshops to free blockages or identify risks

• Taking an overall view to reduce gaps and duplication– If an area of testing is “out of scope” who does own it?– Who is doing what?– Are we using the same standards?

Value Contribution - 2

• Contract negotiation and management– Risk assessment and audit of suppliers– Acceptance criteria and process within contract

• Breadth of skills– Yes ensure your team has testing skills, via training, on

the job coaching and experience– Also think about negotiating, project management, and

communications– Where can different types of testers come from e.g.

business, other areas in IT

Focus Areas

Where it frequently goes wrong!

Information Provision

• What can we provide to demonstrate realisation of business goals

• What risks are we monitoring to see if they are realised or mitigated against

• Remember risks can go up with testing as well as be reduced

• Have set of standard metrics for improvement• And a set of specific metrics for this project• Think about how information is provide, need to

be in the shoes of the receiver

Example

Risks

Coverage goals

Business goals

InformationDrivers

Objective Project PhaseReqs Design Build Integ Systest UAT Trial Prod.

Objectives for each project

phase are easily identified

Example

Viable for FutureEventsSeminar Outline

Booking Process

First Cut SeminarMaterials

Seminar Brochureand Web Site

Prepare SeminarOutline

Prepare SeminarBrochure/Web Site

Prepare Materials

eMailshot

Create BookingProcess

Process Bookings

Run Seminar

Review Seminar

SeminarMaterials Ready

Rework Materials

Walkthrough andRefine

3We have bookings

1Resource Available

2Good Feedback

4Materials Ready by

End June

5Mailshot ready by

End May

6Organise Venue

7First Cut isAcceptable

ExampleProject 1 2 3 4 5 6

Percentage of overall Effort

Analysis & Design 41% 7% 22% 7% 18% 14%

Code & Unit Test 41% 44% 33% 49% 52% 40%

System 11% 26% 33% 2% 20% 29%

UAT 7% 22% 12% 43% 9% 16%

Total Devt percentage 82% 52% 55% 56% 70% 55%

Total Test percentage 18% 48% 45% 44% 30% 45%

Number of Defects Found

Unit 20

System 256 2500 530 8 71 34

UAT 82 150 200 91 94 10

Implementation 0 5

Production 1 1 2

Total Defects Found 338 2650 730 120 171 46

Total Time to Fix Defects 1500 9000 2500 348 1380 85

Average Defect Fix Time (HRS) 4 3 3 3 8 2

Total Time to Find Defects 2400 3900 1238 2180 2267 169

Average Defect Find Time (HRS) 7 1 2 18 13 1

Estimation• Think top down as well as bottom up• Don’t forget coverage by type of testing• Capture actual against planned so you improve• Test support activities – environments, test data,

data migration• Skill types required, and where they will come

from• Plan using average skill level not your best people• Add risk factors to account for number of parties

involved and scale of integration required• How much change are you planning, don’t say

none!

Test effort is only a part of where the time goes on a project

Supplier Management• If it’s a product supplier, has the produce been

developed with testing in mind• What is their track record in support• What does their business model look like• How do they test internally• Be specific about acceptance criteria, for ALL project

phases• If you can’t document them, the supplier can’t meet

them – they are wrong!• Include acceptance process to manage exceptions –

there will always be some• Include test deliverables within contract scope – make

them control documents• Witness their testing• Sample their test plans, test cases and results

Example Proactive and progressive acceptance

Design & Build

System Test

Local IntLegacy

Pilot Beta RolloutUser

TestingLocal Int

New Full

Integration

Baseline and Key Acceptance Milestones

Acceptance Criteria

Transition Transformation

End to end view of testing

• Have a test strategy that goes from project inception through to deployment

• Clearly define who owns which element of test activities

• Clearly define the governance and reporting structure

• Who is going to monitor adherence to strategy and assist with reworking strategy as new information comes available

Example

External Influences

Build & Test Business Operations Support & MaintenanceWarranty

Outsource Outsource Outsource OutsourceOutsource

Own &ImplementPI Strategy And TestStrategy

Unit, SystemIntegrationDataEnvironmentsMuch more NF testing

Manage UATUsability testingSupport business procedures

ManagePerformanceSecurity,RecoveryReliabilityOpps proceduresDR

Basic MonitoringTest fixes, RegressionCausal analysisAutomation coverageBusiness continuity monitoring

Initiation

Senior Test Authority Owns Information Provision End-to-End

Supports Stakeholder Involvement/Project Assurance/Governance

Consultants

SuppliersProducts

Business Partners

Offshore

• Consider your quality plan – how will you know that things have been done to your standards and plan

• Where does management control lie• Where are the physical environments• Where are your subject matter experts• It’s another supplier• 24/7 testing isn’t the goal 24/7 project progress is• Consider your strategy carefully

Strategy Considerations• Reduced Cost. Clearly this is a driving factor behind every organisation to

consider offshore working. Not only for you, or UK organisations but also throughout parts of Europe and the USA.

• No degradation of quality. The geographic distance, different time zones and management of offshore testing cannot be allowed to affect the quality of the delivered projects. In fact, improvements in quality are frequently included in objectives for offshore testing.

• No increased timescales. Time to market is a critical business objective, which must not be compromised by the benefits of offshore working.

• Flexibility. The selected location(s) for offshore working must be prepared to deliver the level of flexibility you require, in particular regarding the timing of management communications and support for error rectification.

• Demand Fluctuation. The selected location(s) must be able to demonstrate how they will manage the increasing market demand from other organisations for their services, and how they will continue to deliver your required level of service.

• Growth Potential. The infrastructure of the offshore location(s) must support the development of relevant skills required for all aspects of testing identified as appropriate for offshore working.

• Maximise skills availability. Different offshore locations are tending to specialise in the development of particular skills relevant to testing and you must be able to benefit from this advantage.

• Risk Managed. With the political tensions arising in different geographies, it’s important that your operation is not compromised by offshore working locations.

Automation• Never seen a business case made on a single

project so it’s an investment• Needs a business case to plan and verify return

on that investment• Skills required are different to traditional tester –

depends on approach as to how much technical skill required

• Managers have to have an appreciation to be able to manage the automation team – or they’ll automate what ever they feel like

• Needs a strategy for automation coverage and goals

Automation Maturity

Capture Replay – use of tools out of the box with little expert knowledge

Capture Replay - simple modifications to automatically generated code - achieve a greater level of automation reuse

Hand crafted test tool script language code to make scripts more robust to environmental conditions

Use of Automation harnesses to separate the skills of the test tools expert and test analyst

Test Process Automation - tools are combined to automated the complete test process (maximum asset reuse)

Shelf ware !0

1

2

3

4

5

Environments

• It’s not just the setting up of the technical environment

• It’s the management of the components within it• If production support providing this service, you’ll

always take second place so plan for it or ring fence resources

• Have someone on your team in test support role so they can effectively communicate your requirements for provision and service level for management

Non-Functional Testing

• Consider all types during strategy• Getting requirements is always harder – needs

proactive approach• Special skills are required for each type of non-

functional testing, may need to bring in specialists from other areas of IT

Staff selection and assessment

• Interviews are the least effective element of selection process

• Plan interviews by role • Think about practical tests and aptitude tests prior to

interviews• Use role requirements with skill levels rather than just

read CV’s• The more senior or specialist the role, the harder it is to

find the right person, and you’ll need the right skills to recognise them in your candidates

• There is a lot of legislation now on recruitment, so make sure you’re up to date

• If you’re taking staff from another area, still measure them against the same criteria as it may change your plans depending on their skill level, best to identify this up front and manage it.

Example Candidate Selection Process

Application Aptitude Test

CV Telephone Interview

Practical Test

Interpersonal Skills Test

Structured interview

Personality Test

Final Structured Interview

Testers

Specialist

Non-functional Roles

Team Leader

Test Manager (teams of 20 plus)

Test Manager (teams of 50 plus)

Test

Architect

Screen out candidates Recruitment Decision

Y N NY Y YY YN

YY Y Y Y YN N N

YYYYY Y Y N N

Y Y Y Y Y Y Y N N

Y

Y

Y Y

Y Y

Y Y

Y Y

Y

Y

Y

Y

Y

YY

N

Thank You