test automation seminar momentum

SQS Group Limited

Scott Armstrong

Belfast, February 2010

Test AutomationA Roadmap For Success

Agenda

Steps for Success

Automation Tool Selection

Test Selection and Prioritisation

Implementing the Strategy

SQS Case Studies

Questions

2© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010

Steps for SuccessApproaching Test AutomationPlanning the Framework

© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010

Approaching Test AutomationSteps for Success

AUTOMATION PROJECT

DESIGN

Build maintainable tests

Target and prioritise test areas

TOOL

Select with care

Pilot tool

Manage Implementation

TEST MANAGEMENT

Defect Management

Config Management

Information and Reporting Requirements

TEAM

Code reuse and collaboration

Partnership between test and dev

PLAN

Treat as a Development Project

Manage Expectations

Commitment at all levels


Planning the FrameworkSteps for Success

What are the Management Information Requirements Contents Format

What are the Execution Requirements? On-demand Automated Nightly Builds

What technologies are we testing? Embedded Operating System

What resources are available? Hardware Technical Expertise

Are there cultural or organisational considerations? Outsourced IT Management – no admin access Current partnership with tool vendor


Automation Tool SelectionSelection ProcessIdentify RequirementsBuy, Open Source or Build


Selection ProcessAutomation Tool Selection

Define Problem

Consider automation

as a solution

Make business

case

Define required features

Shortlist

Evaluate

Demo

Trial

Decide

Constraints


Identify RequirementsAutomation Tool Selection

Start by evaluating your requirements, not by investigating the tool market.

Look at the current problems – what has prompted the search for an automated solution: Manual testing problems (e.g. too time consuming, error

prone); No time for regression testing when small changes are made

to the software; Set-up of test data or test cases is error prone; Inadequate test documentation; Don’t know how much of the software has been tested; Testing is ineffective.

Not all of these problems are best solved by automation.

Rank your problems by importance or current cost to the organisation.


Buy, Open Source or BuildAutomation Tool Selection

•Quickly provides a high level of features

•Well supported in terms of documentation, help and training

•Often an attractive project to be involved with

•Added cost of license fees

Commercial Tool

•No license fee cost

•Limited to support from community forums

•Defects in tool may not be addressed

•Tool may lack a clear roadmap; may not be future-proof

Open Source Tool

•No license fee cost

•Development and maintenance can prove expensive

•Designed to meet project requirements

•Lacks full range of features

Build Bespoke

Tool


Test Selection and PrioritisationTest SelectionTest Prioritisation


Test SelectionTest Selection and Prioritisation

Some tests cannot be automated.

Some tests are not worth automating. May take more time to automate tests than to run them

manually

You may not have to automate everything to get significant benefits 10% of the tests may be run 90% of the time.

Some types of tests are amenable to automation, some not: Functional tests are often the most straightforward Performance tests are notoriously difficult to carry out

manually. These are a good candidate for automation. Non-functional tests can cover maintainability, portablility,

testability, usability, etc, and can be inappropriate for automation.


Test PrioritisationTest Selection and Prioritisation

Do not automate too much too soon. Best practices take time to develop.

Go for ‘quick wins’ – the areas where automation will have the largest impact most quickly.

Prioritisation factors: A set of breadth tests (sample each system area); Tests for the most important functions; Tests that are easiest to automate; Tests that will give the quickest payback; Tests that are run the most often.


Implementing the StrategyAutomation Team FitMaintenance Model


Automation Team FitImplementing the Strategy

Automation Team

Specialist Group/Tool

Support

Subject Matter Expert

Management Buy-in

Development Team

Test Team


Maintenance ModelsImplementing the Strategy

• Client representative with technical background is trained during automation project

• Maintenance of current scripts can be supported independently for future product releases

• Future new tests can be automated in-house and added to the regression pack

Knowledge Transfer

• Client does not have to be involved in the details of automation development, execution or maintenance

• The necessary technical skills do not need to exist in-house

• Assured best practice automation and quality

Managed Service


SQS Case StudiesLloyds TSB Wholesale BankingLocal Utility Company


Lloyds TSB Wholesale BankingSQS Case Studies

Manual Testing Challenges•600+ Tests and

growing•200 man days per

release•Concerns over

coverage•Regression pack

growth impacted by limited resources

Our Engagement• Innovative test automation solution

implementation• Highly maintainable and extensible solution• Successful solution handover to client test team

Customer Success

Return on Investment•Increased test

execution efficiency by 85%

•Reduction in deployed defects >50%

•Enabled daily smoke test execution

•Overall ROI of £1.9m over 6 years

Lloyds TSB

“Without automation,

we would not have gone

live”

“The automated regression suite

made my job more interesting”


Local Utility CompanySQS Case Studies

Manual Testing Challenges•100 test scripts with

multiple complex data-driven iterations

•120 man days regression testing per year

•Low test coverage – limited to ‘spot checks’

•Testing of new functionality impacted by large regression packs

Our Engagement• Collaborative design of framework specific to

client requirements• Full knowledge transfer to existing test team for

ongoing maintenance and development• Modular framework allows new tests to be built

with minimal effort

Customer Success• SQS are now engaged on further automation

projects with the parent company• The existing test team continue to expand and

maintain the framework to increase their ROI

Return on Investment•Saving of 96 man

days of execution per year

•Data-driven scripts increased test coverage by a factor of approximately 200

•Automation solution was ported to a further 2 business units, multiplying the ROI

Local Utility


Thank you for your attentionThank you for your attention

SQS Group Limited

The Concourse | Northern Ireland Science ParkQueen‘s Road | Queen‘s Island | Belfast BT3 9DTPhone: +44 (0) 2890 022 660 E-Mail: [email protected]: www.sqs-uk.com


test automation seminar momentum

Technology