test automation strategy

IBM Global Services – Testing Competency

IBM Confidential © Copyright IBM Corporation 2005

01/23/2005

Strategy, Approach & Planning

Author: Martin Ruddy

20 August 2015


© Copyright IBM Corporation 2005

Agenda

IBM Confidential

2

Introduction to Test Automation

Benefits from Test Automation

Trusted Approach

Real Facts & Think Factor

Considerations for Automation

Guidelines & Approach

Automation Methodologies

Overview on Automation Tools

Evaluation, Planning and Review Checklists

Guidelines to Automation Testers and Developers


20 August 2015



Introduction to Test Automation

IBM Confidential

3

GUI automation is always viewed as the SILVER BULLET, which will save money,

time and resources

Test automation raises our hopes, yet often frustrates & disappoints us

Promises to deliver us from a tough situation, yet can create as many problems as

it solves

"Embarking on a successful automation test project is similar to embarking

on a successful software development project“


20 August 2015



Benefits from Test Automation

IBM Confidential

4

Reduced utilization of manual resources

More efficient use of scarce hardware resources

Reduced test cycle time

Reduced cost of testing, after initial cost of implementation

Improved reliability of the testing

Repeatable tests, thus overcoming boredom-induced missed bugs

Improved test coverage, since testing may be done more frequently

Development of programming skills

May allow the use of less skilled staff to do the actual testing


20 August 2015



Trusted Approach

IBM Confidential

5

Understand the need / requirement for automation

Point out pitfalls to avoid

Promote an evolving implementation

Begin simple and increase in complexity as the team's skills increase

"Learning from others is the most effective way to grow your automation expertise"


20 August 2015



Real Facts & Think factors

IBM Confidential

6

Real Fact

Automation is initially expensive and it may not do a better job than manual testing

in some situations

Think Factors

1. How much will automating and running a test cost, versus running it manually?

2. Automated test has a limited lifetime in which it must recoup the additional cost.

Will this test be reusable long enough? What factors will prevent it from being

reusable?

3. The automated test will find bugs it was designed to test for and in addition it will

find bugs that are not directly related to the test. What is the likelihood that this test

will find additional bugs?


20 August 2015



Considerations for Automation

IBM Confidential

7

Test cases those are long running, repetitive, and non-subjective

Projects / Products that have a working, relatively stable GUI

Projects / Products that will span multiple releases.


20 August 2015



Test Automation Guidelines & Approach

IBM Confidential

8

Record / Playback initially using the Testing tool’s wizard or toolbar will be easier but hard to

maintain because they represent not only WHAT needs to be tested, but HOW it is to be tested

Guideline: 1

Once familiarity in scripting is achieved, continue using the scripting language of an automation

tool to provide your own implementation logic (framework) that is separate from your test case

scripts

Requires more investment upfront, but the code will be less expensive to maintain over the life

span of multiple product releases

Example:

Imagine having recorded test case scripts for a banking application that initially allowed the user to

simply click "OK" to commit a sensitive transaction, then the application developers changed the

navigational logic requiring the user to type "yes" to confirm these transactions. Every recorded

test case that has hard coded "OK" clicks relating to the sensitive transaction commits would

require maintenance.


20 August 2015



Test Automation Guidelines & Approach (Contd..)

IBM Confidential

9

Automation code without any Discipline - Tendency to jump in and start

developing new test scripts without collaborating with other team's members and

without establishing a plan, structure, and/or standards is natural. However, in the

long term this undisciplined approach is not productive. Any test case scripts and

frameworks that are produced in this manner tend to lack such characteristics as:

maintainability, reusability, readability, flexibility, reliability, effectiveness, and efficiency

Guideline: 2

Establish complete Automation plan, Structure, and Standards

“Quality automation effort should follow a software development process”


20 August 2015



Guidelines & Approach (Contd..)

IBM Confidential

10

Guideline: 3

“Separating the What from the How”

An approach to separate what needs to be tested from how it is to be tested. To

accomplish this separation, guideline would suggest to have the test script writers

write the test scripts to identify what needs to be tested and automation developers

write a framework to handle the implementation (i.e., the navigation logic, window

processing logic, and verification logic) of your tests.

Framework - "a supporting or enclosing structure”


20 August 2015


© Copyright IBM Corporation 2005 11 IBM Confidential

Guideline: 4

Use the 6 phases for implementing the best course of Automated Testing action

- Evaluation

- Planning

- Design

- Implementation

- Deployment

- Review

Iteration Cycle – Plan for future changes



20 August 2015




IBM Confidential

12

Guideline: 5

A. Define Automation Goals & Prioritize

At every point in the specification process, you can check the "testability" of the requirement or specification by

ensuring it is S.M.A.R.T.

S - Specific

M - Measurable

A - Attainable or Achievable

R - Realistic

T - Timely

B. Define Automation Strategy

- GUI - Roundtrip / Segment, Define objects, Synchronization

- Types of Testing

- Strategy Considerations

1. Partial automation

2. Time cost

3. Testing delay

4. Inertia

5. Risk of missed bugs


20 August 2015




13 IBM Confidential

Guideline: 6

Define Users and Skill Level Required / Desired

- Test Script Writers

- Automation Developers

Guideline: 7

Resource Planning / Source Code Management

- Hardware

- Software

- People


20 August 2015




IBM Confidential

14

Guideline: 8

Define Design Goals & Prioritize

- Maintainability

- Structured

- Well Documented

- Reviewed

- Framework Specific Goals

- Modularity

- Reusability

- Usability

- Scalability

- Reusability

- Portability

- Robustness

- Reliability

- Efficiency

- Flexibility


20 August 2015




IBM Confidential

15

Guideline: 9

Determine the Automation Interface Architecture

- Data Driven Interface

- Programming Interface

Guideline: 10

Determine the Automation Framework Architecture

- Error Recovery

- Support of Custom Controls

- Product Testability Issues

- NLS – Localization

Guideline: 11

Define Standards – Naming & Code Conventions


20 August 2015




IBM Confidential

16

Components of Automated Test System

Test Script Writers

Automation Developers

Test Scripts

“WHAT”

Interface

Framework

“HOW”

Documents

Framework Usage Notes

Guidelines

Naming Conventions


20 August 2015



Automation Methodologies

IBM Confidential

17

Functional Decomposition Method

This methodology is to reduce all the test cases to their most fundamental

tasks, and write User-Defined Functions, Business Function Scripts, and

"Sub-routine" or "Utility" Scripts which perform these tasks independently of

one another

Key-Word Driven or Test Plan Driven Method

This method uses the actual Test Case document developed by the tester

using a spreadsheet containing special "Key-Words". This method preserves

most of the advantages of the "Functional Decomposition" method, while

eliminating most of the disadvantages. In this method, the entire process is

data-driven, including functionality.

The Keyword controls the entire processing


20 August 2015



Overview on Automation Tools

IBM Confidential

18

Most of the Automated Tools in the market remain as an integrated suite for Test

Documentations, Scripting, Executions & Defect Tracking

Automation tool does exist for specific testing needs such as

Performance Testing

Code Coverage

Memory Leak Detection

Automation Tools available in the market which are more on focus today includes

Rational Tools – Robot, Test Manager, ClearQuest

Mercury Interactive – WinRunner, LoadRunner, QTP, Astra Load Test

Segue – Silk Performer


20 August 2015



Evaluation – Sample Checklist

20 IBM Confidential


20 August 2015



Planning – Sample Checklist

21 IBM Confidential


20 August 2015



Review – Sample Checklist

22 IBM Confidential


20 August 2015



Guidelines for Test Script Writers

23 IBM Confidential

1. Be sure test scripts remain independent of each other

2. Define good input data to catch boundary or NLS errors. For example, for integer

fields, be sure to identify the minimum and maximum values that are supported.

3.To the greatest extent possible, the test scripts should use the published

interface(s) of your automated test system

Guidelines for Automation Developers

1. Be sure to incorporate GUI verification wherever possible.

2. While waiting for the development of the GUI to stabilize, it is a good time to

implement your framework design for a simple task/function in the product under

test

3. Long running suites should have the ability to continually dump status as the

testing progresses. This helps ensure that long running tests that fail have some

interim status reported before the failure.


20 August 2015