the 5 percent rules of test automation

8/14/2019 The 5 Percent Rules of Test Automation

1/38

The 5% Cha llengesof Test AutomationHans Buwa lda , CTO, Log iGear Corpora tion

March 31 April 2, 2009


2/38

2009 LogiGear Corporation. All Rights Reserved

Scope of this Workshop

Raising the bar for automation

the 5% challenges and their rationale

Show techniques, concepts, ideas toachieve the higher barhow could you achieve the challenges

How feasible is all of this in practice . . .


3/38


specification development

Testing Under Pressure

DEADLINE

testing


4/38


specification development testing

Develop tests in time:

Test design Auditing, acceptance Preparations

Automation

Testing Under Pressure

DEADLINE


5/38


The 5% Challenges

No more than 5% of all test cases shouldbe executed manually

No more than 5% of all efforts around

testing should involve automating the tests


6/38 2009 LogiGear Corporation. All Rights Reserved

Why a High Automation Degree ??

The best way to prepare for efficiency in the crunch zone

good manual test cases can help too, but marginally

Buy time to do more "exploratory testing" valuable addition to testing with test cases, since it is likely to find other

and unexpected bugs

Credible pay-off for the cost of introduction of automation

initial costs are: tooling, learning curve, adaptation of existing tests Automation is better positioned to identify bonus bugs

on average 15% of fixes cause new bugs

many of these bugs are hard to find without integral testing

often a result of violating overall architectures the bugs occur because data is left in inconsistent state

Automated test cases have a better chance of being kept up to dateif they form the majority of the testware



Why < 5% Automation Efforts ??

Automation should not dominate testing

it is not a goal in itself

may never be a bottleneck

Testers should be able to focus on testing

better tests (higher ambition level)

communication with stake holders

High automation efforts can aggravate the crunchzone, instead of relieving it

invitation to Murphys law

automation

should deliver,

not dominate



Vision on Automation

Have a good method

integrated method for testing and automation

testers in the driver seat maximum flexibility, best technology

minimal technical efforts

maximum maintainability

Have the right tool specifically designed for keyword based testing

flexible enough to allow for the right mix of high and low level testing

allow the testers to quickly build keyword tests, without hassle

is not overly complicated in the automation Succeed in the three "success factors for automation"

test design

automation solution

organization



Base Requirements

Test development and automation must be fully separated

testers are not / should not be programmers

the automation people must implement the action words, and test them

Test cases must have a clear and differentiated scope

and not deviate from it

The tests must be written in the right level of abstraction

for example higher business level, lower UI level, or both

the tooling must provide this flexibility

The method must be simple, not add to the complexity of theautomation

easy straightforward interpretation of tests, with minimal programming



Approaches to Automation

1. Record and Playback

2. Scripting / programming

3. Keywords

(simplified, see also: "Software Test Automation", Mark Fewster and Dorothy Graham)

it is my believe that an approach based onkeywords is a prerequisite to achieve a highlevel of stable automation



Keywords, Action Based Testing

number name quantitynew product P-9009 Sledge Hammer 5

number quantity

add quantity P-9009 20

add quantity P-9009 3add quantity P-9009 6

number quantity

check uantit P-9009 34

add quantity

check quantity

new product

function action_AddQuantity. . . .

end

function action_CheckQuantity. . . .

end

function action_NewProduct. . . .

end



Keywords, Action Based Testing

Describe test cases as a series of actions withkeywords and arguments, in a sheet like format

Externalize those descriptions out of thescripting

Let the testers (not the automation engineer)define the keywords, as by-product of the testdesign

Organize the test cases into "test modules",each with a clear and differentiated scope

Focus the automation efforts on the actions, not

on the test cases



Example of a Test Module

test objectivetest objective


14/38


Example of a Test Module, "low level"

TEST MODULE Product Maintenance

VERSION 1.0

. . .

TEST CASE Test Case 03 Use an invalid number

test requirement add 003 A number must be divisable by 9

source control destination

click main new new product

window control value

enter new product number P9001

source control

click new product process

text

check message Invalid product number.

test objective


15/38


Achieving the Challenges...

1. Test design

2. Automation architecture

3. Organization

Three major success factors:


16/38


Success Factor 1:

Test Design


17/38


Test Design

Effective test breakdown (into test modules)

make sure every test module has a clear focuskeep different kinds and levels of tests separate

Right level of actions

as high level if possible, hiding as many details asmuch as possible

but not if the details are relevant for the test

It is my believe that test design, not automationor the tool, is the deciding factor in achieving

the 5% challenges


18/38


Breakdown: Divide and Conquer

Mission, Requirements, System Design, Use Cases,Priorities, Planning, Budget, Stakeholders, ...

Identify test modules priority and planning what, when, who, how

Development

Execution

Maintenance

Test Module 1

Analysis

Development

Execution

Maintenance

Test Module 2

Analysis

Development

Execution

Maintenance

Test Module N

Analysis

. . .


19/38


Splitting into Test Modules

UI oriented tests does function key F4 work

does listbox xyz the right values is the tab order correct

Do individual functions work like transactions in a financial system

Alternate paths in use cases like cancel a transaction

End-to-end tests

Simulating business processes Tests with specific automation needs

like multi station tests

Tests of non-UI functions

High ambition level tests (aggressive tests)


20/38


Levels of Actions

High level actions to hide details

to hide unneeded details always hunt down details

implemented in terms of lower level actions

Low level actions to access the

automation technologyeither built-in in a tool like TestArchitector create yourself in a harness

this is the only real programming you would do


21/38


Example of Going High Level


22/38


Success Factor 2:

Automation Architecture


23/38


Automation: Implementing Actions

In a keyword driven approach the automation focuses onthe actions: as opposed to programming test cases in general this will lead to fewer and shorter scripts etc in the

automation

In the Action Based Testing method actions areimplemented in three ways: built-in in the tool, like click, enter, check etc defined in an action definition, using other (lower level) actions

similar to functions in a programming language

programmed in a programming or scripting language


24/38


"Action Definition"


25/38


Automation Architecture

Pay attention to where and how to implement actions using the action definitions (or something similar) for straightforward

actions use programming for

complex actions (typically involving loops) wrap specific technical code to access the interface

Select the right tools and technologies

some of the considerations are subtle technical choices strongly influence the ease of the automation make sure the technology is organized as durable/re-usable as possible Go for a mix of technologies (UI, APIs, SQL, TCP/IP, ...)

Manage the handling of the interface (UI, API, ...) make sure all dialogs, pages, controls etc get meaningful logicalnames (mapped to the actual items in the UI etc)

let the developers help you if necessary by providing easy to identifyproperties of UI elements (for example the id in Web apps, and

AccessibleName in .Net)


26/38


Tips for Stable Automation

Make the system under test automation-friendly developers are not always inclined to do that, but it pays off

ask to add specific property values to the GUI interface controls forautomated identification like accessible name in .Net and Java, or id in Web controls

Use "active" timing

wait based on conditions in the system under test, avoid fixed sleeps Test your automation

develop separate tests for the actions themselves

Use automation to identify differences between versions of the

system under test test in a loop if all control can still be found

Keep an eye on the test design unsuccessful automation virtually always can be traced back to poor

test design


27/38


Success Factor 3:

Organization


28/38


Organization

Much of the 5% success is also gained or lost in how you organizethe process

who does test design who does automation what to outsource, what to keep in-house

Write a plan of approach:

scope, assumptions, risks, planning methods, best practices tools, technologies, architecture stake holders, including roles and processes for input and approvals

team . . .

Assemble the right team testers, lead testers

automation engineer(s) managers, ambassadors, ...

see also:QA All-Stars: Building

Your Dream Team, HansBuwalda, Better

Software September

2006


29/38


Separation of Automation

TestDevelopment

Plan

Automation:

- design- research- development- testing- support

Focus on:- actions- interface handling- technology- never on tests

Test ModuleDevelopment

Test ModuleAcceptance

Plan of Approach

Execution and

Follow Up

intake


30/38


Are the 5% rules Achievable?

To achieve to high automation coverage tends to befairly straightforward

once the actions work all tests are automated dont settle for too little

be ware of "creative" complex solutions in test design andautomation, they tend to obscure readability and maintainability

The automation efforts need to be well managed, butthen can be kept low as well need of automation specialization and intelligence in the team

a method like ABT can provide a proper framework, but is only astart

The key to success is: think it through dont hesitate to revisit your design and automation

make sure the tests are easy to run


31/38


Typical Time Allocation

TEST DEVELOPMENT

AUTOMATION

time

efforts


32/38


What Does It Take More

The method, tooling and success-factors

mentioned earlier in this presentation Focus on your objectives, and monitor:

make sure stuff gets automatedmake sure the method is applied consistently

make sure the test designs are good

go aggressively after "can't do" claims

Give clear direction


33/38


Case Study

Large IT provider

New version of one of their major web-

sites

Test scope was user acceptance test

(functional acceptance)

the users were the business owners

Development was off-shore


34/38


Case Study

Test development was done separate from

automationtime-line for test development: May Oct

time-line for automation (roughly): Jan Feb

All tests were reviewed and approved by

the business owners

acceptance was finished by the end of thetest development cycle


35/38


Test Development PlanNr Module Business Owner Date to BO1 Portal Navigation, Audience Robyn Peterson 05 / 232 Portal Navigation, Search Ted Jones 05 / 273 Membership, registration Steve Shao 06 / 03

4 Portal Navigation, Category Ted Jones 06 / 085 Portal Navigation, Topic and Expert Ted Jones 06 / 13

6 Access Control Mike Soderfeldt 06 / 177 Portal Navigation, Task Ted Jones 06 / 228 Contact DSPP Ted Jones 06 / 279 Portal search Mike Soderfeldt 07 / 0110 Membership, review and update Steve Shao 07 / 05

11 Program contact assignment Alan Lai 07 / 1112 Company, registration Steve Shao 07 / 14

13 Catalog, view and query Robyn Peterson 07 / 1914 Site map Ted Jones 07 / 2515 Membership, affiliation Steve Shao 07 / 2816 Learn about DSPP Ted Jones 08 / 0117 Products and services Steve Shao, Robyn Peterson 08 / 08

18 What's new Ted Jones 08 / 11

19 Company, life cycle Steve Shao, Alan Lai 08 / 1720 Specialized programs Ted Jones, Steve Shao 08 / 2221 Customer surveys Ted Jones 08 / 2922 Software downloads Mike Soderfeldt 09 / 01

23 Newsletters Ted Jones 09 / 0624 Internationalization and localization Ted Jones 09 / 1325 Membership, life cycles Steve Shao 09 / 19

26 Collaboration, forums Ted Jones 09 / 2327 Collaboration, blogs Mike Soderfeldt 09 / 2828 Collaboration, mailing lists Ted Jones 10 / 03


36/38


Review Process with Stake Holders

Test Team sends draftModule to Stake Holder

Stake Holder reviews:- coverage- correctness

Stake Holder returnsnotes:- additions- corrections

Test Team receives andprocesses notes

changes needed?Stake Holder returnsnotice of approval

Test Team marks theModule as "Final"

END

no

yes

START


37/38


Case Study, Results

All tests were developed and reviewed on schedule many notes and questions during test development phase

The automation was 100% of the tests all actions were automated, thus automating all test modules

The test development took an estimated 18 personmonths one on-shore resource, two off-shore resources

The automation took between one and two months focused on actions most time was spent in handling changes in the interface (layout

of pages etc)


38/38

Summary

Good to set aggressive targets for the automation: it costs money and efforts it should not get in the way of the testers making good tests

Elements for success: test design automation architecture

organization

Focus on the method first the tool is only a minor element use a method like Action Based Testing that is based on keywords and

addresses topics like test design

The 5% targets can be achieved: automation coverage can commonly achieve, can even reach 100% to limit efforts (to 5% of all efforts) requires serious planning and

thinking ahead. It also does not include a learning curve for the method

the 5 percent rules of test automation

Documents