John Fodeh, Cognizant Technology Solutions

Adventures in Test Automation – Breaking the Boundaries of Regression Testing

www.eurostarconferences.com

@esconfs #esconfs

Insert speaker picture here, no more than 150x150 pixels

John Fodeh

Cognizant Technology Solutions

john.fodeh@cognizant.com

Adventures in Test Automation Breaking the Boundaries of Regression Testing

| ©2012, Cognizant

Outline

• Innovation and testing

• The promise of automation

• Going beyond regression testing

• Automated Test Monkeys

• Model-Based and Behaviour-Driven approaches

• The human factor in automation

3

| ©2012, Cognizant

The Promise of Test Automation

• Replacing repetitive and tedious manual tasks

• Ensuring the consistency and repeatability of tests

• Performing tests that are difficult to run manually

• Accelerating test execution

4 * Image source: Rosemont Engineering

| ©2012, Cognizant

Traditional Test Automation

5

Typically

Automation of regression tests

Purpose

Testing of a previously tested program following

modification to ensure that defects have not been

introduced or uncovered in unchanged areas of the

software, as a result of the changes made. It is performed

when the software or its environment is changed.

[ISTQB Glossary, v2.1]

| ©2012, Cognizant

Characteristics of Traditional Test Automation

• Static

• Simple

• Synchronized

• Vulnerable

6

| ©2012, Cognizant

Monkey Testing

7

Monkey Testing refers to the

process of randomly exercising a

software program by means of an

automated test tool

| ©2012, Cognizant

Test Monkey

8

#script file: Test4.asc CP_KEY FOCUS_UP; TRACKBALL 0, 64; KEY L; KEY 5, LONG; KEY 8;

#script file: Test4.asc CP_KEY FOCUS_UP; TRACKBALL 0, 64; KEY L; KEY 5, LONG; KEY 8; CP_KEY FOCUS_UP;

Random Test Tool

#script file: Test4.asc KEY L; KEY 5, LONG; CP_KEY FOCUS_UP; #KEY BOX_SIZE_UP KEY 8; TRACKBALL 0, 64; KEY SHIFT, R;

Action List

Log file

Application Under Test

#script file: Test4.asc KEY L; KEY 5, LONG; CP_KEY FOCUS_UP; #KEY BOX_SIZE_UP KEY 8; TRACKBALL 0, 64; KEY SHIFT, R;

Utilities

| ©2012, Cognizant

Basic Features

A test monkey should:

• Select randomly from input range

• Enter input to AUT through the user interface

• Detect “life signs” of AUT

• Have robust logging facility

• (Re-) start and initialize AUT

• Have limited application knowledge but general environment awareness

9

| ©2012, Cognizant

Using Test Monkeys

• Indicator of application reliability

• Incorporated into daily “smoke” test

• Establish confidence in application

• System test tool

• Evaluation of release candidates

• Previously failed scripts are run again as part of regression tests

• Focus tests on specific areas of interest

• Combinations of sequential and random scripts

• Random scripts with weight on specific actions

10

| ©2012, Cognizant

Metrics – Application Reliability Stability Trend for 4101

0

20000

40000

60000

80000

100000

120000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Build no.

Me

an

nu

mb

er

of

ran

do

m o

pe

rati

on

s b

etw

ee

n

Fa

ilu

res

Random operations Start Criterion Release Stability Criterion

11

Entry

Criterion

Release

Criterion

| ©2012, Cognizant

Added Value (1/2)

• Early testing

• Cost effective

• Negative testing

• Long and complex test runs

• Stability indicator

12

| ©2012, Cognizant

Limitations

• Misses obvious defects

• Does not emulate real use situations

• Long tests runs can be difficult to debug

• For a reliable statistical basis, long and multiple test runs are needed

13

Monkey Testing is a supplement

to your manual and automated

testing - not a replacement!

| ©2012, Cognizant

Advanced Test Monkeys

• Wide application knowledge

• Application modeled in state table

• Illegal input to test error handling and recovery

• Emulate real and complex use scenarios

• Effective in finding defects

• Higher development and maintenance cost

14

| ©2012, Cognizant

Test Monkey

15

Random Test Tool

State table

Log file

Application Under Test

Utilities

Start

State

Action End

State

Script Start state Action Probability

Input range End state

Scanning Adjust frequency 45 3 20 Scanning

Adjust brightness 20 1 100 Scanning

Abort 5 Displaying (abort)

Edit preferences 10 Editing preferences

Increment depth 20 Scanning

| ©2012, Cognizant

Intelligent Randomness

• Probability Tables

• Focus on selected areas

• Weight on specific use scenarios

• Continuous expansion and improvement

• Possibility to exclude certain parts

16

Select Text

Enter Text

Delete

Copy

Cut

Task n

…

Action

25%

5%

10%

5%

5%

* source: Whitmill, Kelly

| ©2012, Cognizant

Model-Based Test Monkeys

17

Random Test Tool

Log file

Application Under Test

Utilities

Start

State

Action End

State

Script State table Model

| ©2012, Cognizant

Next Step - Behavior-Driven approach

• Based on Behavioral specifications

• User Stories defined using ubiquitous language

• Principles of Test Driven Development

• Collaborative approach for business analysts, Software development & testing

• Shared tools and process

18

| ©2012, Cognizant

People Issues

• Applying test automation involves people • Your team members are not interchangeable machine

parts • Take that into account when implementing test

automation

19

20

12

Jo

hn

Fo

de

h

| ©2012, Cognizant

Summary

• No “one size fits all” solution

• The tools, processes and skills are there

• Maturity is about analyzing the current practice and finding ways to do evolve

• Are you ready for your innovation journey?

20