john kent - europe's largest software testing …...‘scripted’ tools e.g.winrunner, qarun,...

International Conference On

Software Testing, Analysis & ReviewNovember 19 - 23 Stockholm, Sweden

P r e s e n t a t i o n

W4Wednesday 21st November, 2001

State Of The ArtAutomation

John Kent

Wednesday 21 November 2001

W4

State of the Art Automation John Kent John Kent is a leading consultant specialising in test automation and test management. He writes two columns for Professional Tester Magazine: The View From Kent and Technology for Testers. He has a special interest in the Web and is co-author of the Official Netscape Guide to JavaScript1.2 (Netscape Press, 1997). John regularly presents to international audiences on a range of software testing subjects including his own training course: Using Automated Test Tools.

John Kent M.Sc. John Kent M.Sc. CISS LtdCISS Ltd

Tel: 01483 720500Tel: 01483 720500

Email: [email protected]: [email protected]

Web: http://www.cisstest.comWeb: http://www.cisstest.com

© CISS Ltd 2001

State of The Art State of The Art AutomationAutomation

nn Overview of Test AutomationOverview of Test Automation

nn Current Automated Test MethodsCurrent Automated Test Methods

nn Reasons For New ApproachesReasons For New Approaches

nn State of The Art AutomationState of The Art Automation–– Advanced ArchitecturesAdvanced Architectures

–– Test Code Creation MethodsTest Code Creation Methods

–– Reusable Test ObjectsReusable Test Objects

–– Test Process IntegrationTest Process Integration

ee

ContentsContents

© CISS Ltd 2001

nn Automated Functional Testing using Automated Functional Testing using ‘scripted’ tools e.g.WinRunner, QARun, ‘scripted’ tools e.g.WinRunner, QARun, SQA RobotSQA Robot

nn Aiming at large regression tests Aiming at large regression tests -- large large number of automated testsnumber of automated tests

nn Use automated Testing to increase Use automated Testing to increase ‘testedness’ of system‘testedness’ of system

nn Integration into test management toolsIntegration into test management tools

© CISS Ltd 2001

ee 1. Overview of Test 1. Overview of Test AutomationAutomation

ee Overview of Test Overview of Test AutomationAutomation

LogLog

Test Management Test Management ToolTool

Step01Step01 PassPass

Step02Step02 FailFail




Test ExecutionTest Execution

SUTSUT

Test01 Add CustTest01 Add Cust

Test AutomationTest Automation

nn Record/PlaybackRecord/Playback

nn Program/PlaybackProgram/Playback––Basic Data DrivenBasic Data Driven

––Advanced ArchitectureAdvanced Architecture

© CISS Ltd 2001

ee 2. Current Approaches 2. Current Approaches to Test Automationto Test Automation

1 Record/Playback1 Record/Playback

© CISS Ltd 2001

2 Data Driven2 Data Driven

ee Test Automation Test Automation Maturity ModelMaturity Model

BeginnersBeginners

Most Automation ProjectsMost Automation Projects

Program/PlaybackProgram/Playback

Manual Test

Procedures

Add customer “Bloggs”

Create an order for product X

Translated into test

tool language

Automated Test Code

Menuitem

”customer”

Type(“Bloggs”)

.....

Menuitem

”Order”

© CISS Ltd 2001

ee

Record/PlaybackRecord/Playback

TestsTests

“Bloggs”, “John”

DataDataFileFile

Test ProgramsTest Programs

Navigates to screen/window and inputs data

© CISS Ltd 1999

ee

Basic DataBasic Data--DrivenDriven

nn Automation not fulfilling its promiseAutomation not fulfilling its promise

nn Current architectures not deliveringCurrent architectures not delivering

nn Importance of MaintenanceImportance of Maintenance

© CISS Ltd 1999

ee 3. Reasons for New 3. Reasons for New ApproachesApproaches

nn Test data is hardTest data is hard--codedcoded

nn SUT Behaviour is hardSUT Behaviour is hard--codedcoded

nn Test Procedure: test program ratio =1:1Test Procedure: test program ratio =1:1

nn Many test programs to maintainMany test programs to maintain

nn Many test programs to testMany test programs to test

nn Becomes victim of own success: the Becomes victim of own success: the more tests, the more test programs to more tests, the more test programs to maintain and testmaintain and test

© CISS Ltd 1999

ee Disadvantages of Disadvantages of Record/PlaybackRecord/Playback

nn Recording does not workRecording does not work

nn Recording produces unmaintainable Recording produces unmaintainable codecode

nn We are interested in what We are interested in what maymayhappen, not what happen, not what hashas happenedhappened

nn An automated test should be an An automated test should be an interactioninteraction between the tests and between the tests and System Under Test(SUT)System Under Test(SUT)

© CISS Ltd 2001

ee Disadvantages of Disadvantages of Record/Playback(Cont)Record/Playback(Cont)

nn Most SUT Behaviour is still hardMost SUT Behaviour is still hard--codedcoded

nn Test Analysts are not free to test what Test Analysts are not free to test what they like they like -- totally business scenario totally business scenario drivendriven

nn Repetitive data fed into system Repetitive data fed into system -- not not like a testlike a test

© CISS Ltd 1999

ee Disadvantages of Disadvantages of Basic Data DrivenBasic Data Driven

nn DevelopedDeveloped

nn TestedTested

nn MaintainedMaintained

Test Automation Code Test Automation Code Must Be:Must Be:

© CISS Ltd 1999

ee Automated Testing is Automated Testing is Software EngineeringSoftware Engineering

nn Advanced Automation ArchitecturesAdvanced Automation Architectures

nn Test Analysts EmpoweredTest Analysts Empowered

nn New Ways to Create Test Automation New Ways to Create Test Automation CodeCode

nn Reusable Test ObjectsReusable Test Objects

nn Test Process IntegrationTest Process Integration

© CISS Ltd 1999

ee 4. State Of The Art 4. State Of The Art AutomationAutomation

““The software architecture of a program The software architecture of a program or computing system is the structure or or computing system is the structure or structures of the system, which structures of the system, which comprise software components, the comprise software components, the externally visible properties of those externally visible properties of those components, and the relationships components, and the relationships among them.”among them.”

Bass, Clements, and Kazman of Carnegie Mellon’s SEI Software Architecture in Practice see .sei.cmu.edu/ata/ata init.html

© CISS Ltd 2000

ee What is a Software What is a Software Architecture?Architecture?

“...designing and specifying the overall system structure emerges as a new kind of problem. Structural issues include gross organization and global control structure; protocols for communication, synchronization, and data access; assignment of functionality to design elements; physical distribution; composition of design elements; scaling and performance; and selection among design alternatives."

Mary Shaw and David Garlan in An Introduction To Software Architecture (1993) see .sei.cmu.edu/ata/ata init.html

© CISS Ltd 2000

ee What is a Software What is a Software Architecture?Architecture?

Interface Interface ModulesModules--WrappersWrappers

Add customer “Bloggs”

DataDataFileFile


Interaction

InfrastructureInfrastructure

Drivers, Library functions, Etc..

© CISS Ltd 1999

ee Advanced Automation Advanced Automation ArchitecturesArchitectures

EXCEL

Text

csv

Test DataTest Data+Navigation+Navigation+Actions+Actions

QARun

WinRunner TSL

Test BASIC

Test Code Test Code (Programs)(Programs)

ActionsActions

© CISS Ltd 2000


Advanced Architectures seek to move the navigation and test actions out of the test

tool code into the test data

© CISS Ltd 2000


Advanced Architectures seek to move the navigation and test actions out of the test

tool code into the test data

This empowers the Test Analyst by allowing them to choose what the test actions are

Does not hide test actions by automation code making decisions on the fly

nn Business Object LevelBusiness Object Leveluu Business tasks are automatedBusiness tasks are automated

uu Tests specified at business language levelTests specified at business language level

uu Test Programs are ‘wrappers’ for business Test Programs are ‘wrappers’ for business tasks tasks -- must be codedmust be coded

uu Not all actions/navigation removed from codeNot all actions/navigation removed from code

© CISS Ltd 1999


Two Types of Advanced Architecture:Two Types of Advanced Architecture:

nn Screen/window Level ArchitecturesScreen/window Level Architecturesuu Screen/windows are ‘wrapped’ by test Screen/windows are ‘wrapped’ by test

programs. Test program/screen ratio = 1:1programs. Test program/screen ratio = 1:1

uu UI ObjectUI Object--toto--UI Object Reference ratio of 1:1UI Object Reference ratio of 1:1

uu All actions/navigation removed from codeAll actions/navigation removed from code

uu Test Analysts given complete control to create Test Analysts given complete control to create the tests they wantthe tests they want

uu Can use new ways to create automation code Can use new ways to create automation code by analysing the SUT windowsby analysing the SUT windows


Two Types of Advanced Architecture:Two Types of Advanced Architecture:


© CISS Ltd 2001


3 Advanced Architecture3 Advanced Architecture


ATAA (CISS)TestFrame(CMG+SDT)


BeginnersBeginners


Program/PlaybackProgram/Playback

CUST ADD Smith John Perregren 29 Abbey Rd

ORDER ADD Widget Green 2inch

CUST UPDATE Smith Jon 30 Abbey Rd

CUST CHECK Smith Jon Perregren

CUST ADD B;loggs Jonhansen CUST-01

nn Actions in Test DataActions in Test Data

nn Navigation in Test DataNavigation in Test Data

nn WrappersWrappers


#Screen Main Menu ===># TEST TESTSTEP AUTOSTEP RUNDATE ACTIONKEYTESTACTIONOPTION Message# B L=3 B L=2

MGO000C1 ACCT023 1 1 Return CDBMGO000C1-01ACCT023 1 2 Return CRG#Screen CBO500F1 =>Direct Select# TEST TESTSTEP AUTOSTEP RUNDATE ACTIONKEYTESTACTIONFHUSER FHHEAD FHTIME FHDATE# O L=10 O L=50 O L=6 O L=10

CBO500F1 ACCT023 1 3 F9#Screen CBO56010 =>Title :# TEST TESTSTEP AUTOSTEP RUNDATE ACTIONKEYTESTACTIONFHUSER FHHEAD FHTIME FHDATE# O L=10 O L=50 O L=6 O L=10

CBO56010 ACCT023 1 4CBO56010 ACCT023 1 5 Return Check#Screen GPO020F2# TEST TESTSTEP AUTOSTEP RUNDATE ACTIONKEYTESTACTIONPOSTCODEADDR1 ADDR2 ADDR3# O L=10 B L=39 B L=39 B L=39

GPO020F2 ACCT023 1 6 F10#Screen CBO56010 =>Title :# TEST TESTSTEP AUTOSTEP RUNDATE ACTIONKEYTESTACTIONFHUSER FHHEAD FHTIME FHDATE# O L=10 O L=50 O L=6 O L=10

CBO56010 ACCT023 1 7 CheckCBO56010 ACCT023 1 8 ReturnCBO56010 ACCT023 1 9 F10

#ONLINE: Create an Account# ACCT023 1

ee Empowering Test Empowering Test AnalystsAnalysts

nn For Screen/Window level Architectures For Screen/Window level Architectures OnlyOnly

nn Automatic Analysis of SUT to produce Automatic Analysis of SUT to produce model of User Interfacemodel of User Interface

nn Test Automation Code Generated from Test Automation Code Generated from model.model.

nn Generate/PlaybackGenerate/Playback

ee New Ways to Create New Ways to Create Automation CodeAutomation Code

Write Write Automation Automation ScriptsScripts

© CISS Ltd 2000

Create Create Test Test DataData

MaintainMaintainAutomation Automation ScriptsScripts

Test TestsTest Tests


Automation Project Tasks: Program/PlaybackAutomation Project Tasks: Program/Playback

© CISS Ltd 2000

Create Create Test Test DataData

Test TestsTest Tests

Automation Project Tasks: Generate/PlaybackAutomation Project Tasks: Generate/Playback

GenerateGenerateAutomation Automation ScriptsScripts

Automatic Impact Automatic Impact Analysis &Analysis &ReRe--generate generate ScriptsScripts



© CISS Ltd 2001


3 Advanced Architecture3 Advanced Architecture




BeginnersBeginners


Program/PlaybackProgram/Playback 4 Mapping Methods4 Mapping Methods

5 Script Generators5 Script Generators

CISSTestCISSTest

SAG (CISS)RADStar (ICT)

SAG (CISS)RADStar (ICT)

Generate/PlaybackGenerate/Playback

nn Each Screen/window has test code Each Screen/window has test code ‘Wrapper’‘Wrapper’

nn Wrapper generated for each Screen/WindowWrapper generated for each Screen/Window

nn Layout of data for test analysts also Layout of data for test analysts also generatedgenerated

ee

Wrapper GenerationWrapper Generation

nn Test Object is a reusable piece of Test Object is a reusable piece of test datatest data

nn Can use it in other testsCan use it in other tests

nn Can overCan over--ride valuesride values

nn Puts inheritance into automated testsPuts inheritance into automated tests

ee

Test Object ReTest Object Re--useuse

nn Automated tests should integrate into Automated tests should integrate into the manual test processthe manual test process

nn Should be able to report on all tests, Should be able to report on all tests, manual and automatedmanual and automated

ee Test Process Test Process IntegrationIntegration

ee Test Process Test Process IntegrationIntegration

InfrastructureInfrastructure

Add customer“Bloggs”

DataDataFileFile


Interaction

Drivers, Libraryfunctions, Etc..

© CISS Ltd 1999

ATAAATAA

Screen/Screen/windowwindowwrapperwrapper

LogLog

Test Management Test Management ToolTool


Step02Step02 FailFail




Test01 Add CustTest01 Add Cust

© CISS Ltd 1999

State of the art automation is about State of the art automation is about empowering the test analyst.empowering the test analyst.

Automation should be a conduit Automation should be a conduit between the tests and the system between the tests and the system under testunder test

ee State of the Art State of the Art AutomationAutomation

john kent - europe's largest software testing …...‘scripted’ tools e.g.winrunner, qarun,...

Documents