developing a test automation strategy by brian bayer

test automation strategyDeveloping a

why are we here?

regression testing

the greatest cost of development

regression testing

you have already paid for this at least once

why regression test?


quality ensures what was working, still works


quality

risk

moneylost revenue, market, contractual

ensures what was working, still works

deliverables, release, schedule


a project withoutthorough and continuous regression testing

is like a road tripwhere you do not look out the window

until the end of the trip

why automate? (vs. manual)


TimeMoney


Money


Time

automation strategywhat to consider...

tests are living specifications of the system


ARE THE TESTS GOING TO BE USED?


business needs

return on investment

organizational structure and culture

automation strategybusiness needs...

determine areas of focus of the automation

automation strategyreturn on investment...

limits and prioritizes the areas of focus of the automation

determines at which level to automate a test

determines the automation framework

determines the success or failure

determines the speed, ability & openness to change

determines the communication

determines the ability to learn

automation strategyorganization and culture...

automation strategyorganization and culture...

determines who uses and maintains the tests

determines the automation framework too

automation strategywhat else to consider...

automation strategywhat else to consider...

development process and lifecycle

maintenance of the tests

technology stacks

automation strategydevelopment process and lifecycle...

legacy (system without tests) to greenfields


system under test development process

methodology: waterfall agile xp ad hoc

is automation development part of the same process?


is the automator part of the development team?

how much access to the system developers?

automation strategymaintenance of the tests...

who and how much can they maintain?

how are the system developers involved?

automation strategymaintenance of the tests...

how timely can/must tests be maintained

what reporting needed to maintain tests

automation strategyfinally, technology stacks...

languages and technology already used

unit and integration testing

source code control, continuous integration

devops or operations

where to automate?

who is going to use the tests?

developers, maybe...

depends

where to automate?

https://msdn.microsoft.com

“Its essential point is that you should have many more low-level unit tests than high level end-to-end tests running through a GUI.” Martin Fowler

where to automate?

UNIT LEVELINTEGRATION LEVEL

GUI INTEGRATION LEVELXPtest driven development

ACCEPTANCE LEVEL BDD


where to automate?

acceptance testsonly if the business is engaged and

going to use them


where to automate?what to consider...

DRY

look for low-hanging fruit

risk-based testing

what computers do well, humans not so much

fragility of test

where to automate?what to consider...

which automation framework to use?


Gherkin/Cucumber

Robot Framework

Selenium Webdriver

Spock/Geb

JUnit

Jasmine

TestNG

which automation framework to use?what to consider...

ORGANIZATIONAL STRUCTURE AND CULTURE

which automation framework to use?what to consider...

who is reading the tests?

who is writing and maintaining the tests?

balance of right tool with existing technology, knowledge, and ability to learn

ORGANIZATIONAL STRUCTURE AND CULTURE

which automation framework to use?what else to consider...

natural language for high level specification

goal is to develop tests NOT framework

and...

grids...

integration...

reporting...

teh sexy

gherkin/cucumber/ruby

but JRuby (JVM), Java, Groovy, JavaScript,.NET (using SpecFlow)

libraries (Selenium, Watir)

Ruby has the richest ecosystem with gems

gherkin/cucumbergenerally ruby based...

Feature: Cash Withdrawal

Scenario: Successful withdrawal from an account in credit Given my account has been credited with $100 When I withdraw $20 Then $20 should be dispensed And the balance of my account should be $80

Given /^my account has been credited with (#{CAPTURE_CASH_AMOUNT})$/ do |amount| my_account.credit(amount)end

also runs on Jython (JVM) & IronPython (.NET)

libraries (Selenium, Watir, SOAP, etc.)

keyword based

supports “given when then”

supports tabular/data driven

robot frameworkpython based...

*** Settings ***Library Selenium LibrarySuite Setup Start Selenium ServerSuite Teardown Stop Selenium ServerTest Setup Setup TestTest Teardown Close All Browsers

*** Testcases ***Login Should Succeed When the Correct Username and Password are Entered Enter Username AUser Enter Password TestPass Click the Login Button Login is Successful

*** Keywords ***Setup Test Open Browser http://zdiles.chaosnet.org/ googlechrome Maximize Browser Window

Enter Username [Arguments] ${username} Input Text uname ${username}

Login Is Successful Page Should Contain Welcome

*** Testcases ***Login Should Succeed When the Correct Username and Password are Entered Given I Enter Username AUser And I Enter Password TestPass When I Click the Login Button Then Login is Successful

given when then

spock/geb/groovy

developing in java or grails?

groovy-based browser automation tool

built on Selenium WebDriver

works with JUnit, TestNG, Spock

page models and jQuery selector syntax

spock/geb/groovygeb...

groovy-based test and specification framework

supports “given when then”

tabular/data driven

spock/geb/groovyspock...

import geb.spock.GebSpec

class GebHomepageSpec extends GebSpec {

def "can access The Book of Geb via homepage"() { given:”One is on the Geb Home Page” to GebHomePage

when: “I click the jQuery Link” highlights.jQueryLikeApi.click()

then: “I should be at the jQuery Section” highlights.jQueryLikeApi.isSelected }}

import geb.Page

class GebHomePage extends Page { static url = "http://gebish.org"

static at = { title == "Geb - Very Groovy Browser Automation" }

static content = { highlights { $("#sidebar .sidemenu").module(HighlightsModule) } sectionTitles { $("#main h1")*.text() } }}

FRAMEWOR

KS

other options...

raw Selenium (Java)

excel spreadsheet data driven scripts

script capture record/playback

flat files in source control

other options...


the simplest thing that could possibly work

testsare specifications of the system...

how does it fit?

how does it fit...

design your approach into the organization

how does it fit...

find the gaps and tight squeezes

the specifics...

the mechanics...

so what’s left?

documentation of the strategy...

execution of the strategy...

if project management...

estimation...

resources...

developing a test automation strategy by brian bayer

Technology