you have selenium... now what?

You have Selenium... Now what?

Hans Buwalda

©2016 LogiGear Corporation

www.logigear.comwww.testarchitect.comwww.happytester.com

@[email protected]@happytester.com


• Topics:– domain language approaches– test design as driver for automation– automation friendly test design techniques and examples– special cases (data driven, multi-station, graphics, etc)– drone perspective: overview with some selective details

• What this talk is not:– an interactive workshop– specific to Selenium only– a technical class on automation– an interactive workshop– about component based development– introduction to unit testing

Scope

Warning: there is an overlap with my talk yesterday!!


• A combination of open source products and standards, aimed at web testing (including a Firefox based IDE)

• In the later versions it works with an interface called “WebDriver”, implemented with drivers in browsers

• WebDriver has been implemented in open source libraries, but is now also becoming implemented in browsers

• For example Microsoft Edge can only be accessed programmatically via WebDriver

• Products like Appium try to implement WebDriver for mobile browsing and even mobile apps

• Our product (TestArchitect) is moving to WebDriver, expect this to be a common practice

Selenium


Relation to code Quality / depth Automation Scalability

Unit TestingClose relationship with the code

Singular test scope, but deep into the code

Fully automated by nature

Scalable, grows with the code, easy to repeat

Functional/Integration Testing

Usually does not have a one-on-one relation with code

Quality and scope depends on test design

In particular UI based automation can be a challenge

Often a bottle-neck in scalability

Exploratory Testing

Human driven, not seeking a relation with code

Usually deep and thorough, good at finding problems

May or may not be automated afterwards

Not meant to be repeatable. Rather do a new session

Some test kinds and their scalability (simplified)


Test Pyramid

• Proposed by Mike Cohn (in his book: "Succeeding with Agile").• My version would be less "pointy"• In particular I like to separate technical and design considerations• Also, I've seen API's that, for higher level operations, are quite complex. Trying to use

them can become like re-implementing the UI again• Web UI's are getting richer and more complex to test (SPA's, AngularJS, etc)

UI

Service orComponent

Unit

UI

Unit

original my version (for now)

Service orComponent


Issues are not always obvious...

Downton Abbey


Brief test design exercise

• View this imaginary application• It adds two numbers

• What tests would you create?• Which tests would you automate?


The 5% Rules of Test Automation

• No more than 5% of all test cases should be executed manually

• No more than 5% of all efforts around testing should involve automating the tests


Don't just automate manual testing


Don't just automate manual testing

Good automated testing is not the same as automating good manual testing. . .


Example Scripting, Java + Selenium

WebElement element = driver.findElement(By.name(name));

element.sendKeys("mystery magic");element.submit();

(new WebDriverWait(driver, 10)).until(new ExpectedCondition<Boolean>() {

public Boolean apply(WebDriver d) {return d.getTitle().toLowerCase().startsWith("mystery");

}}

);

System.out.println(driver.getTitle());


• Frameworks, libraries, patterns– libraries: sets of functions, you call them– frameworks: also sets of functions, but the framework is typically in

charge and calls you– patterns: standardized ways to do certain tasks

• Selenium is often used with frameworks– to execute tests and collect results (like unit testing frameworks)– to offer higher level API's– to support BDD, data driven testing or keywords

• A common practice is the Page Object pattern– shield from HTML– provide an application specific API, with functions like "getSongTitle"

• Examples of keyword frameworks: Robot Framework, and our own TestArchitect

Libraries, frameworks, patterns

"don't call us, we'll call you"

Framework


Domain language approaches: Actions

4 actions, each with an action keyword and arguments

read from top to bottom

fragment from a test with actions

acc nr first last

open account 123123 John Doe

acc nr amount

deposit 123123 10.11deposit 123123 20.22

acc nr expected

check balance 123123 30.33

• Stays out of the programming language

• The test developer creates tests using actions with keywords and arguments

• Checks are, as much as possible, explicit (specified expected values)

• The automation task focuses on automating the keywords, each keyword is automated only once


• In "low level" tests interaction details are not hidden, since they are the target of the test

• The right level of abstraction depends on the scope of the test, and is an outcome of your test design process

TEST MODULE Screen Flowuser

start system john

TEST CASE TC 01 Order button

window buttonclick main create order

windowcheck window exists new order

FINAL

close application

Example of a low level test


Variables and expressions with keywords

• This test does not need an absolute number for the available cars, just wants to see if a stock is updated

• As a convention we denote an assignment with ">>"• The "#" indicates an expression

TEST CASE TC 02 Rent some more cars

car availableget quantity Chevy Volt >> volt

first name last name carrent car John Green Chevy Voltrent car Jane White Chevy Volt

car expectedcheck quantity Chevy Volt # volt - 2


Data driven testing with keywords

• The test lines will be repeated for each row in the data set• The values represented by "car", "first" and "last" come

from the selected row of the data set

TEST CASE TC 03 Check stocksdata set

use data set /cars

car availableget quantity # car >> quantity

first name last name carrent car # first # last # car

car expectedcheck quantity # car # quantity - 1

repeat for data set

DATA SET carsfirst last carJohn Doe Chevy VoltMary Kane Ford EscapeJane Collins Chrysler 300Tom Anderson Buick VeranoHenry Smyth BMW 750Vivian Major Toyota Corolla


ACTION DEFINITION check balanceuser

argument customerargument amount

window control valueenter balance inquiry last name # customer

window controlclick balance inquiry view balance

window control expectedcheck balance inquiry balance # amount

Re-use actions to make new actions

• In the below example we make a new action• Existing actions are strung together to create new ones with a

broader scope• Often steps in low level tests are re-used to create these action

definitions

:customer amount

check balance Smith 223.45check balance Jones 0.00check balance James -330.45

:

use many times in tests:

define in one place:


Example action implementation in Python

This action script (in Python, Java or C#) verifies whether the rows in a table are sorted:

# get table object, column number and column countwindowName = LIBRARY.NamedArgument("window")tableName = LIBRARY.NamedArgument("table")columnName = LIBRARY.NamedArgument("column")table = ABT.OpenElement(windowName, tableName)column = table.GetColumnIndex(columnName)rowCount = table.GetRowCount()

# check the sort order, row by rowprevious = table.GetCellText(0, column)for i in range(1, rowCount): current = table.GetCellText(i, column) if current < previous : LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0) return previous = current

LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1)

find the table in the UI

if a value is smaller than before, fail the test

if all rows are ascending, pass the test

get arguments from the test line

def action_checkSortOrder():

Note that the approach resembles the Page Object pattern, but in my view is not the same. The abstraction is “any table”, even in non-Web UI’s or non-UI’s altogether. The Page Object pattern could also work on top of this.


Using the new action

• By keeping an action generic it can be applied for a variety of situations

• Some examples of using "check sort order":

window table columncheck sort order view orders orders table ID

window table columncheck sort order annual results regions revenue

window table columncheck sort order inventory cars price

window table columncheck sort order registration students last name


Behavior Driven Development (BDD)

• Uses natural language scenarios

• Tools like JBehave and Cucumber map these to code

• Structure is "Given-When-Then" (GWT)

• Example:

Given a customer previously bought a black sweater from me

And I currently have three black sweaters left in stock

When he returns the sweater for a refund

Then I should have four black sweaters in stock(source: Wikipedia)


BDD with keywords (example)GIVEN A customer previously bought a black sweater from me

first last id

add client John Jones >> client

AND I currently have three black sweaters left in stock

article color quantity

get stock sweater black >> sweaters

WHEN He returns the sweater for a refund

customer article color

return article # client sweater black

THEN I should have four black sweaters in stock

article color quantity

check stock sweaters black # sweaters + 1


Equivalence, conversion

Given a customer previously bought a black sweater from me

And I currently have three black sweaters left in stock

When he returns the sweater for a refund

Then I should have four black sweaters in stock

customer buys, article, colora customer previously bought a {color} {article} from me

set stock, article, color, amountI currently have {amount} {color} {article} left in stockmy stock of {color} {sweater} is {amount}

return article, article, colorhe returns the {color} {article} for a refund

check stock, article, colorI should have four {color} {article} in stock

article color

customer buys sweater blackarticle color amount

set stock sweater black 3article color

return article sweater blackarticle color amount

check stock sweater black 4 ?

Test Module Sections

Test Module

Objectives

Initial - setup

Test cases

Final - cleanup

High Level Test Design - Test Development Plan

Objectives

Test Module 1

Test Cases

Test Module 2 Test Module N

Actions

. . .

AUTOMATION

Objectives Objectives

interaction test business test

define the "chapters"

create the "chapters"

create the "words"

make the words work

Test Cases Test Cases

window control valueenter log in user name jdoeenter log in password car guy

window control property expectedcheck property log in ok button enabled true

user passwordlog in jdoe car guy

first last brand modelenter rental Mary Renter Ford Escape

last totalcheck bill Renter 140.42

Action Based Testing


Organize the tests

• What tests would you create for this application?• How would you organize them?• What, if anything, would you avoid?


Why Better Test Design?

• Better test design can improve quality of test – many tests are often quite "mechanical" now, no surprises– one to one related to specifications, user stories or requirements,

which often is ok, but lacks aggression– no combinations, no unexpected situations, lame and boring– such tests have a hard time finding (interesting) bugs

• Better test design can give (much) better automation– when unneeded details are left out of tests, those details don't have

to be maintained– avoiding "over checking": creating checks that are not in the scope

of a test, but may fail after system changes– limit the impact of system changes on tests, making such impact

more manageable

I have become to believe that successful automation is usually less of a technical challenge as it is a test design challenge.


What's the trick...


• Business objects and business flows– objects are like cars, invoices, locations, etc– flows are like create, fulfill, pay and close an order

• Other tests– functions and features, like premium calculation or PDF output– administration, users, security, authorizations– graphics– technologies, protocols, ...– customization, extensibility– . . .

• Business versus interaction– differentiate within business objects and other categories– interaction can be further differentiated into: values, UI, keyboard,

etc

Example of considerations


Example Top Level StructureProject create, update, delete/close

copy, movecategorize, enumerate, identifyconvert, serialize, export/import, ...

UI, dialogs, forms, pagesinput (validation, defaulting, dependencies)flows (primary paths, alternate paths)keyboard shortcuts, keyboard controls, ...

. . .

<business object 1>

Lifecycles, data operations

Interaction

Functions and Features

Technologies, protocols, controls

Data (handling, quality, ETL, ...)

Security, authorization, admin

Graphics, multi-media, charts, ...

Load, performance

Integration

Customizing, extensibility

Business Flows

Concurrency, race conditions, ...

Business Objects

processes, transactions, end-to-end, day in the life,combinations of flows, ...

calculations, analyses, PDF output, ...

Repeat for eachcomponent or sub-system


Example: E-commerce site

• Articles• Categories• Customers• Promotions• Orders• Invoices• Payments• Credit cards• Staff members• Countries• . . .


• Promotion kinds– percentage discounts– fixed dollar discounts– comprehensive promotions– regional promotions– volume discounts– prompt payment discounts– article based, comprehensive– . . .

• Life cycles– creation– modification

• various properties• before or after activation

– expiration– cancellation

Promotions -- business tests


Promotions -- interaction tests

• Screen by screen• Discount percentage/fixed

– does the amount field change

• Regional checkbox– does the region list show up– are the regions correct– can the regions be selected/deselected

• Start, duration, end date– are the date checks working– can either duration or end date be entered

• Articles, categories– can categories be selected, do they show up– do the right articles show up, can they be selected

• . . .


What tests could look like

Business tests for business object "Promotions"

Interaction tests for business object "Promotions"

window controlclick main new promotion

window typeselect promotion town

window list valuecheck list item exists promotion towns Tietjerksteradeel

name start finish percentage categorycreate promotion christmas 12/1/2016 12/25/2016 11 allcreate promotion tablets 12/20/2016 1/1/2017 20 tablets

datetime travel 12/23/2016

article pricecheck nett price iPad Mini 4 284.79


Example 2: a testing company in California

• The company sells services like test development and automation

• It also has consultancy and training offerings• And it has a product for test development and

automation• These various offerings are often combined in the orders

they receive• Invoicing for services as monthly• They're working on an internal system for order handling

and invoicing


Example 2: a testing company in California

• Example business objects:– projects, orders ("SOW's"), customers, employees, invoices,

payments

• Example business flows:– customer place orders, often multiple ones over time– one order can have multiple items– tools are invoiced (and delivered) immediately– services are delivered over time– staff members write time-sheets with hours spent

• has elaborate UI's and processes– invoices can combine items from multiple orders– payments can cover multiple invoices


What about existing tests?

• Compare to moving house:– some effort can't be avoided– be selective, edit your stuff,

• look at the future, not the past– first decide where to put what, then put it there– moving is an opportunity, you may not get such chance again soon

• Follow the module approach– define the modules and their scope as if from scratch– use the existing test cases in two ways:

• verify completeness• harvest and re-use them for tests and for actions

– avoid porting over "step by step", in particular avoid over-checking


Eye on the ball, Scope

• Always know the scope of the test module

• The scope should be unambiguous

• The scope determines many things:– what the test objectives are– which test cases to expect– what level of actions to use– what the checks are about and which events should generate a

warning or error (if a “lower” functionality is wrong)


Use the right level actions

Low level of UI interaction detail makes sense only with the module scope is to test the UI

Better to have a business level action and hide the details in an action definition:

But do show details when it matters. The example below is too high level, requires drill down into action definition to understand what is tested.


Example of a Test

Step Description Expected

step 16 Open http://www.bigstore.com The "BIG Store" main page is displayed, with a "sign in" link

step 17 Click on "Sign In", upper right corner A sign in dialog shows, the "Sign in" button is disabled

step 18 Enter "johnd" in the user name field The "Sign In" button is still disabled

step 19 Enter "bigtester" in the password field Now the "Sign In" button is enabled

step 20 Click on the "Sign in" button The page now shows "Hello John" in the upper right corner

step 21 Enter "acme watch" in the search field The "Search" button is enabled

step 22 Click on the "Search" button 5 watches of Acme Corporation are displayed

step 23 Double click on "Acme Super Watch 2" The details page of the Acme Super Watch 2 is displayed

step 24 Verify the picture of the watch The picture should show a black Acme Super Watch 2

step 25 Select "red" in the "Color" dropdown list The picture now shows a black Acme Super Watch 2

step 26 Type 2 in the "Order quantity" textbox The price in the right shows "$79.00 + Free Shipping"

step 27 Click on "Add to cart" The status panel shows "Acme Super Watch 2" added

step 28 Click on "Check out" The Cart Check Out open, with the 2 Acme Super Watches

is this a good test? is it good for automation?


GWT scenario

Given User turns off Password required option for Drive TestAnd User has logged in by Traffic Applicant accountAnd User is at the Assessments Take a Test pageAnd User clicks the Traffic Test linkAnd User clicks the Next buttonAnd User clicks the Sheet radio button in Mode page if displayedAnd User clicks the Start buttonAnd User waits for test startAnd User clicks the Stop Test buttonWhen User clicks the Confirm Stop Test buttonAnd User enters the correct applicant passwordAnd User clicks the Confirm Stop Test buttonThen The Test is Over should be displayed in the Message labelAnd the value of the Message label should be The test is overAnd The Welcome to Traffic Testing page should be displayed


"Anti-patterns" (informal)

• Interaction Heavy – Not having many business tests• Lame – No depth or variety, no testing techniques used• Enter Enter Click Click – Test steps are too detailed • No Life – Missing life cycle steps of business objects• Clueless – No clear scope for the tests• Cocktail – Interaction tests mixed with business tests• Over-Checking – Checks not relevant for the scope• Sneaky Checking – Checks hidden in actions• Action Explosion – Many actions, hardly different• Mystery Actions – Cryptic actions, unclear what they do• Techno – Actions and tests that look like code

– often _NOts0EasY_2REad – but great to impress non-technical people


Example of a test module

TEST MODULE Order processing

start system

TEST CASE TC 01 Order for tablets

user passwordlogin jdoe doedoe

windowcheck window exists welcome

order id cust id article price quantitycreate order AB123 W3454X tablet 198.95 5

order id totalcheck order total AB123 994.75 . . .


Low-level, high-level, mid-level actions

• Low-level (system level) actions– detailed interaction with the UI (or API)– generic, do not show any functional or business logic– examples: "click", "expand tree node", "select menu"

• High-level (application and business level) actions– represent a business function or event fitting the scope of the test– hides the interaction– examples: "enter customer", "rent car", "check balance"

• Mid-level actions: auxiliary actions that represent common sequences of low level actions

– usually to wrap a page or dialog– enhance maintainability– example: "enter address fields“– complex pages could be further split up

like with the Page Object pattern

enter customer

enter address fields

enter select set . . .. . .

Tip: Provide default values in actions

ACTION DEFINITION login

name default valueargument user testerargument password testerpw

window control valueenter login window user name # userenter login window password # password

window controlclick login window login

user passwordlogin tamaraj tj1234

textcheck message Hello Tamara

logindate payee amount

make payment 1/1/12 Gas Co. 85.00

Use login action w/ arguments Use login default values


ABT in Agile

Test ModuleDefinition(optional)

Test Module Development

Interface Definition

Action Automation

Test Execution

Sprint ProductsProduct Backlog

Test re-use

Automation re-use

product owner team prod owner

& team

User storiesDocumentation

Domain understanding

Acceptance CriteriaPO Questions

SituationsRelations

Agile life cycle

Test development

Main Level Test Modules

Interaction Test Modules

Cross over Test Modules


Using ABT in Sprints (1)

• Try to keep the testers in the same sprint as the rest of the team– don't let automated tests clutter up and lack behind– note that in many environments tests and their automation are not

highest priority

• Agree on the approach:– is testability a requirement for the software?– do we regard tests, or some of the tests, as products?

• would potentially become part of the backlog• would not only be up to the team which tests to create

• Just like for development, use discussions with the team and product owners – deepen understanding, for the whole team– help identify items like negative, alternate and unexpected situations– tests can work as driver for development (TDD and ATDD)


Using ABT in Sprints (2)

• Create good starting conditions for a sprint:– automation technology available (UI, non-UI, custom controls, graphics, ...)– know how you will deal with data and environments– understanding of subject matter, testing, automation, etc

• Do interface mapping by hand, using developer provided identifications

– saves time by not having to use the viewer or other spy tools– recording of actions (not tests) will go better

To discuss an approach, consider daily "sit down" meetings with some or all members to coach and evaluate- an end-of-day counterpart to the early-morning "stand up" meetings- short and friendly, not about progress and impediments, but about practices and

experiences with them (like "what actions did you use?")- a few meetings may suffice

TIP

Multiple System Access

SystemUnder Test

Action Automation

Web serviceaccess

commandline access

Web UIaccess

databaseaccess

Test Modules


Testing in, or near, production

• In an agile and DevOps approach continuous deployment is becoming the norm

– in particular for service based system (like web, apps, SaaS, but also client-server)– logic in services is easier to manage and updated than distributed and variable clients

• In a DevOps approach tight cooperation between development, test and deployment pays off

– automation testability– testable system designs– good test design remains a success factor – tight integration in the build, deployment and production processes

• A/B testing can help test vary complex systems– used not only for testing, also for trying out new features– dividing the incoming traffic into an A and B flow (B is the new situation)– automated tests can use the B flow– to do this, have it well integrated in your system designs

• Continuous testing with random regression testing or monkey testing

*see also: Ken Johnston's chapter in the book of Dorothy Graham and Mark Fewster, and his keynote at StarWest 2012


A/B testing with a reverse proxy

• A/B testing means part of traffic is routed through a different server or component (see if it works, and/or how users react)

• B could be a real-life user, a test user or also an automated test• The strategy can be done at any component level• Watch your test design, easy to drown in technical only

A

A

B

ReverseProxy

UsersServers

A

B

newcurrent

AB


• Passive timing– wait a set amount of time– in large scale testing, try to avoid passive timing altogether:

• if wait too short, test will be interrupted• if wait too long, time is wasted

• Active timing– wait for a measurable event– usually the wait is up to a, generous, maximum time– common example: wait for a window or control to appear (usually the test tool will do this for you)

• Even if not obvious, find something to wait for...

• Involve developers if needed– relatively easy in an agile team, but also in traditional projects, give this priority

• If using a waiting loop– make sure to use a "sleep" function in each cycle that frees up the processor (giving the AUT time

to respond)– wait for an end time, rather then a set amount of cycles

Active Timing


Things to wait for...

• Wait for a last control or elements to load– developers can help knowing which one that is

• Non-UI criteria– API function– existence of a file

• Criteria added in development specifically for this purpose, like:– "disabling" big slow controls (like lists or trees) until they're done loading– API functions or UI window or control properties

• Use a "delta" approach:– every wait cycle, test if there was a change; if no change, assume that the loading time is

over:– examples of changes:

• the controls on a window• count of items in a list• size a file (like a log file)


• Should be a "must have" requirement– first question in a development project: "how do we test this?"

• Design of the system(s) under test:– components, tiers, services,

• Hidden identifying properties

• Hooks for timing

• White-box access to anything relevant:– input data (ability to emulate)– output data (what is underlying data being displayed)– random generators (can I set a seed?)– states (like in a game)– objects displayed (like monsters in a game)

• Emulation features, like time-travel and fake locations

Testability, key items


Versions, environments, configurations

• Many factors can influence details of automation– language, localization– hardware– version of the system under test– system components, like OS or browser

• Test design can reflect these– certain test modules are more general– others are specific, for example for a language

• But for tests that do not care about the differences, the automation just needs to "deal" with them– shield them from the tests

minimum safe distance from a bear is 91 meters

LOCALIZATIONconverting yards to meters

Capture variations of the system under test in the actions and interface definitions, rather than in the tests (unless relevant there).

Variation Variation Variation

"Variations"

"Master Switch"

Actions, Interface Definitions

. . .

Possible set up of variations

linked variation

keyworded variation

Specify for example in a dialog when you start an execution:


• Not every test is straightforward UI based

• Always go back to the test: what is it I want

• Express that with actions– make them up as you go– assume everything is possible

• For the automation side, ask for advise– you pattern might have been seen before

Special situations


"Lead Deputy" Testing

• For "multi station" testing, when multiple machines have to participate in a test in real-time

• For example if a supervisor needs to approve a withdrawal in a bank teller system

• Can be "synchronized" and "parallel" (with a rendezvous point)

lead machine playing the "bank teller"

deputy machineplaying the"supervisor"

acc nr amount tx id

request 123123 10,000 >> txname

use deputy supervisortx id amount

approve # tx 10,000

use leadtx id amount

pay out # tx 10,000

lead machine playing the "bank teller"

...

...

Device Testing

Software Under Test

AgentABTAutomation

InterfaceInfo

Testing HostDevice

Android


• Approach applicable for pictures like graphics or icons

• The tester will add a line "check picture", that includes a "question" as one of the arguments

• While the test is executed TA keeps the recorded pictures

• After execution the pictures are shown to a manual testing for approval

• Once approved unchanged same pictures won't be presented to the manual tester again in future runs

Multimedia: The "check picture" Approach


• Automation is not as much a technical challenge as it is a test design challenge

• Domain language based approaches like BDD and Actions can make tests are accessible, and easier to maintain

• However, some kind of systematic approach, like the ABT method, is necessary to keep the tests manageable

• Testing is teamwork, including automation engineers, developers and product owners

• From a test design perspective approaches for non-UI and UI driven testing are very similar

Summary [email protected]

That’s all folks for this talk. However, I have much more stuff, please let me know any questions.


• Automation is not as much a technical challenge as it is a test design challenge

• Domain language based approaches like BDD and Actions can make tests are accessible, and easier to maintain

• However, some kind of systematic approach, like the ABT method, is necessary to keep the tests manageable

• From a test design perspective approaches for non-UI and UI driven testing are very similar

Summary [email protected]

That’s all folks for this talk. However, I have much more stuff, please let me know any questions.

you have selenium... now what?

Technology