integration testing with selenium

32
Integration Testing with Selenium Hans Buwalda ©2015 LogiGear Corporation www.logigear.com www.testarchitect.com www.happytester.com @hansbuwalda [email protected] [email protected]

Upload: all-things-open

Post on 15-Apr-2017

540 views

Category:

Technology


1 download

TRANSCRIPT

Page 1: Integration Testing with Selenium

©2015 LogiGear Corporation

Integration Testing with Selenium

Hans Buwalda

www.logigear.comwww.testarchitect.comwww.happytester.com

@[email protected]@happytester.com

Page 2: Integration Testing with Selenium

©2015 LogiGear Corporation

• Topics:– position of functional/integration testing– domain language approaches– test design as driver for automation– automation friendly test design techniques and

examples– special cases: data driven, multi-station, graphics, etc

• What this talk is not:– specific to Selenium only– a technical class on automation– about component based development– introduction to unit testing

Scope

Page 3: Integration Testing with Selenium

©2015 LogiGear Corporation

• A combination of open source products and standards, aimed at web testing (including a Firefox based IDE)

• In the later versions it works with an interface called “WebDriver”, implemented with drivers in browsers

• WebDriver has been implemented in open source libraries, but is now also becoming implemented in browsers

• For example Microsoft Edge can only be accessed programmatically via WebDriver

• Products like Appium try to implement WebDriver for mobile browsing and even mobile apps

• Our product (TestArchitect) is moving to WebDriver, expect this to be a common practice

Selenium

Page 4: Integration Testing with Selenium

©2015 LogiGear Corporation

Relation to code Quality / depth Automation Scalability

Unit TestingClose relationship with the code

Singular test scope, but deep into the code

Fully automated by nature

Scalable, grows with the code, easy to repeat

Functional/Integration Testing

Usually does not have a one-on-one relation with code

Quality and scope depends on test design

In particular UI based automation can be a challenge

Often a bottle-neck in scalability

Exploratory Testing

Human driven, not seeking a relation with code

Usually deep and thorough, good at finding problems

May or may not be automated afterwards

Not meant to be repeatable. Rather do a new session

Some test kinds and their scalability (simplified)

Page 5: Integration Testing with Selenium

©2015 LogiGear Corporation

Example Scripting: Java + Selenium

WebElement element = driver.findElement(By.name(name));

element.sendKeys("mystery magic");element.submit();

(new WebDriverWait(driver, 10)).until(new ExpectedCondition<Boolean>() {

public Boolean apply(WebDriver d) {return d.getTitle().toLowerCase().startsWith("mystery");

}}

);

System.out.println(driver.getTitle());

Page 6: Integration Testing with Selenium

©2015 LogiGear Corporation

Domain language approaches: Actions

4 actions, each with an action keyword and arguments

read from top to bottom

fragment from a test with actions

acc nr first last

open account 123123 John Doe

acc nr amount

deposit 123123 10.11deposit 123123 20.22

acc nr expected

check balance 123123 30.33

• Similar to the Page Object pattern, but stays out of the programming language

• The test developer creates tests using actions with keywords and arguments

• Checks are, as much as possible, explicit (specified expected values)

• The automation task focuses on automating the keywords, each keyword is automated only once

Page 7: Integration Testing with Selenium

©2015 LogiGear Corporation

Behavior Driven Development (BDD)

• Uses natural language scenarios

• Tools like JBehave and Cucumber map these to code

• Structure is "Given-When-Then" (GWT)

• Example:

Given a customer previously bought a black sweater from me

And I currently have three black sweaters left in stock

When he returns the sweater for a refund

Then I should have four black sweaters in stock(source: Wikipedia)

Page 8: Integration Testing with Selenium

©2015 LogiGear Corporation

BDD with keywords (example)GIVEN A customer previously bought a black sweater from me

first last id

add client John Jones >> client

AND I currently have three black sweaters left in stock

article color quantity

get stock sweater black >> sweaters

WHEN He returns the sweater for a refund

customer article color

return article # client sweater black

THEN I should have four black sweaters in stock

article color quantity

check stock sweaters black # sweaters + 1

Page 9: Integration Testing with Selenium

©2015 LogiGear Corporation

Equivalence, conversion

Given a customer previously bought a black sweater from me

And I currently have three black sweaters left in stock

When he returns the sweater for a refund

Then I should have four black sweaters in stockcustomer buys, article, color

a customer previously bought a {color} {article} from me

set stock, article, color, amountI currently have {amount} {color} {article} left in stockmy stock of {color} {sweater} is {amount}

return article, article, colorhe returns the {color} {article} for a refund

check stock, article, colorI should have four {color} {article} in stock

article color

customer buys sweater blackarticle color amount

set stock sweater black 3article color

return article sweater blackarticle color amount

check stock sweater black 4 ?

Page 10: Integration Testing with Selenium

Test Module Sections

Test Module

Objectives

Initial - setup

Test cases

Final - cleanup

Page 11: Integration Testing with Selenium

©2015 LogiGear Corporation

• In "low level" tests interaction details are not hidden, since they are the target of the test

• The right level of abstraction depends on the scope of the test, and is an outcome of your test design process

TEST MODULE Screen Flowuser

start system john

TEST CASE TC 01 Order button

window buttonclick main create order

windowcheck window exists new order

FINAL

close application

Example of a low level test

Page 12: Integration Testing with Selenium

©2015 LogiGear Corporation

Variables and expressions with keywords

• This test does not need an absolute number for the available cars, just wants to see if a stock is updated

• As a convention we denote an assignment with ">>"• The "#" indicates an expression

TEST CASE TC 02 Rent some more cars

car availableget quantity Chevy Volt >> volt

first name last name carrent car John Green Chevy Voltrent car Jane White Chevy Volt

car expectedcheck quantity Chevy Volt # volt - 2

Page 13: Integration Testing with Selenium

©2015 LogiGear Corporation

Data driven testing with keywords

• The test lines will be repeated for each row in the data set• The values represented by "car", "first" and "last" come

from the selected row of the data set

TEST CASE TC 03 Check stocksdata set

use data set /cars

car availableget quantity # car >> quantity

first name last name carrent car # first # last # car

car expectedcheck quantity # car # quantity - 1

repeat for data set

DATA SET carsfirst last carJohn Doe Chevy VoltMary Kane Ford EscapeJane Collins Chrysler 300Tom Anderson Buick VeranoHenry Smyth BMW 750Vivian Major Toyota Corolla

Page 14: Integration Testing with Selenium

©2015 LogiGear Corporation

ACTION DEFINITION check balanceuser

argument customerargument amount

window control valueenter balance inquiry last name # customer

window controlclick balance inquiry view balance

window control expectedcheck balance inquiry balance # amount

Re-use actions to make new actions

• In the below example we make a new action• Existing actions are strung together to create new ones with a

broader scope• Often steps in low level tests are re-used to create these action

definitions

:customer amount

check balance Smith 223.45check balance Jones 0.00check balance James -330.45

:

use many times in tests:

define in one place:

Page 15: Integration Testing with Selenium

©2015 LogiGear Corporation

"Lead Deputy" Testing

• For "multi station" testing, when multiple machines have to participate in a test in real-time

• For example if a supervisor needs to approve a withdrawal in a bank teller system

• Can be "synchronized" and "parallel" (with a rendezvous point)

lead machine playing the "bank teller"

deputy machineplaying the"supervisor"

acc nr amount tx id

request 123123 10,000 >> txname

use deputy supervisortx id amount

approve # tx 10,000

use leadtx id amount

pay out # tx 10,000

lead machine playing the "bank teller"

...

...

Page 16: Integration Testing with Selenium

High Level Test Design

Tests

Objectives

Test Module 1

Actions

. . .

AUTOMATION

user passwordlog in jdoe StarEast

first last brand modelrent car John Renter Ford Escaperent car John Renter Chevrolet Volt

last totalcheck bill Renter 140.42

interaction test business testwindow control value

enter log in user name jdoeenter log in password StarEast

window control property expectedcheck prop log in ok button enabled true

car guycar guy

Action Based Testing

Tests

Objectives

Test Module 2

Tests

Objectives

Test Module n

Page 17: Integration Testing with Selenium

©2015 LogiGear Corporation

What's the trick...

Page 18: Integration Testing with Selenium

©2015 LogiGear Corporation

• Business objects and business flows– objects are like cars, invoices, locations, etc– flows are like create, fulfill, pay and close an order

• Other tests– functions and features, like premium calculation or PDF output– administration, users, security, authorizations– graphics– technologies, protocols, ...– customization, extensibility– . . .

• Business versus interaction– differentiate within business objects and other categories– interaction can be further differentiated into: values, UI, keyboard,

etc

Example of considerations

Page 19: Integration Testing with Selenium

©2015 LogiGear Corporation

Example Top Level StructureProject create, update, delete/close

copy, movecategorize, enumerate, identifyconvert, serialize, export/import, ...

UI, dialogs, forms, pagesinput (validation, defaulting, dependencies)flows (primary paths, alternate paths)keyboard shortcuts, keyboard controls, ...

. . .

<business object 1>

Lifecycles, data operations

Interaction

Functions and Features

Technologies, protocols, controls

Data (handling, quality, ETL, ...)

Security, authorization, admin

Graphics, multi-media, charts, ...

Load, performance

Integration

Customizing, extensibility

Business Flows

Concurrency, race conditions, ...

Business Objects

processes, transactions, end-to-end, day in the life,combinations of flows, ...

calculations, analyses, PDF output, ...

Repeat for eachcomponent or sub-system

Page 20: Integration Testing with Selenium

©2015 LogiGear Corporation

What about existing tests?

• Compare to moving house:– some effort can't be avoided– be selective, edit your stuff,

• look at the future, not the past– first decide where to put what, then put it there– moving is an opportunity, you may not get such chance again soon

• Follow the module approach– define the modules and their scope as if from scratch– use the existing test cases in two ways:

• verify completeness• harvest and re-use them for tests and for actions

– avoid porting over "step by step", in particular avoid over-checking

Page 21: Integration Testing with Selenium

©2015 LogiGear Corporation

Eye on the ball, Scope

• Always know the scope of the test module

• The scope should be unambiguous

• The scope determines many things:– what the test objectives are– which test cases to expect– what level of actions to use– what the checks are about and which events should generate a

warning or error (if a “lower” functionality is wrong)

Page 22: Integration Testing with Selenium

©2015 LogiGear Corporation

Example of a Test

Step Description Expected

step 16 Open http://www.bigstore.com The "BIG Store" main page is displayed, with a "sign in" link

step 17 Click on "Sign In", upper right corner A sign in dialog shows, the "Sign in" button is disabled

step 18 Enter "johnd" in the user name field The "Sign In" button is still disabled

step 19 Enter "bigtester" in the password field Now the "Sign In" button is enabled

step 20 Click on the "Sign in" button The page now shows "Hello John" in the upper right corner

step 21 Enter "acme watch" in the search field The "Search" button is enabled

step 22 Click on the "Search" button 5 watches of Acme Corporation are displayed

step 23 Double click on "Acme Super Watch 2" The details page of the Acme Super Watch 2 is displayed

step 24 Verify the picture of the watch The picture should show a black Acme Super Watch 2

step 25 Select "red" in the "Color" dropdown list The picture now shows a black Acme Super Watch 2

step 26 Type 2 in the "Order quantity" textbox The price in the right shows "$79.00 + Free Shipping"

step 27 Click on "Add to cart" The status panel shows "Acme Super Watch 2" added

step 28 Click on "Check out" The Cart Check Out open, with the 2 Acme Super Watches

is this a good test? is it good for automation?

Page 23: Integration Testing with Selenium

©2015 LogiGear Corporation

Use the right level actions

Low level of UI interaction detail makes sense only with the module scope is to test the UI

Better to have a business level action and hide the details in an action definition:

But do show details when it matters. The example below is too high level, requires drill down into action definition to understand what is tested.

Page 24: Integration Testing with Selenium

©2015 LogiGear Corporation

Example of a test module

TEST MODULE Order processing

start system

TEST CASE TC 01 Order for tablets

user passwordlogin jdoe doedoe

windowcheck window exists welcome

order id cust id article price quantitycreate order AB123 W3454X tablet 198.95 5

order id totalcheck order total AB123 994.75 . . .

Page 25: Integration Testing with Selenium

©2015 LogiGear Corporation

Example action implementation in Python

This action script (in Python, Java or C#) verifies whether the rows in a table are sorted:

# get table object, column number and column countwindowName = LIBRARY.NamedArgument("window")tableName = LIBRARY.NamedArgument("table")columnName = LIBRARY.NamedArgument("column")table = ABT.OpenElement(windowName, tableName)column = table.GetColumnIndex(columnName)rowCount = table.GetRowCount()

# check the sort order, row by rowprevious = table.GetCellText(0, column)for i in range(1, rowCount): current = table.GetCellText(i, column) if current < previous : LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0) return previous = current

LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1)

find the table in the UI

if a value is smaller than before, fail the test

if all rows are ascending, pass the test

get arguments from the test line

def action_checkSortOrder():

Note that the approach resembles the Page Object pattern, but in my view is not the same. The abstraction is “any table”, even in non-Web UI’s or non-UI’s altogether. The Page Object pattern could also work on top of this.

Page 26: Integration Testing with Selenium

©2015 LogiGear Corporation

Using the new action

• By keeping an action generic it can be applied for a variety of situations

• Some examples of using "check sort order":

window table columncheck sort order view orders orders table ID

window table columncheck sort order annual results regions revenue

window table columncheck sort order inventory cars price

window table columncheck sort order registration students last name

Page 27: Integration Testing with Selenium

©2015 LogiGear Corporation

Low-level, high-level, mid-level actions

• Low-level (system level) actions– detailed interaction with the UI (or API)– generic, do not show any functional or business logic– examples: "click", "expand tree node", "select menu"

• High-level (application and business level) actions– represent a business function or event fitting the scope of the test– hides the interaction– examples: "enter customer", "rent car", "check balance"

• Mid-level actions: auxiliary actions that represent common sequences of low level actions

– usually to wrap a page or dialog– enhance maintainability– example: "enter address fields“– complex pages could be further split up

like with the Page Object pattern

enter customer

enter address fields

enter select set . . .. . .

Page 28: Integration Testing with Selenium

Tip: Provide default values in actions

ACTION DEFINITION login

name default valueargument user testerargument password testerpw

window control valueenter login window user name # userenter login window password # password

window controlclick login window login

user passwordlogin tamaraj tj1234

textcheck message Hello Tamara

logindate payee amount

make payment 1/1/12 Gas Co. 85.00

Use login action w/ arguments Use login default values

Page 29: Integration Testing with Selenium

Multiple System Access

SystemUnder Test

Action Automation

APIaccess

protocolaccess

UIaccess

databaseaccess

Test Modules

Page 30: Integration Testing with Selenium

Device Testing

Software Under Test

AgentABTAutomation

InterfaceInfo

Testing HostDevice

Android

Page 31: Integration Testing with Selenium

©2015 LogiGear Corporation

• Approach applicable for pictures like graphics or icons

• The tester will add a line "check picture", that includes a "question" as one of the arguments

• While the test is executed TA keeps the recorded pictures

• After execution the pictures are shown to a manual testing for approval

• Once approved unchanged same pictures won't be presented to the manual tester again in future runs

Multimedia: The "check picture" Approach

Page 32: Integration Testing with Selenium

©2015 LogiGear Corporation

• Automation is not as much a technical challenge as it is a test design challenge

• Domain language based approaches like BDD and Actions can make tests are accessible, and easier to maintain

• However, some kind of systematic approach, like the ABT method, is necessary to keep the tests manageable

• From a test design perspective approaches for non-UI and UI driven testing are very similar

Summary [email protected]

That’s all folks for this talk. However, I have much more stuff, please let me know any questions.