How to Make Your UI Tests

Stable, Scalable, and

Maintainable

2

SmartBear’s technology stack

3

Quick Intro

Nikhil Kaul Jeffrey Martin

4

On our plate today

1

Why you shouldn't tolerate flakiness?

2

Common reasons for flaky UI tests

4

What to do with flaky UI tests?

5

How to separate signal from noise?

3

How to overcome these challenges?

5

Flaky tests shouldn't be tolerated

• 300 tests

• Each test has .5% failure rate

• Pass rate for each tests: 99.5%

• Pass rate for the suite: (99.5%)^300 = 22.23%

6

Understand the nature of flakiness

Tech Complexities

Test Enviroment

Test Data

Test Design

7

Understand the nature of flakiness

Tech Complexities

Test Enviroment

Test Data

Test Design

8

Don’t interact with UI elements directly

Test services seperately from the UI

Use page objects

Automate at three levels

‒ Unit

‒ Service

‒ UI

Use UI encapsulation Build modular test design

‒ Identify user journeys

‒ Unit

‒ Service

‒ UI

Automate at three levels Use UI encapsulation

‒ Page Objects

UI Service Unit

9

How to implement in real world

Add Item

Click on an item with name XYZ Clickitem()

Set element with name ABC to quantity Z Set element()

Click on button labeled place in cart clickbutton()

What? How?

10

Understand the nature of flakiness

Tech Complexities

Test Enviroment

Test Data

Test Design

11

Separate the test data from test design

Test services seperately from the UI

Use page objects

Automate at three levels

‒ Unit

‒ Service

‒ UI

Use page objects Build modular test design

– Test specific data

– Reference data

Limit data

dependency

between tests Tests should run

independently

Using databases

‒ Conduct tests

inside a transaction

12

Understand the nature of flakiness

Tech Complexities

Test Enviroment

Test Data

Test Design

13

Limit your dependency on environments

Test services seperately from the UI

Use page objects

Automate at three levels

‒ Unit

‒ Service

‒ UI

Use page objects Build modular test design

‒ Application API

‒ HTML API

‒ Identify user journeys

Limit calls to external systems

Parameterize connections

Hermetic Servers

14

Understand the nature of flakiness

Tech Complexities

Test Enviroment

Test Data

Test Design

15

Automate at three levels

‒ Unit

‒ Service

‒ UI

Use page objects

Build modular test design ‒ Application API

‒ HTML API

Limit calls to external systems

Parameterize connections

Hermetic Servers

Technical complexities can be overcome

AJAX Calls

‒ Callback

‒ Polling Interval

‒ WaitUntil

No Response Calls

‒ Fake objects

‒ Mock

‒ Virtual Service

16

Acceptance test builds should never be red

Separate flaky

from healthy tests

Limit the number of tests as well as the time

17

Use statistics for measuring test stability

Sample Size (N)

• 50

Desired confidence (α)

• 95% confidence

P

• 29/50 = .58

1-p

• 21/50

𝐵𝑖𝑛𝑜𝑚𝑖𝑎𝑙 𝐶𝐼 = 𝑝 ± z1−α 2

𝑝(1 − p)

n

𝐶𝐼 = .58 ± 1.96.58(1 − .58)

50

𝐶𝐼 = .58 ± .14

𝐶𝐼 = between 44% and 72%.

±z1−α 2 = . ±z.95 = 1.96

18

Takeaways

1

Flaky tests shouldn't be tolerated

2

Understand the nature of flakiness

4

Use statistics for measuring test stability

3

Acceptance test builds should never be red

Questions?