Backstop JSVisual Testing

1

I would define visual testing as using a headless browser to capture and compare images of components of a website.

This can be used to implement unit tests for how the CSS is rendering the HTML in a browser.

Not intended to test how the HTML is generated or replace Unit tests in a back end service layer.

Not intended to replace cross browser testing.

Allows comparison between sites or comparing against a reference image captured form the same site before changes were made.

What do you mean by Visual Testing ?

1. Chrome Headless

2. Setup and Examples

3. What to test

4. Setup in your project

Contents :

Chrome Headless

3

What has changed ? Why is this a big deal ?

4

Headless Chrome is shipping in Chrome 59. It's a way to run the Chrome browser in a headless environment. Essentially, running Chrome without chrome! It brings all modern web platform features provided by Chromium and the Blink rendering engine to the command line.

Why is that useful?

A headless browser is a great tool for automated testing and server environments where you don't need a visible UI shell. For example, you may want to run some tests against a real web page, create a PDF of it, or just inspect how the browser renders an URL.

Announcement blog : https://developers.google.com/web/updates/2017/04/headless-chrome

How to Get it:

As long as you have a version of chrome installed greater than Chrome 59 –( Chrome 61 on Windows) you already have it

Chrome Headless

5

BackstopJS automates visual regression testing of your responsive web UI by comparing DOM screenshots over time.

Backstop is a open source project to run visual tests using headless browsers to capture screenshots.

Originally ran using either PhantomJS or SlimerJS headless browser libraries. These were implementations of WebKit (Chrome) or Gecko (Firefox).

From release v3.0.19 this now supports Chrome headless.

The latest release has a lot of stability improvements v3.0.27

Git Hub URL : https://github.com/garris/BackstopJS

Features :• Allows capture and comparison of screenshots form headless browser• Excellent comparison tool to identify and highlight differences• Allows setup of multiple breakpoints to test responsive sites• Uses simple CSS selectors to identify what to capture• Pre capture scripts can be enabled to be ran

Backstop JS

6

Chrome is based on WebKit but using PhantomJS as a headless browser caused issues making it unusable.

There would be styling issues that would be thrown that would not be present on the site when viewed through chrome.

As visual testing is about how the site renders this causes the tests to be unreliable.

When I investigated implementing visual testing this was the stumbling block each time.

Chrome != Webkit

Setup and Examples

7

8

• backstop init : Set up a new BackstopJS instance -- specify URLs, cookies, screen sizes, DOM selectors, interactions etc. (see examples directory)

• backstop test : BackstopJS creates a set of test screenshots and compares them with your reference screenshots. Any changes show up in a visual report. (Run this after making CSS changes as many times as needed.)

• backstop approve : If the test you ran looks good, then go ahead and approve it. Approving changes will update your reference files with the results from your last test. Future tests are compared against your most recent approved test screenshots.

The BackstopJS workflow

9

backstop init

10

backstop test (initial)

11

backstop approve

12

backstop test

13

{"id": "backstop_default","viewports": [{"label": "phone","width": 320,"height": 480},{"label": "tablet","width": 1024,"height": 768}],"onBeforeScript": "chromy/onBefore.js","onReadyScript": "chromy/onReady.js","scenarios": [

…………],"paths": {"bitmaps_reference": "backstop_data/bitmaps_reference","bitmaps_test": "backstop_data/bitmaps_test","engine_scripts": "backstop_data/engine_scripts","html_report": "backstop_data/html_report","ci_report": "backstop_data/ci_report"},"report": ["browser"],"engine": "chrome","engineFlags": [],"asyncCaptureLimit": 5,"asyncCompareLimit": 50,"debug": false,"debugWindow": false}

Default Configuration

{

"label": "BackstopJS Homepage",

"cookiePath": "backstop_data/engine_scripts/cookies.json",

"url": "https://garris.github.io/BackstopJS/",

"referenceUrl": "",

"readyEvent": "",

"readySelector": "",

"delay": 0,

"hideSelectors": [],

"removeSelectors": [],

"hoverSelector": "",

"clickSelector": "",

"postInteractionWait": "",

"selectors": [],

"selectorExpansion": true,

"misMatchThreshold" : 0.1,

"requireSameDimensions": true

}

14

On the CIPD PM build we have incorporated it in to the HTML dev process.

When components are created a new test scenario is added to the backstop tests.

This captures the components from the locally setup demo site.

This creates a reference image for that component that is then checked in to source control.

Project Setup

What to Test

15

16

Visual regression testing is the process of creating reference images for each component as they are created. This allows a comparison over time to monitor changes.

Ran by the developer on their local development environment after changes made to ensure no regressions issues have occurred due to their changes.

Why is that useful?

• This allows the developer greater test coverage than they could do manually when ensuring their changes have not had regression impact on other components.

• Allows greater detail comparison than possible reviewing the site manually. E.g. If the font size is increased by 1px then it will be caught which could easily be missed.

• Allows developers without a full understanding of the project have a safety net about what the site should look like before their changes.

Visual Regression Testing

17

Using visual testing to compare different environments involves comparing one site against another . The reference images are not stored beyond the test.

A reference image set is generated form one site and then another site is compared against them

Why is that useful?

• To ensure that the HTML created has been integrated within the website correctly and is rendering out in the same way. This highlights mistakes with the integration or conflicting integrations that are not present on the HTML Demo site.

• Running regression tests against the Staging and Live instances of a site to see where the changes have occurred.

• When setting up comparison between a demo and dev site powered by a CMS this can make the tests brittle so the tests pages need to be isolated from changes in content as possible.

Environment Comparison

18

Within a site their are hover states and interactive components. These need the mouse or user interactions to trigger them.

How to test these in a headless browser?

Backstop allows scripts to be ran before a screenshot is captured. Using the chromy API you can move the mouse and manipulate the page through script.

This allows testing of :

• Field entry• Hover states• Interactive items• Video loading – Playing• Ajax content loading

Chromy Git Hub URL : https://github.com/OnetapInc/chromy

Interacting with the site

19

When capturing items it is much quicker to capture components on a page than the whole page to compare them

• It is much quicker to compare the smaller images

• It is more relevant the items that are being tested.

This allows the components themselves to be bested in isolation and most of our development work is done using component based development.

Smaller is better

Setup in your project

20

Process to include in you project

21

Install Backstop

init within you project Create

scenarios

Run Baseline generation

Keep running and

monitoring

22

Prerequisites

• Node – (If you don’t already have it)• Chrome - (61 needed but latest is 62.0.3202.62 at the moment)• Update node to v8+• Update npm to v5+

On Windows you will also need to have python installed

> PhantomJS needs Python -- please make sure you have Python installed... see https://github.com/garris/BackstopJS/issues/185

Installation

• npm install -g backstopjs

• Installs backstop globally

This allows the backstop command to be ran from anywhere on the PC.

Install Backstop - Globally

23

Backstop can also be installed as a grunt dependency and called through grunt.

var backstopjs = require('backstopjs');grunt.registerTask('backstop', 'BackstopJS integration', function(cmd) {

// cmd is either 'reference', 'test', or 'openReport'var done = this.async();backstopjs(cmd).then(function() {

done(true);}).catch(function() {

done(false);});

});

This allows it to be integrated in to the existing grunt tasks.

Including it in grunt also allows the dynamic creation of the scenarios using variables instead of them being held in a JSON file.

Installing Backtop in Grunt

24

There is a basic backstop.json file that is created when you run backstop init.

This can be edited to add in your own scenarios.

This is an example of the setup we are using.

Key Points:

• The Viewports are defined for use in all tests• resembleOutputOptions define the comparison colours and display style• The asyncCaptureLimit and asyncCompareLimit are used to control the

number of threads and memory the tests use.• "engine": "chrome“ defines that the chrome headless browser will be used.

Configuration Setup

{

"id": "pm",

"viewports": [

{

"label": "phone",

"width": 320,

"height": 1024

},

{

"label": "tablet",

"width": 768,

"height": 1024

},

{

"label": "desktop",

"width": 1268,

"height": 1024

}

],

"onBeforeScript": "chromy/onBefore.js",

"onReadyScript": "chromy/onReady.js",

"scenarios": [

{

……..

}

],

"resembleOutputOptions": {

"errorColor": {

"red": 255,

"green": 0,

"blue": 255

},

"errorType": "movement",

"transparency": 0.3,

"ignoreAntialiasing" : true

},

"asyncCaptureLimit": 25,

"asyncCompareLimit": 400,

"engineFlags": [],

"engine": "chrome",

"report": ["browser","CI"],

"debug": false,

"debugWindow": false

}

25

For scenarios setup there are two key parameters :• url• selectors

These define the page the what will be captured on the page. The selectors used simple CSS selectors to identify elements on the page to capture.

To capture the contents of the <div id=“header”> then the #header is used as a selector.

If a referenceUrl is specified then this will be used instead of the url when generating the reference images. This is used when comparing sites.

The hideSelectors parameter is used to define the elements on the page that should not be shown. This is useful to hide cookie notifications for example that could obscure capture of components.

Scenario Setup{

"label": "header",

"url": "http://people-management.local/bb-prototype-

website_example_grid.html",

"selectors": [

"#header"

],

"hideSelectors": [

"#monitor_mq",

".grid",

".visible-grid-btn"

],

"readySelector": ".page-loaded“

},

{

"label": "podcast",

"url": "http://beta.pmdev.co.uk/test/podcast",

"referenceUrl": "http://internal.people-

management.demo.building-blocks.com/bb-prototype-

website_components-podcast.html",

"selectors": [

".podcast-container"

],

"hideSelectors": [

"#monitor_mq",

"#SEStartButton",

"#cookieMsgWrapper2",

".grid",

".visible-grid-btn"

]

}

26

Scenario Setup :

"onReadyScript": "burgerMenuTest.js",

Chromy Script :

module.exports = function (chromy, scenario) {var clickSelector = scenario.clickSelector;var postInteractionWait = scenario.postInteractionWait; // selector [str] | ms [int]

if (clickSelector) {chromy.wait(clickSelector).click(clickSelector)

}

if (postInteractionWait) {chromy.wait(postInteractionWait);

}};

Chromy

Backstop JS the future …..

27

• Catch Regression• Allow comparison between DEMO HTML and Website• Quick to setup• Easily integrate in to existing HTML build process