visual regression testing
TRANSCRIPT
VISUAL REGRESSION TESTING USING GALENJYOTHY MARIAM JOHN, RISHABH RAO
16TH JULY 2016
Change BlindnessChange blindness is a surprising perceptual phenomenon
that occurs when a change in a visual stimulus is introduced and the observer does not notice it. https://www.youtube.com/watch?v=ubNF9QNEQLA
Visual regression tests
Tests the layout of a page, the way in which components - texts, images, grids etc are set out on a page.
Functional Test will pass for both the cases though layout is broken in the second image. NOTE: Tooltip error message is overlapping text box
Why Visual regression tests?
CI / CD Environment - Frequent Releases
Responsive Web Design
Themes / Skins
Responsive Web Design
Responsive web design (RWD) is an approach to web design aimed at crafting sites to provide an optimal viewing and interaction experience—easy reading and navigation with a minimum of resizing, panning, and scrolling—across a wide range of devices (from desktop computer monitors to mobile phones)
Themes / Skins
Applying a skin changes a piece of software's look and feel—some skins merely make the program more aesthetically pleasing, but others can rearrange elements of the interface, potentially making the program easier to use.
Automated Visual Regression Tests thus become QA’s lifeline in tackling such
complexities
Agenda
✓ Why visual regression tests are important
Existing tools & Overview of Galen
My journey, challenges faced / learnings
Galen
A test framework originally designed for testing layout of web applications. When it comes to testing a responsive layout it works in the following way
Open a page in browserResize it to specified size
Test the layout according to user-defined specsIt uses Selenium for interacting with elements on page and getting their locations and dimensions.
Once it sees that something is wrong – Galen reports the error, makes a screenshot and highlights the misbehaving element on it.
http://galenframework.com/docs/all/
Galen
Galen unlike existing image comparison tools, provides a way to test components on a page with reference to other components present.
Galen also supports image comparisons.
Basics
Galen CommandsGalen Specification FileGalen Reports
Galen commands refer spec file for page validation
Spec File Format
Object Definitions, for Galen to find elements on test page. Supported locators are id, css & xpath
Spec Sections, for describing the page layout for each viewport size using spec reference keywords like near, below, above, left-of, right-of, inside etc.
http://galenframework.com/docs/reference-galen-spec-language-guide/#SpecsReference
Sample Spec File
Spec Reference
Or be less strict and saynear button 5 to 15 px left
CSS Property
Test the value of a CSS property for a specific element
Image
Supports Image Comparison
Reports error if mismatching pixels are more than acceptable range 12%
The red pixels on comparison map are the mismatching pixels that are far away from tolerance value, yellow pixels are in the range of 30 to 80 colour difference from the tolerance value, Green pixels are the closest to tolerance value.
By default tolerance is set to 30, if we increase this parameter, the amount of mismatching pixels will be decreased
Command Line Tests
In order to perform a single page test you can use the command “check”
JavaScript Tests
Provides list of API’s to write JavaScript testsE.g load, createDriver, createGridDriver, resize, checkLayout, checkPageSpecLayout, parsePageSpec etcAlso provides a lightweight javascript testing framework (GalenPages) to work with page object model
test("Home page", function() {
var driver = createDriver(“http://galenframework.com","1024x768");
checkLayout(driver, "homePage.gspec", ["all", "desktop"]);
});
http://galenframework.com/docs/reference-javascript-tests-guide/
Java Tests
Use Galen inside Java tests
Java + Maven + TestNG + Galen
Integrate visual regression tests into your existing functional regression tests framework and reuse navigation / actions which is already defined for existing tests
import com.galenframework.api.Galen;
Agenda
✓ Why visual regression tests are important
✓ Existing tools & Overview of Galen
My journey, challenges faced / learnings
Reference Tests
Pixel Tests
http://galenframework.com/post/2014-09-28-version-1.3.0-image-comparison-in-layout-testing/
Image Comparison
Configuration File
runMode = createBaseline / validateUI
Image ComparisonChallenges
Dynamic contentVarying image colours on different screens
Image RenderingDifferences
Image RenderingDifferences
Addressing dynamic content
Validate @Component Level
Addressing dynamic content
Change RunTime Behaviour Of AUT
Addressing dynamic content
Mock responses
https://github.com/mrak/stubby4node
Addressing dynamic content
Inject javascript to modify the text displayed
Reports
Inherit all your test class from com.galenframework.testng.GalenTestNgTestBase
Add com.galenframework.testng.GalenTestNgReportsListener in your pom file
Reports
Baseline Actual Comparison Map
Reports
Baseline Actual Comparison Map
Error Vs Tolerance
Error Vs Toleranceblues.png, tolerance 30 blues.png, tolerance 80
When tolerance is increased red area (in image2) has decreased and less error value is reportedMore the tolerance, lesser the error..