agile and the automated testing of accessibility

Agile and the automated testing of accessibility

Agile Tools

5© 2015 - All Rights Reserved

Shift Left Testing

Catching a defect in the design and requirements phase can reduce cost to fix by up

to 100:1

Software Defect Reduction Top 10 List, Barry Boehm and Victor R. Basili, January 2001

https://www.cs.umd.edu/projects/SoftEng/ESEG/papers/82.78.pdf


Rise of Continuous

• Lag between committing code and getting feedback allows damage to magnify– Costs rise– Velocity decreases

• Continuous Integration• Continuous Deployment

http://www.startuplessonslearned.com/2009/06/why-continuous-deployment.html


Automation



Accessibility Testing’s Image

Aim of Software Testing

• Reduce the risk of the existence of a serious problem on a supported configuration to an acceptable level– Define “serious problem”– Define “supported configuration”– Define “acceptable level”


Android Fragmentation


Android Fragmentation

We have seen 24,093 distinct devices download our app in the past few months. In our report

last year we saw 18,796. In 2013 we saw 11,868

OpenSignal 2015


Combinations

4 operating systems, 2 supported versions5 Assistive technologies4 Browsers, 2 supported versions3 Responsive breakpoints

216 combinations excluding device combinations


Sparse Testing Matrix

Browser/AT

Chrome KB IE KB

Firefox KB Safari KB Dragon IE NVDA FF Jaws IE VO Safari TB FF Orca FF

Operating

Sytem

Windows

OS X

iOS

Android

Linux


Accessibility “Acceptance Automation”


Acceptance Automation

• Comes from the “requirements”– Designers mark up their wireframes and comps– Stories must contain specific accessibility

acceptance criteria• Can go to any level of detail– Test the specific alt attributes on images– Test the DOM order specified in the wireframe– Etc.


Open Source Rules Engine


aXe Manifesto

• Must have a zero false positive rate• Must be lightweight and fast• Must work in all modern browsers on all

platforms• Must be tested automatically


aXe-Core

• Small ~100KB• Fast

– No network connection, no new browser start• Intelligent

– Supports cross domain iframes– Turns document level rules on/off automatically

• Unlimited– No throttling, no server limits

• Reliable– No double-execution or missing CSS problems– No network connectivity problems– No server capacity problems

• Secure– No server access across the wire– No sending data to a server

• Mozilla Public License 2.0– Available via GitHub, npm and Bower


aXe Chrome extension


aXe Firefox extension


aXe extension

• Developers– Use when coding

• Testers– Use during functional testing

• Accessibility Experts– Use for validation


aXe API

• Two major calls– axe.getRules() – returns a list of the rules loaded– axe.a11yCheck() – analyze a page or portion of a page

• Getting started– Install package

• bower install –-save-dev axe-core

– Include library• <script src=“(path_to)/axe.js”></script>

– Call function• a11yCheck(target, options, callback);


aXe API demo


https://youtu.be/zy7Of9WO5JM

Integrations

• Java Selenium• JavaScript Selenium (axe-webdriverjs)• Grunt (grunt-axe-webdriver)• Cucumber and R-Spec (axe-matchers)• Firefox Dev Tools• Chrome Dev Tools• Ember-aXe (ember-axe)

more…


Summary

• Manage your matrix– Optimize for resources and risk

• Leverage automation– Generic automation– Specific automation– Optimize for up-front vs. ongoing effort


Questions

• https://addons.mozilla.org/firefox/addon/axe-devtools/

• Chrome: search Google for “chrome axe extension”


https://addons.mozilla.org/en-US/firefox/addon/axe-firefox-devtools/






agile and the automated testing of accessibility

Documents

agile process

continuous feedback

agile framework

rights reserved8

rights reserved7this

entire process

new tools

feedback loops