Cynthia C. Shelly, Microsoft Mike Barta, University of Washington

Web Applications Are interactive applications first “Web” is an implementation detail History of the Web as a collection of

interconnected static documents obscures this

Accessibility is an aspect of software quality The same methods used to find security or

performance or functionality bugs can be used to find accessibility bugs

User TestingAutomated ValidationAuditing

Traditional Software Testing?

What is it End users Influence or validate design Focused on discoverability and ease of use

Types Usability Testing Beta Testing Reviews by end users outside the product

team Application to Accessibility

Testing with ‘real’ people with disabilities

Uses Understanding design issues How the product feels Interaction with a particular AT product▪ User who is comfortable using the AT product

Issues Isolating technical issues so developers can fix them Usability testing uses small samples of like users▪ Difficult to cover all disabilities▪ Difficult to get a group of users with same disability and AT

configuration Can be expensive, in time or money Beta testing and user reviews have response bias, skew

towards more technical users Need working code for AT interop testing, prototypes

less useful for accessibility user testing

What is it Static analysis of code looking for common

problems Done on markup, generally not on script

Types General purpose tools test against a

technology specification Product-specific tools test against the product

specification Application to Accessibility

Test against an accessibility standard (WCAG, 508, etc.)

Uses Quickly find common, known problems in markup Find regressions Spot trends, find areas where manual testing

resources would be best spent Issues

Many accessibility issues are difficult to automate

The issues that are easy to automate aren’t always the most important to test

General purpose tools have a lot of warnings and false postitives

Real bugs can get lost in the noise It’s a tool, it’s not testing

What is it Done on finished or mostly finished

product By experts not involved in development Combination of automated testing and

expert evaluation Measures conformance to a set standard

Application to Accessibility Expert evaluation against accessibility

standards

Uses Evaluates the state of a product at a given point in

time Useful for reporting Useful for development teams without the expertise

to do their own testing Issues

Measures quality, does not assure it Late in product cycle Limited ramp-up time Limited time to conduct audit Organizational dynamics can work against outside

auditors Difficult to convince organizations to fix technically

compliant but poorly accessible/usable sites

What is it Focus is on assuring a level of quality Types of testing typically done during the process of

developing software Discrete tools such as Unit, Functional, System,

Integration, Regression, etc. Procedural tools, such as exit criteria and design

validation The goal of testing is not to perform evaluations, but to pass them

Application to Accessibility Test accessibility as one of many metrics of software

quality Test the functional accessibility of an interactive

application Test at various architectural levels for the exposure of

information needed for accessibility

Uses Change the output on a quality measure Test throughout the software life cycle: requirements gathering,

design, coding, testing, release Isolate technical issues so developers can fix them Find issues early, and educate the team, to prevent issues Targeted: picking test cases most likely to uncover bugs Understand the most important bugs, based on the purpose of the

product Constant stream of information allows adjustments can be made

during the development process for the current release Work with the organizational dynamics and rhythm of the

business to impact organizational behavior with respect to quality metrics▪ Help managers plan for resource allocation to impact accessibility at various

points in the software life cycle▪ Raise the expertise of organization▪ Devote significant time to testing▪ Test accessibility the same way other quality metrics are tested▪ Leverage existing processes to reduce marginal cost

We can improve the accessibility of a Web application by treating accessibility the same as other quality metrics, and the application the same as other applications

Traditional Software Testing has a goal of changing the quality of a product, not just measuring it

There is much to be gained by applying the knowledge in the software testing field to web accessibility