test automation as a service (taaas) · pdf filetest automation as a service (taaas) - slide 1...

Test Automation as a Service

(TaaaS) Next Generation Testing:

Innovations and Challenges Jonathon Lee Wright

Director of Testing

[email protected]

@jonathon_wright

www.TaaaS.net (blog)

www.automation.org.uk

automation.sharepoint.com

www.linkedin.com/in/automation

18th May 2011 Test Automation as a Service (TaaaS) - Slide 1

Introduction

Agenda

• Why the cloud?

• Automation in the cloud;

• New opportunities and benefits;

• New risks;

• What we are doing now;

• The future;

• Live demo.

Test Automation as a Service (TaaaS) - Slide 2

Introduction

18th May 2011

Why the cloud?

• The cloud provides no up-front investment,

featuring instant scalability, flexibility and availability;

• Pay as you use (PAYU), no more expensive vendor licencing

costs or un-utilised maintenance agreements;

• Complex test environments on demand:

“built to expect & withstand failure”;

• Established technology avoiding support issues with non-

commercial products;

• Avoid the minefield of complexity requiring internal specialist

staff skills following industry best practice;

• 100% web delivered providing collaboration and mobility.


Cloud Services

18th May 2011

New Testing Opportunities

• Cross Platform and Compatibility - browser, operating system,

mobile devices, script runtimes, languages, HTML versions;

– Market Configuration Testing – 10+ Applications (Shop / ShelfReview / Nominations),

10+ Clients (Tesco / Macy‟s), 10+ Workstreams (Tesco UK / Tesco France),

10+ Languages (en-GB / fr-CA), 10+ Platforms (XP-SP1, SP2 & SP3),

10+ Browsers (I.E 7, 8, 9), 10+ Runtimes (Flash 9.0.44 ~ 10.3.181).

• Accessibility/Usability/UXD/WAI - due diligence in guaranteeing

accessibility to people with disabilities;

– 88% of the delegate organisations here today failed W3C compliance tests;

• Globalisation – realistic performance & load testing plus

compliance with the commercial and civil law of the country;

• Search Engine Optimisation – periodic ranking checks;

• Penetration, Performance & Load Testing.


New Opportunities

18th May 2011

New Risks (in the cloud)


New Risks

18th May 2011

Commercial Ecosystems Fulfilment Systems

• What happens when these Services FAIL?

– Systemic Failure? (e.g. DNS or Network link)

– Alternative service solution provider?

– Unique Services? (such as GPS)

Taking Automation to the Cloud


Cloud Automation

18th May 2011

Customer Systems

Intranet

Site

Test Suite CMMI/ISO 9000

Rule StoreSoftware Build

Process

Desktop

Application Web Site

Web services PersistenceVirtual lab

The Cloud (System Infrastructure)

Customer Automation Suite

Business Level

Keyword Definition

United Reporting

Credentials

Flexible Cost

Planning

Support

Test Data

DashboardTest Asset

Management

Automation Lifecycle Management

(Test Automation as a Service Model)

Monitoring/

Operational management

Configuration

Reporting

Billing

Customer Take on Customer support

Development

DeploymentMaintenance

Why this approach?


Cloud Automation

18th May 2011

Approach BPM BPT Keyword Data Modularity Tools

Traditional ○ ○ ○ ○ ● Vendor Tool (Market Leader)

Hybrid ● ● ● ● ● .NET Development

(Visual Studio 2010 + Coded UI APIs)

○ - Limited Support

● - Full Support

Traditional (Vendor Tool)

Advantages Constraints

Vendor Support

(maintenance subscription)

Object Repository

(object-based Record /

Replay)

Single Technology Stack

(SPOF)

Tool Vendor Dependency

C# Project incompatible with

Visual Studio

Technology Blocks

(Custom Flex Objects/Classes)

Limited non-technical tester

interface

Hybrid (.NET Development)

Advantages Constraints

Multiple Technology Stack

(VS2010 / C#.net / Silverlight /

Azure / SQL / WCF /

CodedUI / Ranorex API /

XML / XPATH / TFS)

Integration with development

(Visual Studio 2010)

Integration with deployment

(TeamCity)

Integration with testing

(Microsoft Test Manager)

In-house framework

development & maintenance

effort.

Additional skills / knowledge

resources

Object Repository

(limited Record / Replay

support)

What is different with this approach?


Cloud Automation

18th May 2011

• Design tests before release ‘first day’ automation testing – Create complex test scenarios before the application has even been written;

• The test language is self-validating and human readable – The verbs and nouns in the domain specific language are written in natural

language which uses context sensitive validation;

A1

B1

Login.Process

Buy.Tickets

Check.Tickets

My.Account

Logout.Process

D1

C1

B2B4

C3B3C2

B5

D3C4 D2

D4 D5

C5

Lo

gin

.Pro

cess

Action Object Event

Enter Username Text

Enter Password Text

Press Login Button

Nouns

Verbs Generic Reusable Libraries

Act

ions

Obj

ect

Han

dler

Eve

nt

Han

dler

Press | Login | Button Example Business Level Keyword

BPM

What is different with this approach?


Cloud Automation

18th May 2011

• The test definition is not implementation specific – Test execution is performed by cloud based technology adapters which act as

interpreters for the vendor agnostic and implementation agnostic language;

Test Run Generate Environment

Under Test (EUT) Select Platform

Adapter Select Test

Adapter Select Technology

Adapter

• Application Server

• Database Server

• End-Client Machine

• Windows 2008 RC1

• Windows 2003 RC2

• Windows 7 SP1

• Load Testing

• Performance

• Functional

• Coded-UI (Win32)

• Selenium (ASP.net)

• Ranorex (Flex 4.0)

• The definition of a test artefacts is self-maintaining – Asset loader uses fuzzy logic matching to identify UI objects;

Enter password here ……

Please enter your password to logon

logon logoff

Enter password here ……

Please enter your password to login

login logoff

1. XPATH Check

2. Regular Expressions

3. Descriptive Programming

4. Fuzzy Logic

5. Artificial Intelligence

n/a

Event Based / BDD Testing


Cloud Automation

18th May 2011

• Automated testing based on trigger events (passive & active): – Third party system events such as user story or defect status workflow change;

‒ Continuous build and integration or source code gated check-in.

Passiv

e

Acti

ve

Where are we now?


Cloud Automation

18th May 2011

Development (.NET 4.0)

AutomationDatabase

Automation Portal (TaaaS)

Application Under Test

2003/2007/2010

Lab Management

Windows

Service

ObjectHandler.dll

DataHandler.dll

EventHandler.dll

Application Programming Interface

Unified Reporting Model

DashboardReportingTest Management Tool Team Foundation Server

Application Lifecycle Management

Test Automation as a Service (TaaaS)

Cloud Web Endpoint

Test Scheduler Service

How it works (in the cloud)


Cloud Automation

18th May 2011

End Users

QA Person

Run a test«uses»

Schedule a future

test

«uses»

TaaaS Portal

Silverlight

Dashboard

Test Runner Environment - A Virtual Machine

created by the Framework VM Dispenser

TestRunner

Test Data

Set

Technology

AdapterTest Adapter

Platform

Adapter

Client System Test Repository

Business

Level

Keywords

(BPM/BPT)

Test Data

SetsResults

Object

Repository

View real time

test status

«uses»

Business User

«uses»

View historic test

results

«uses»

Dashboard Service

Test Assets Repository

(MTM / Quality Centre)

TD-API

Adapter

Results Service

Test Management

Unified Reporting (Dashboard)Check Development

Progress

Framework

Results

Adapter

Silverlight

Dashboard

Developer

View Live Defect

Build Client System

Build Automation

(Team City / TFS)

Framework

Object

Adapter

Asset Scanner Service

Benefits to the Customer


Cloud Automation

18th May 2011

On Demand

No capital

Investment

Minimal Time

Investment

Cost

Controlled

Public facing Web sites

Source code

Public facing Web Services

Test Assets

Intranet

Desktop applications

Business Process Model

Test System

Software Build Process

Customer Assets

Domain Experts

Use Existing

Test Assets

TaaaS

Load Testing

Penetration Testing

ISO/CMMI Progress

Contribution

Market Compliance

Search Engine Optimisation

Code Standards

Regression Testing

Accessibility Standards

Performance Testing

Customer Defined Tests

‘Keep it Simple’ Automation


Cloud Automation

18th May 2011

User stories

represented by a

number of Business

Level Keywords

Save Test (Selectable

in the future from the

Your Saved Tests List

Box)

Start Test (As per

the custom list of

business level

keywords)

TaaaS vs. DIY


Cloud Automation

18th May 2011

TaaaS

Future forecast "Cloudy"

1. Avoid the Hype: – Ensure that the critical business flow can be automated followed by

complex custom components "avoid automating the simple components

first“ e.g. simple login pages;

2. Don't be tempted: – By third generation automation vendors offering so-called "script-less"

tools. Be realistic, you need to be able support custom classes and the

latest technologies without having to wait on the possibility that it may be

supported in the next release;

3. Integration into testing process: – Automation is no longer an "us and them" scenario between manual and

automated tests;

4. Make the necessary plans: – How easy is it to start using the service and how easy is it to move away

from the service.


The Future

18th May 2011

Something to think about

“The best aspects of these proven test approaches demonstrate

how they have evolved over the past decade, and this echoes

some of the changes towards more lean and agile business

methodologies. They are in a constant state of evolution – just

as the underpinning technology evolves over time.”

my section in „Experiences in Test Automation’ book

Dorothy Graham & Mark Fewster 2011


The Future

18th May 2011

Automation - Demo


Cloud Automation

18th May 2011

test automation as a service (taaas) · pdf filetest automation as a service (taaas) - slide 1...

Documents