automation as a project - lessons learnt

Poznań Testingand Quality Group

Automation as a project –

and Quality Group

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Automation as a project –lessons learnt

Zbyszek Moćkun

Agenda

• Introduction• Introduction

• Business

• Technology review

• Management

• Summary

• Discussion


Few words about the author

Introduction

Few words about the author


Experience and domains

Experience Domains

• 7 years in tests

• Three different companies

• Several domains

• Product based and project

based

• Different methodologies

• Mobile infrastructure

• Billing and Roaming

systems

• Business Assurance

application

• Medical

• Content Management


• Different methodologies

(Waterfall, Scrum, Kanban)• Content Management

Systems

Automation vs. profitability

Business

Automation vs. profitability


It’s all about the money


http://www.br-automation.com

How to talk with stakeholders

• Time of money return• Time of money return

• Use statistics/metrics

• Remeber that statistics show incorrect data (only regression

issues should be calculated)

• It’s not about how many issues were found

• Statistics fall down without continuous investement

• Code resistance on the same tests


• Code resistance on the same tests

• Decrease release time (time for manual tests)

• Number of issues found influence on time needed for

regression

How to start?

• Show metrics• Show metrics

• Why does we need it?

• How much does it cost?

• What does we want to achieve?

• What are the goals?

• Small steps

• First step should be cheap and quick


• First step should be cheap and quick

• Show results for stakeholder

• Plan and estimate next steps

• Gather metrics and statistics

Approaches

Product-based

Project-based


Product against project

Product-based Project-based

• Scalability

• Modularity

• Coverage – as much as

possible

• Expensive - infinite number

of runs

• No need of scalability

• Only basic modularity

• Coverage – main features

• Cheap - finite number of runs

• Reuse for other projects (not

always possible)

• No place for planning new


• Reuse for other products

• Planing new features and

improvements

• No place for planning new

features and improvements

• Manual tests are cheaper at

the most cases

Technology

Automation

Technology


One place for automation management

Continuous Integration

One place for automation management


Where should automated tests be run?

• On developer machine if possible, before commit

• As sanity check on other machines

Developer machine Nightly Iteration Staging Production

• As sanity check on other machines

• Developer iterative work (migration as example)

• Should we run autoamted tests on production (live)?

• Independent test data


machine

CI tool

Reports

• Nightly tests – preferred frequency

• Build results management (disc space)• Build results management (disc space)

• Reports improvements

• Minimalize time spend on report (verification/debugging)

• Display as much data as possible for failures

• One issue failing several tess script

• Environment errors - stability

• Red reports effect

• Issues are not fixed


• Issues are not fixed

• A lof of failed tests cases

• Noone look at the results

Reports


How to build good test suite?

Dependent or

independent?

How to build good test suite?


Independent and random

• All tests are independent

• Pre – precondition (set require state)

• Post – postcondition

• Test – test script

• Rec – recovery procedure after test failure (pre i post should have defined too)

• LogData – gather all logs after failure, very important for „random” failures

• Random order

• Repeat counter


• Dependent only for CI and adult teams (founded issues are

fixed as fast as possible)

How to manage test data? Load them or

Test data

How to manage test data? Load them orcreate?


Does it work?

Record & Play

Does it work?


Selenium IDE as example

• Stability issues

• Recorded values are not meaningfull• Recorded values are not meaningfull

• Require rewrite

• User-extension.js (write own commands)

• Quick and easy debug

• Project-based – preferred as cheap and quick

• Product-based – as starting point

• More info link


• More info link

Product against project

• Quick and easy (+)

Cost return (+)

HTML Java

• Cost return (+)

• Coverage – max 10%-20% (-)

• Modularity using user-

extension.js (=/-)

• No need to environment

setup (+)

• No scalability (-)

• Cnvert to other languages

• Flexibility (+)

• Scalability (+)

• Modularity (+)

• Coverage (+)

• Development knowledge (+/-)

• Environment setup (-)


• Cnvert to other languages

(+)

• HTML Suites (works with CI)

(+)

• Integrated reports

• No convert possibility (+/-)

• Debug time (-)

• Reporting (JUnit, TestNG) (-)

Survey

Two similar application and different technologies based on

Selenium: Selenium:

• HTML based on Selenium IDE

• Java based on Selenium RC/Eclipse

HTML – 50% more test cases written

Java – 20% more time spend on maintenance/debugging


Which technology would you like to use?

Java devs choose HTML/Selenium IDE as more friendly

and faster

Management

Automation

Management


Manage, monitor and plan

Coverage

Manage, monitor and plan


20/80 rule


Coverage

Manage your coverage and use it for planning manual tests

• Remember about 20/80 rule

• Power function

• 100% - is it possible?

• Network architecture as example (100% achieved)

• Human factor caused that 100% is not achievable


• Red report effect

Defect management

• Red report effect• Quick fixes for CI• Labels


Who should write and maintenance

Automation responsibility

Who should write and maintenanceautomated tests?


Developer helping hand

Can developer deliver automated tests together with

functionality?functionality?

• Who should write test cases?

• Scripts written using invalid test cases

• Scripts written for not working functionality

• Lack of time – functionality is more important

• Control needed (sometimes devs forget)

• Two unsuccessful attempts


• Two unsuccessful attempts

Triangle: Budget, Time, Scope (Aim)

Automation as a project

Triangle: Budget, Time, Scope (Aim)


There is no success without:

• Continuous work• Gather statistics for stakeholders• Gather statistics for stakeholders• Divide budget on manual and automated tests• Planning the future – where want to be in next year?• Control your costs• Remember – it’s all about the money


Automated Exploratory Tests for Regression

The future

Automated Exploratory Tests for Regression


Aspects

CICI

Client

User


AnalyticsLayout Functionality Performance

Symptoms

Layout

Bug

Content

CodeLink checker

URL strategy


JS errors

Response (404)Analitycs

W3C

Heuristic Oracle

The balance between comparison and comparison not at all

Layout Content Code Performance


Robots

• Scripts• Shared between projects• Shared between projects• Easy and quick to run


JS errors Responses(404) W3C Analytics Link checker Url strategy

Q&A


Poznań Testing and Qaulity Group

Discussion & beer

Poznań Testing and Qaulity Group


automation as a project - lessons learnt

Education

cognifide limited

commercial confidence

release time time

tests billing

autoamted tests

manual tests number

improvements manual

test datahow