emerging paradigms in testing - anztb vendors using mixed approaches to development, agile and...

© ThoughtWorks 2012

Emerging Paradigms in Testing

Refocus your view of testing to drive superior results

© ThoughtWorks 2012

A good testing experience

“The quality of the application, rapidity of testing, as well as the ability to easily and confidently update the service once live, were major benefits. The test driven development and continuous integration approach was the most professional I’ve seen and provided very high quality and stability.”

Ian Frew, Leader IT, The Buzz at IAG

© ThoughtWorks 2012

The case study !   Multiple vendors using mixed approaches to development, agile and waterfall -

Approximately 60 developers !   Mainframe backend, SOA and multiple front-end systems and external 3rd

party system. !   Testing covered performance, security (including PCI), disaster recovery, deploy/

rollback, failover/recovery, user acceptance, usability and functional.

!   5 full–time testers

!   After go live could make a code change and deploy in the same day no Severity 1 or Severity 2 defects in 9 months.

!   6 months development, after last feature developed 1 week to go live. no Severity 1 defects in first 9 months.

© ThoughtWorks 2012

Testing Principles

We believe testing should focus on building quality in rather than "testing it in". We do this by involving everyone in the testing effort, using fast feedback to make decisions. We believe tests are an asset of the product and not the project. Working in this way enables faster delivery into production, provides a clear and consistent view of testing across the organization and helps business optimize value within the software.

© ThoughtWorks 2012

Before we get into the details

!  Describe the principle !   Present a practice demonstrating the principle in action !   Provide a real world example from the case study highlighting

the principle in action

© ThoughtWorks 2012

Principle: Build quality in

!   Remove the cost of fixing the defects by preventing them from

occurring.

!   Find mistakes early and prevent them from becoming prevalent.

!   Reduce reliance on band aid fixes.

“You can not inspect quality into the product; it is already there.” – W. Edwards Deming

© ThoughtWorks 2012

Practice: ATDD (Acceptance Test Driven Development)

BA works with business to define a feature in a testable, bite sized vertical slice (implementable in

2-3 days)

BA and Tester work with business to define how the

feature will be tested

Tester works with developers to implement the tests and

developers implement the feature

BA and Tester get handover of feature from developers

BA and Tester test the feature and how it fits in to

the broader system and business process

X test fails

✓ Test passes

© ThoughtWorks 2012

Example: Defects as a measure

Preventing Defects

Finding Defects Preventing Defects

Finding Defects

© ThoughtWorks 2012

Principle: Fast feedback

!   Get feedback to the people who need it as soon as possible.

!   Work with up to date information and make informed decisions.

!   Regression suites should be run in hours not weeks.

© ThoughtWorks 2012

Practice: Test Pyramid

Unit

Integration

Acceptance

Scenario

More tests Increased execution Time

© ThoughtWorks 2012

Example: Mainframe developers

http://www.kome-computers.com/image-files/mainframe.gif

© ThoughtWorks 2012

Principle: Involve everyone

!   Target utilization of testing resources.

!   Get better tests through diverse input and the team looking after the tests.

!   Better use of resources – equalizing.

Testing ≠ Testers

© ThoughtWorks 2012

Practice: Automation Framework Write Test Specification in Text

Use Code

Write Code

BUSINESSREPORT

TECHNICALREPORT

TEST INTENTION DATA

HTML SERVICE DATABASE REPORT

DOMAIN SPECIFIC LANGUAGE (DSL)

TEST RUNNER

© ThoughtWorks 2012

Example: Rating Table

http://www.car-insurance-for-learner-driver.com/images/group-2-car-insurance-pic.jpg

© ThoughtWorks 2012

Principle: Tests as an asset

!   Reuse tests across projects, don’t create new tests all of the time.

!   Better understanding of test coverage.

!   Treat the test cases and the application code as the product – don’t separate them.

Code

Tests Product +

© ThoughtWorks 2012

Practice: Version Control

!   Automated Tests are code – treat them like code

!   Test code is likely to be the largest code base in your organisation

!   Version controlling application code is common practice – so it should be for test code.

© ThoughtWorks 2012

Example: Can’t rollback tests

http://images.coplusk.net/projects/10202/steps/20603/full_20603.jpg

© ThoughtWorks 2012

Principle: Faster delivery into production

!   Do not have months of testing after development is complete.

!   Return on investment early – start earning revenue as soon as you can.

!   Testers should stop focusing on preventing a product going live and instead focus on how to get it live.

© ThoughtWorks 2012

Practice: Build Pipeline

© ThoughtWorks 2012

Example: Dev to Prod in same day

FrontEnd Dev CMS Dev Frontend Dev 3rd Party Doc Dev 3rd Party

Claim Dev

Security Dev

Claims Call Center

Call Center

Dev

Dev Test

Dev

Pre-Int Int Sys SIT UAT

DR

Prod

Dev Dev Dev

Operations

Security

Performance

Services

Acceptance

Exploratory

Integration

© ThoughtWorks 2012

Principle: Clear & consistent view

!   Simple reporting so you can compare one project with another.

!   Metrics should not drive behavior.

!   Open and honest.

© ThoughtWorks 2012

Practice: Example Report

© ThoughtWorks 2012

Principle: Optimize business value

!   Testing is not just insurance. !   Target testing to features that are changing not a scatter gun test

everything approach. !   Help drive new features and functionality. !   Allow business to try new ideas in a safe environment providing quick

feedback

© ThoughtWorks 2012

It is not just small web apps!

Data warehousing - consolidation of multiple (approx 30) data warehousing solutions into one. Multiple streams in the program of approx 150 people.

Trainline - Train ticketing system: distributed team, UK and India approximately 200 People.

Financial Services – integration of over 70 systems (internal and 3rd party). Distributed team, Australia and India approximately 100 People

© ThoughtWorks 2012

It is about the principles!