testing the limits of testing

45
Testing the limits of testing… Phil Robinson LonsdaleSystems.com

Upload: phil-robinson

Post on 05-Dec-2014

362 views

Category:

Technology


0 download

DESCRIPTION

 

TRANSCRIPT

Page 1: Testing the limits of testing

LonsdaleSystems.com 1

Testing the limits of testing…

Phil Robinson

LonsdaleSystems.com

Page 2: Testing the limits of testing

LonsdaleSystems.com 2

In theory there is no difference between theory and practice. In practice there is.Yogi Berra

Blah blah blah blah…

Teaching software testing…

Page 3: Testing the limits of testing

LonsdaleSystems.com 3

The project…

Page 4: Testing the limits of testing

LonsdaleSystems.com 4

The project objective…

Front End (ASP)

Front End(Oracle ADF)

Transaction Queue

Transaction Queue

(Oracle AQ)

DatabaseBack End (PL/SQL)

Page 5: Testing the limits of testing

LonsdaleSystems.com 5

The project plan…

Testing

Project Deadline

Page 6: Testing the limits of testing

LonsdaleSystems.com 6

The test strategy…

Page 7: Testing the limits of testing

LonsdaleSystems.com 7

The waterfall “mantra”

Analysis

Design

Code

Test

Unit

Integration

System

Acceptance

Page 8: Testing the limits of testing

LonsdaleSystems.com 8

Testing is actually three different activities…

Test Design

Test Execution

Defect Repair

Page 9: Testing the limits of testing

LonsdaleSystems.com 9

Testing is about quality not project phases…

• Fit for intended purpose

• Conform to spec• Absence of Defects and other quality goals

Page 10: Testing the limits of testing

LonsdaleSystems.com 10

Testing objectives…

• Validate that the software is fit for its intended purpose

• Verify that the software conforms to its specification

• Identify failures caused by defects

• Measure non-functional attributes of the software (security, performance, reliability…)

• Build confidence in the software

Page 11: Testing the limits of testing

LonsdaleSystems.com 11

Test to pass…• Validate• Verify• Measure• Build confidence

Test to fail…• Identify failures• Measure (stress)

Testing objectives…

Page 12: Testing the limits of testing

LonsdaleSystems.com 12

Vertical integration…

Unit test

Page 13: Testing the limits of testing

LonsdaleSystems.com 13

Vertical integration…

Integrate

LonsdaleSystems.com

Page 14: Testing the limits of testing

LonsdaleSystems.com 14

Vertical integration…

Unit test

LonsdaleSystems.com

Page 15: Testing the limits of testing

LonsdaleSystems.com 15

Vertical integration…

Integrate

LonsdaleSystems.com

Page 16: Testing the limits of testing

LonsdaleSystems.com 16

Vertical integration…

Unit test

LonsdaleSystems.com

Page 17: Testing the limits of testing

LonsdaleSystems.com 17

Vertical integration…

Integrate

LonsdaleSystems.com

Page 18: Testing the limits of testing

LonsdaleSystems.com 18

Horizontal integration…

Feature Feature Feature Feature

Database

Page 19: Testing the limits of testing

LonsdaleSystems.com 19

Horizontal integration…

Feature Feature Feature Feature

Database

Page 20: Testing the limits of testing

LonsdaleSystems.com 20

Testing should not be “black and white”…

• “Black-box”• Functional• Specification based

• “White-box”• Structural• Code based

Page 21: Testing the limits of testing

LonsdaleSystems.com 21

Designing test cases…

Test Item Test

Cases

Spec

Code

Other Docs

Test Basis

Existing System

Page 22: Testing the limits of testing

LonsdaleSystems.com 22

A Better way to define test strategy…

• Spec• Code• Other docs• Existing

systems

• Fit for intended purpose

• Conform to spec

• Identify failures

• Measure non-functional attributes

• Build Confidence

Test Item Test Basis Objective Who?

• Developer• Technical

specialist• Test

Analyst• Business

Analyst• Subject

matter expert

LonsdaleSystems.com

Page 23: Testing the limits of testing

LonsdaleSystems.com 23

Early and extensive unit testing…

Page 24: Testing the limits of testing

LonsdaleSystems.com 24

Minimal unit testing…

Coding

System

Page 25: Testing the limits of testing

LonsdaleSystems.com 25

Minimal unit testing…

Project Deadline

System Test

Acceptance Test

Quality acceptable for user involvement

Page 26: Testing the limits of testing

LonsdaleSystems.com 26

Extensive unit testing…

Coding

System

Unit Testing

Page 27: Testing the limits of testing

LonsdaleSystems.com 27

Extensive unit testing…

Project Deadline

System Test

Acceptance Test

Quality acceptable for user involvement

Verify conforms to spec

Validate fit for intended purpose

Page 28: Testing the limits of testing

LonsdaleSystems.com 28

One in three developers dedicated to unit testing…

66%coding

full time

33%testing

full time

Page 29: Testing the limits of testing

LonsdaleSystems.com 29

Feature testing…

Page 30: Testing the limits of testing

LonsdaleSystems.com 30

Feature testing to build confidence…

Feature

Feature

Feature

Feature

Feature Testing

Page 31: Testing the limits of testing

LonsdaleSystems.com 31

Feature

Feature

Feature

Feature

Feature

Test Case

Test Case

Test Case

Test Case

Test Case

Modelling features…

Page 32: Testing the limits of testing

LonsdaleSystems.com 32

End-to-end testing…

Page 33: Testing the limits of testing

LonsdaleSystems.com 33

End-to-end verification of business rules…

Feature Feature Feature Feature

LonsdaleSystems.com

Page 34: Testing the limits of testing

LonsdaleSystems.com 34

Feature Feature Feature Feature

End-to-end verification of business rules…

LonsdaleSystems.com

Page 35: Testing the limits of testing

LonsdaleSystems.com 35

Modelling business rules…

State State

StateStateState

Feature

Feature

Feature

Choice

Page 36: Testing the limits of testing

LonsdaleSystems.com 36

Generating scenarios…

Scenario

State

Feature

Feature

Feature

Page 37: Testing the limits of testing

LonsdaleSystems.com 37

Date-based testing…

Page 38: Testing the limits of testing

LonsdaleSystems.com 38

Period 1 Period 2 Period 3Scenario

1

2

3

Feature

Feature

Feature

FeatureFeature

Feature

Feature

Feature

Feature

Feature

Feature Feature

Scenario A

Scenario B

Scenario C

Date-based testing…

Page 39: Testing the limits of testing

LonsdaleSystems.com 39

Exploratory testing…

Page 40: Testing the limits of testing

LonsdaleSystems.com 40

Encouraging testers to get creative…

• Testing skills• Subject matter expertise

• Build confidence• Verify business rules80%

20%

Test Cases

LonsdaleSystems.com

Page 41: Testing the limits of testing

LonsdaleSystems.com 41

Triaging bugs…

Page 42: Testing the limits of testing

LonsdaleSystems.com 42

Triaging bugs…

Spreadsheet

Spreadsheet

Spreadsheet

Test

Execution

JIRA

Page 43: Testing the limits of testing

LonsdaleSystems.com 43

Triaging bugs…

• Duplicates• Vague descriptions• Can’t duplicate• Tester misunderstanding

Bugs

Time

Report

ed

Resolved

Page 44: Testing the limits of testing

LonsdaleSystems.com 45

Coming in 2014

Still teaching software testing…

PracticalSoftwareTesting

LonsdaleSystems.com

LonsdaleSystems.com

Page 45: Testing the limits of testing

LonsdaleSystems.com 46

Questions

LonsdaleSystems.com