Redefining Boundaries ofBoundary Testing

James Bach

Satisfice, Inc.

james@satisfice.com

This material benefited from review and conversation at the 4th Workshop on Heuristic and Exploratory Techniques.

The following people contributed to that conference: Jon Bach, James Bach, Robert Sabourin, Karen Johnson, Cem Kaner, Henrik Andersson, Keith Stobie, Scott Barber, David Gilbert, Doug Hoffman, Mike Kelly, Harry

Robinson, Ross Collard, Dawn Haynes, Timothy Coulter, and Michael Bolton

4.3.2 Boundary value analysis (K3)

“Behavior at the edge of each equivalence partition is more likely to be incorrect, so boundaries are an area where testing is likely to yield defects. The maximum and minimum values of a partition are its boundary values. When designing test cases, a value on each boundary is chosen.”

ISTQB Foundation Syllabus

Scenario:

Dear Tester,

I just added a document description field to the metadata (“File | Properties”). It should take up to 32768 bytes of text, and truncate anything more than that.

Thanks!

32K ?

32K

Ambiguity in units and limits

(64K)

32K Ambiguity in units and limits Foreign boundaries (upstream)

Unknown boundaries

?64K

32K

Ambiguity in units and limits

32K Ambiguity in units and limits Foreign boundaries (upstream) Dynamic boundaries

Unknown boundaries

?

(64K+)

64K+

32K

(64K+) 32K (16K)

32K Ambiguity in units and limits Foreign boundaries (upstream/downstream) Dynamic boundaries

Unknown boundaries

16K ?64K+

32K(64K+) (16K)255

32K16K Ambiguity in units and limits Foreign boundaries (upstream/downstream) Dynamic boundaries Quasi-functional boundaries

Unknown boundaries Undeclared boundaries

255 ?64K+

32K(64K+) (16K)255

32K16K255 Ambiguity in units and limits Foreign boundaries (upstream/downstream) Dynamic boundaries Quasi-functional boundaries

Unknown boundaries Undeclared boundaries Mis-declared boundaries

(2K)

?64K+2K

The Boundary Risk Hypothesis:

All other things being equal, any given thing is more likely to be misclassified, mishandled, or misdirected when “near” a suspected boundary than when “far away.”

This hypothesis arises because we areaware of specific mechanisms of boundary-related failure,

over and above other mechanisms.

But the actual boundaries in a productmay not be the ones we are told about.

That’s why we must explore.

Toward a Better Definition of Boundary Testing

Boundary testing: Any testing to the extent that it involves the discovery or evaluation of boundaries or boundary-related behavior.

Boundary: 1. A dividing point between two otherwise contiguous regions of behavior; or 2. A principle or mechanism by which things are classified into different sets.

To Test Boundaries WellFocus and De-focus

De-focusing strategy for boundary testing Don’t “believe” what you read about specific boundaries, nor what you are told. Study product behavior while driving inputs/outputs toward infinity and zero.

(Stopping heuristics: flat line, sufficient drama, or cost exceeds value.) Look for subtle clues: performance degradation; obscure data corruption; garbage. You never know that a boundary test is finished. The failure may not become

manifest right away, and not necessarily in the function you are testing at that moment. Explore around.

Focusing strategy for boundary testing Study product claims about boundaries. Interview programmers about boundaries. Don’t just test on the boundary, test in “epsilons” off the boundary. Envision data moving through a multi-path maze of “boundary gates” in the product.

Are those gates consistent with each other, and with the purposes of the product? Push boundaries while considering all known routes data may take in their lifecycle. Pay special attention to interactions between sub-systems written by different people. Perform bisection search to home in on specific boundaries.

http://www.satisfice.com/presentations/boundary-testing.ppt