Random Testing

Tor StålhaneJonas G. Brustad

What is random testing

The principle of random testing is simple and can be described as follows:1.For each input parameter, generate a random but legal value.2.Apply the full set of inputs to the SUT3.Register the result and go back to step 1.

Chen’s observation

Inputs that are close to each others in the input domain tends to go through the same path.

Thus, in order to find most of the errors, we should spread the test cases as much as possible.

This approach is called Adaptive Random Testing. We will look at four approaches:• Partition Adaptive Random Testing• Basic Random Testing – RT • Basic Adaptive Random Testing – ART• Mirror Adaptive Random Testing – MART

Block failure pattern

Strip failure pattern

Point failure pattern

Some notation • D: input domain size• n: number of test cases• m: number of tests that fail• F = D/m. Note that– Large F => small m => few errors detected– Small F => large m => many errors detected

• = 1/F – failure rate • Frel = Fobs *

Partition Adaptive Random Testing

Tor StålhaneJonas G. Brustad

ART by random partitioning – 1 Algorithm for a two-dimensional case1.Start with C = {(Xmin, Ymin), (Xmax, Ymax)}

2.Draw a random point in C = (X1, Y1). This will split C into four regions – R1, R2, R3 and R4. Select T = max area {R1, R2, R3, R4}See next slide.

ART by random partitioning – 2 (Xmax, Ymax)

{(Xmin, Ymin),

(X1, Y1)

R1 T = R2

R3 R4

Select a test (X1, Y1) in T. If it is a failure, report the failure and stop.Otherwise, split T in the same way as we split C – see nextslide

ART by random partitioning – 3

Select a test (X2, Y2) in T. If it is a failure, report the failure and stop.Otherwise, repeatthe process

(Xmax, Ymax)

{(Xmin, Ymin),

(X1, Y1)

R1 T

R3 R4

(X2, Y2)

ART by bisection – 1 Algorithm for a two-dimensional case1.Start with C = {(Xmin, Ymin), (Xmax, Ymax)}

2.Draw a random test in C – (X1, Y1). If it fails we are finished. Otherwise split C in two equal parts – see next slide.

ART by bisection – 2 (Xmax, Ymax)

{(Xmin, Ymin),

(X1, Y1)

Select a test (X2, Y2) in the untested half of C. If it is a failure, report the failure and stop.Otherwise, split C again – see nextslide

(X2, Y2)

ART by bisection – 3

Select a test (X2, Y2) in T. If it is a failure report, the failure and stop.Otherwise, repeatthe process for eachpart that we havenot tested – (X3, Y3) and (X4, Y4)

(Xmax, Ymax)

{(Xmin, Ymin),

(X1, Y1)

(X2, Y2)

(X4, Y4)

(X3, Y3)

The exclusion factor – 1

All types of Adaptive Random Testing (ART) can be improved by introducing the exclusion factor, usually denoted by f. This factor will force the new tests away from the tests that have already been run.The optimal factor value will vary, depending on the failure rate and on the failure pattern – block, strip or point.

The exclusion factor – 2

The exclusion factor – 3

Based on this, we have chosen f = 0.4 as the best value.

Comparisons

Basic Adaptive Random Testing

Tor StålhaneJonas G. Brustad

Fixed Size Candidate Set - FSCS

Max distance – 1

Let a and b be two n-dimensional inputs {a1, a2, ..., an} and {b1, b2, ..., bn}.

m

iii babadist

1

2)(),(

E.g. in a two-dimensional space we have the twoparameters a ={1, 2} and b = {2, 5}. Then we havedist(a, b) = sqrt(1 + 9) = 3.16

Max distance – 2 Let T and C be two disjoint sets T = {t1, t2, ...tn} is the set of executed testsC = {c1, c2, ...ck} is the candidate set.Find the ch that satisfies:

),(min),(min},,...2,1{ 11 ijniih

ni tcdisttcdistkjfor

This criterion will spread the test cases evenly by finding the largest minimum distance betweenthe next test case – selected from C – and the already executed test cases in T.

Max distance algorithm

A small example – 1 We have two data sets:•T = {(1, 1), (3,4)} – already executed tests•C = {(1, 2), (3, 1)} – candidate test set

3

2

1

1

2 3

t1c2

c1

4 t2

A small example – 2

Using the max distance algorithm we get:•j = 1 => (c1, t1) dist = 1.0, (c1, t2) dist = 2.8

•j = 2 => (c2, t1) dist = 2.0, (c2, t2) dist = 3.0

•min(dist) = 1.0 and the first distance larger than min(dist) is 2.0 => Next test is c2 = (3, 1)

Test comparison – 1

Defect types seeded:• AOR: Arithmetic Operator Replacement• ROR: Relational Operator Replacement• SVR: Scalar Variable Replacement• CR: Constant Replacement

Results with RT

1

Results with ART (a) and FSCS (r)

N

Test comparison – 2

r

ar

F

FFI

Mirror Adaptive Random Testing

Tor StålhaneJonas G. Brustad

The problem with ART

All versions of ART require a large amount of computations due to the distance calculations and comparisons.The MART – Mirror ART – is simpler and requires less computation.

The MART procedure The procedure has four steps:1.Partition the input domain into m disjoint subdomains. One is chosen as source subdomain. The rest are mirror subdomains 2.Apply the D-ART process to generate the next test case from the source subdomain. Execute this test case and quit if we find a defect.3.Apply the mirror function to the test case from step 2 to generate a test case for each mirror subdomain. Execute the test cases in sequential order and stop when we find a defect.4.Repeat steps 2 and 3 until finding the first failure or until reaching the stopping condition.

Mirror partitioning Below we see several ways to create mirror partitions:X2Y1 => X is bisected, y is unchangedX2Y2 => both X and Y are bisectedX4Y2 => X is split into four parts, Y is bisectedX4Y1 => X is split into four parts, Y is unchanged

The D-ART process

),(min),(min},,...2,1{ 11 irniij

ni EcdistEcdistkrfor

1. Set E to be the empty set2. Select a random test case from the input domain

and execute it. If no failure, add the test case to E, otherwise stop.

3. Construct C = {c1, c2,…, ck}, where all ci are randomly selected and E and C are disjoint.

4. Let n = |E| and select cj so that

5. Repeat steps 3 ad 4 until first defect is found