Structural Coverage

•Verilog code is available to help generate testsoCode can be analyzed statically and/or simulated

•Easier to detect “additive” design errorsoTests what the code actually does, not what it is supposed to do

•Possible to automate test generation

Structural Coverage Metrics

• Coverage metric is a numerical measurement of the adequacy of an arbitrary test set for verification.

• High coverage is a goal for test generation• The measure is based on code execution

1. Toggle Coverage - Fraction of signals which have toggled

2. Statement coverage - Fraction of statements executed

3. Branch coverage - Fraction of branch directions executed

4. Path coverage - Fraction of control-flow paths executed

Toggle Coverage

•Counts the fraction of signals/variables whose bits have toggled

ex. x = 4b0000 and x=4b0111, 75% toggle coverage

•Easy to compute

•Little functional meaning (came from manufacturing/memory test)

Statement Coverage

•Ensure that the “code” is completely covered during testing•Statement Coverage: Requires that all statements are executed

a = in1 + in2;b = 0; c = 0;while (c < a) c = c + in1;if (c < in2) out = a + b;else out = a + c;

a = in1 + in2;b = 0; c = 0;

c<a

c<in2c = c + in1;

out = a + c; out = a + b;

Test 1: in1, in2 = 0,0Test 2: in1, in2 = -1, 1

TF

TF0,0

-1,1

Branch Coverage

•Branch Coverage: Requires that all branch directions are executed•2 branches, 4 possibilities, 3 are executed, 75% branch coverage

a = in1 + in2;b = 0; c = 0;while (c < a) c = c + in1;if (c < in2) out = a + b;else out = a + c;

a = in1 + in2;b = 0; c = 0;

c<a

c<in2c = c + in1;

out = a + c; out = a + b;

Test 1: in1=0, in2=0Test 2: in1=-1, in2=1

TF

TF0,0

-1,1

Test Sequence for Higher Coverage

•100% statement and branch coverage•High coverage goal directs test generation

a = in1 + in2;b = 0; c = 0;while (c < a) c = c + in1;if (c < in2) out = a + b;else out = a + c;

a = in1 + in2;b = 0; c = 0;

c<a

c<in2c = c + in1;

out = a + c; out = a + b;

Test 1: in1,in2 = 1,0Test 2: in1,in2 = -1,1

TF

TF

-1,1 1,0

Path Coverage

•Each pattern covers a different control path•Infinite number of paths with a loop

a = in1 + in2;b = 0; c = 0;

c<a

c<in2c = c + in1;

out = a + c; out = a + b;

TF

TF

-1,11,0

0,0

Domain (Boundary) Coverage

•Each conditional predicate defines partitions the input space•A small error may move the partition

foo (in1, in2) if (in1 < in2) out = 1; else out = 0;

in1

in2

out=0

out=1

•What if in1 < in2 should be in1 <= in2?

•What if in1 or in2 has a slightly incorrect value?

Domain Test Goal

•Test with input data on the boundary and just on either side of the boundary

foo (in1, in2) if (in1 < in2) out = 1; else out = 0;

in1

in2

out=0

out=1

•Test 1: in1, in2 = 1,1 - False, on the boundary

•Test 2: in1, in2 = 1,2 - True, just off the boundary

1

1

2

Domain Test Example

a = in1 + in2;b = 0; c = 0;while (c < a) c = c + in1;if (c < in2) out = a + b;else out = a + c;

Test 1: in1, in2 = 0,0•Sets c = 0, and a = 0•False and on-boundary for c < a•False and on-boundary for c < in2

Test 2: in1, in2 = 1,0•Sets c = 0, and a = 0• True and near-boundary for c < a

•c set to 1 in loop•False and on-boundary for c < in2

•Need to satisfy 2 conditions for each predicate:True and on/near-boundaryFalse and on/near-boundary

•3 of 4 conditions satisfied, 75% coverage

Domain Test Complications

•Non-standard Domain Boundaries

Piecewise Linear Boundaries: Test each linear piece separately

(in2 > in1) and (in1 > 2)

in1

in2

2Higher Degree Boundaries: No standard solution

in1

in2(in2 > in12)

Finite State Machine Metrics

•These metrics assume that the HDL is written in an FSM styleoA ‘case’ statement which switches on a single ‘state’ varoEach control path assigns the state variable

case state iswhen 0 => x = 0;

state = 3;when 1 => x = 1;

state = 2;..

•State Coverage - All states are entered•Transition Coverage = All transitions are traversed

a = in1 + in2;b = 0; c = 0;while (c < a) c = c + in1;if (c < in2) out = a + b;else out = a + c; out = a - c;

Erroneous statement

•Test 0,0 executes erroneous statement but does not detect the error•Detection constraint includes a - c != a + c

Optimism in a Coverage Metric

•100% statement coverage does not guarantee error detection•Test constraints may not be the same as detection constraints

Pessimism in a Coverage Metric

•100% coverage may not be possibleFault detection conditions may not be satisfiableUndetectable faults are redundant, can be ignoredIdentifying redundant faults in NP-complete

a = in1 + in2;b = 0; c = 0;

c<a

c<in2c = c + in1;

out = a + c; out = a + b;

TF

TF

•This path cannot be executed

•Sub-100% coverage is misleading

Functional Coverage

•Test goals are defined based on the features in the specification

•The implementation (HDL) is not used to create tests

•Coverage events are “interesting” events which might reveal errors

•Coverage events must be defined manually because they are extracted from the specification

Ex. Did overflow occur? Was x=17? Did an add follow a subtract?

Functional Coverage Results

add

sub

shl

shr

noop

illegal

port1 port2 port3 port4

X

X

X

X

•Status report indicates which events have been covered (and how often)

•Can compute the cross product of events to get complex events•Many such events may be impossible or uninteresting