how do we test it? process/cmu... · white box testing white box testing access through ports and...

Many elements © Don Thomas, 2014, used with permission

!

18-545 — Advanced Digital Design ECE Department

Lecture

How do we test it?

08

with

cre

dit t

o G

. Lar

son

What’s up?

❏ Better yet, what’s not up? ❏ What you know

◆ Lots about how to write Verilog descriptions of designs ❏ What you don’t know

◆ Techniques and strategies to use for debugging and validating a design

◆ Most design time is spent here — more than designing it in the first place

◆ Validation Engineering — It’s often the first job/internship you will get in digital electronic system design

❏ So, how have you debugged your descriptions? ◆ Probably in haphazard ways!

Today's Agenda

❏ Theory of Testbenches ❏ FSM Testbenches (review) ❏ Randomization in Testbenches ❏ Testbench Potpourri

❏ Classes ❏ Tasks, Functions (again) ❏ Hierarchical Names ❏ Force/Release ❏ $display, $monitor, $strobe ❏ File I/O ❏ Foreach ❏ Finally!

Some common approaches…❏ Smoke test

◆ Let ‘er rip, watch the system crash and burn, even in the simulator ❏ Ask the TAs

◆ Assuring yourself that the testbench must be wrong ❏ Shoot some test vectors into the design

◆ Direct: Specific vector sequences even for something simple ◆ Random: Any number will do, just try something

❏ Check the pieces first ◆ Then check on piece integration next

❏ Indirect test ◆ Model checking — assertions

❏ Boot the OS ◆ That should check lots of stuff

❏ OK, these are some ways, but let’s step back a little…

How has Digital Design Changed?

❏ Majorly: TTL Handbooks — ABEL/ Verilog/VHDL — Cadence tools — Synopsys Synthesis — …

How has system debug changed?

System-on-Chip (or board)

❏ Virtual design methods are more widely used ◆ Language-based (model-driven)

simulation and synthesis ❏ Pre-Silicon validation

◆ Simulation ◆ Assertion based validation

❏ Post-Silicon validation ◆ Embedded hardware and software are

now being more widely used ◆ Components testing themselves and

other components — using both hardware and software

Why is this important?

❏ Software and validation are about 80% of total SoC development cost — about equally split

◆ Software — OS, middleware, firmware, drivers ◆ (Recent semi-quote from Lip-Bu Tan, CEO of Cadence) ◆ 40% each is high, more than twice the design costs

❏ If an SoC doesn’t work, you can’t sell it ◆ Not even in a garage sale ◆ Re-spins are in the 10s of millions of dollars for ASICs ◆ Less pricey for FPGAs but there are still NRE design

costs ❏ The validation aspect

◆ Do the design’s pieces work together? ◆ Not just “running some testbench vectors through”

Validation Engineer

❏ A designer who knows enough to realize what might be wrong or missed

◆ Can create a fault model listing what could be wrong ❏ A designer who can create a testbench to test against the fault

model ◆ Either a hardware or software testbench

❏ A designer who can measure how much of the fault model has been covered

◆ Functional coverage ❏ A designer who can “think different” ❏ Building a Verification Mindset

◆ In industry, there are teams of validation engineers ◆ Waking up every morning knowing there are bugs to find

❏ Aha! and embarrass the designers

Building a Validation Mindset

There’s a bug in the offense’s plan!

At 10K feet

❏ Pieces ◆ Device under test (“dut”) — what’s being tested – duh ◆ Stimulus generator — figures out what test vectors (stimulus)

to present to the dut ◆ Monitor and check — checks certain features and logs outputs ◆ Scoreboard — A check-off list of what still needs (to be)

tested. Figures out what needs closer testing

Design under test

Stimulus generator

Monitor and

checker

ScoreboardThe Testbench

A lot more smarts

❏ These elements imply a lot of smarts behind them ◆ a plan for the testing and validation ◆ a fault model — a list of faults that are to be checked ◆ a measure of functional coverage —the percent of faults checked ◆ a dynamically reactive capability — change the stimulus generator

to focus on remaining faults

The Testbench

Design under test

Stimulus generator

Monitor and

checker

ScoreboardThe Testbench

The next step in testbench building

❏ There are teams of verification engineers who build testbenches ◆ Playing defense ◆ Knowing there is a bug in the design somewhere and they’ll

find it (break anything!) and embarrass the designers ❏ Verification/validation engineering generalizes on this

◆ i.e., this structure (or similar) could be applied to any dut ◆ we won’t go there — at least not too far

❏ What we’ll do ◆ Consider ways to build the stimulus generator and monitor/

checker ◆ SV constructs and kernel stuff to support testbenches

Black box testing

❏ Black box testing ◆ Only have access

to the ports ◆ No (little)

knowledge of internal structures

❏ How come? ◆ Synthesis

generated the internals — don’t know what optimizations were used

◆ Tests can be used across parts in a family

testbench design

top

Normal port connections

White box testing

❏ White box testing ◆ Access through

ports and hierarchical naming

◆ Can see lots of internal structures

❏ How come? ◆ Can test lots more

of the details ◆ Good but doesn’t

necessarily cross to new versions of a system

testbench design

top


Gray box testing

❏ Gray box testing ◆ Somewhere in the

middle between black and white

testbench design

top


Styles of checking: Direct

❏ Direct — apply vectors, check results (maybe some time later)

❏ Think of something that might go wrong

◆ Aha! The ALU carry chain might not work

❏ Think of how to test for that specific problem

◆ What values can I add to check that it works: -1 + 1

❏ Just two values? or all pairs? or just some? — then which?

◆ Select specific values — typically specific corner cases

◆ Select random values — typically general case

design

testbench

Expected results

Actual results

OK

Test vectors

It’s never quite this clean…

Styles of checking: Indirect

❏ Model Checking ◆ A piece of the design is checked

against a model while other tests are being run

◆ Sometimes called assertion-based ❏ Typically a general check

◆ Assert that “a always equals b + c” ◆ If it doesn’t, report an error!

❏ How’s it work? ◆ You’re testing other things and,

oops, the alu screwed up! ❏ This would be part of the monitor

box in earlier diagram

design

testbench

Expected results

Actual results

OK

Test vectors

assert property (a == b + c)

Testbench Stuff

❏ The testbench includes ◆ how you want to test your system

❏ reset, reading in data files, printing stuff, logs, … ◆ the rest of the world — the environment surrounding your design

❏ clocks, other designs not yet completed, upstream/downstream systems

❏ It’s not synthesizable, it’s just time-based behavior ◆ just about anything — programming-wise — goes

❏ Today, just some basics ◆ fault models — what types of things can go wrong? ◆ random test generation and constraints

❏ Other important concepts for the future ◆ assertions — what things should always be true? ◆ object-oriented subset of the language

SVbook 7.3

Testbench

❏ Basic organizations

top

tb design top

tb code

design

top

tb code1

design

tb code2

top

tb code

design

tb code

instantiated module

Today's Agenda



0 0 0 1

1 0

A

C

B

R 1/00

1/10

0/010/01

s1 s0Legend

i/a,b

x/00

Testing an FSM

❏ Reset the system, and follow a sequence of states ◆ These should lead your design through

a path of its own states ◆ Do this multiple times, following

different paths (eventually all paths) ◆ You will probably want to be able to

reset the design under test (“dut”) without resetting your testbench

❏ The answer is no. What’s missing? ◆ …

Is this path a complete check?

0 0 0 1

1 0

A

C

B

R 1/00

1/10

0/010/01

s1 s0Legend

i/a,b

x/00

How does the sequence happen?

❏ Testbench has to generate the right inputs (i for this FSM) plus clock, reset ◆ … in the right sequence

❏ Implicit FSM ◆ This is a style of writing a SystemVerilog

description ◆ More abstract, less error-prone than with

all the details — no modules to instantiate or wire

◆ More amenable to making inline checks

Big idea: Testbench must be an FSM!

Implicit FSM — How?

❏ Write a state machine procedurally ◆ use “@(posedge ck)” for testing FSMs

❏ Defining a state ◆ Each “@posedge clock” defines a new state and what values

should appear ◆ The code after the @posedge clock is what executes and updates

at the clock edge ❏ just like when writing a dFF

◆ outputs of the FSM use non-blocking assignments (<=) ❏ After all, it’s another FSM, just written in a different style

The beginning of an implicit FSM

module TB; initial begin clock = 0; forever #5 clock = ~clock; end

initial begin $monitor (blah blah); // only one of these reset_l = 0;

i = 0 reset_l <= #1 1;

@(posedge clock); //any statement here will execute at time 5 i <= 1;

@(posedge clock); // time 15 i <= 0;

@(posedge clock); … $finish;

end

endmodule

clock

0 5 10 15

0 0 0 1

1 0

A

C

B

R 1/00

1/10

0/010/01

s1 s0Legend

i/a,b

x/00

i equals 1

Timing between testbench and designmodule TB; initial begin clock = 0; forever #5 clock = ~clock; end initial begin $monitor (blah blah); // only one of these reset_l = 0;

i = 0 reset_l <= #1 1;



@(posedge clock); …

end

endmodule

module explicitFSM (output logic a, b, input ck, r_l, i); // output logic not shown always_comb case (state) A: nextState = (i) ? B : A; B: nextState = C; C: nextState = (~i): A : C; default: nextState = A; endcase always_ff @(posedge ck, negedge r_l) if (~r_l) state <= A; else state <= nextState; endmodule: explicitFSM

What happens simultaneously

with this?

0 0 0 1

1 0

A

C

B

R 1/00

1/10

0/010/01

s1 s0Legend

i/a,b

x/00

0

0

0

i

state

nextState

5 15

A

B

A

A

B

C

What this boils down to

module TB; initial begin clock = 0; forever #5 clock = ~clock; end initial begin $monitor (blah blah); // only one of these reset_l = 0;

i = 0 reset_l <= #1 1;




end

endmodule


clock

D Q

D Q i1 (or 0)

nextState state

TB

FSM

What happens at the clock edge?

there’s a clock edge at time 5

clk becomes 1, always_FF and initial run,

evaluates state and i, schedules non-blocking

update.

all within time 5

module explicitFSM (output logic a, b, input ck, r_l, i); … always_comb case (state) A: nextState = (i) ? B : A; B: nextState = C; C: nextState = (~i): A : C; default: nextState = A; endcase always_ff @(posedge ck, negedge r_l) if (~r_l) state <= A; else state <= nextState; endmodule: explicitFSM

both always_comb

blocks execute, in

arbitrary order

module TB; initial begin clock = 0; forever #5 clock = ~clock; end initial begin @(posedge clock);

//any statement here will execute at time 5 i <= 1;

…

Nothing else to do, state and i are updated simultaneously

Oh my, what about this?

module TB; initial begin clock = 0; forever #5 clock = ~clock; end initial begin $monitor (blah blah); // only one of these reset_l = 0;

i = 0 reset_l <= #1 1;

@(posedge clock); //any statement here will execute at time 5 i = 1;

@(posedge clock); // time 15 i = 0;


end

endmodule

module explicitFSM (output logic a, b, input ck, r_l, i); // output logic not shown always_ff @(posedge ck, negedge r_l) if (~r_l) state <= A; else state <= i; endmodule: explicitFSM

What could happen?

clock

D Q

D Q i1 (or 0)

i state

?

Note, the design changed for this

illustration

Consider the situation where the simulator updates i first and state second… (not simultaneously)

TB

FSM

How do you check the output’s value?module TB; initial begin clock = 0; forever #5 clock = ~clock; end initial begin $monitor (blah blah); // only one of these reset_l = 0;

i = 0 reset_l <= #1 1;

@(posedge clock); //any statement here will execute at time 5 i <= 1; #1 if (a != 1’b0) $display (“oops”);



end

endmodule


0 0 0 1

1 0

A

C

B

R 1/00

1/10

0/010/01

s1 s0Legend

i/a,b

x/00

0

0

0

i

state

nextState

5 15

A

B

A

A

B

C

Here

Thought you said not to put #1 in your code — something must be wrong! For checking, this is OK.

There are better ways (see SVbook)

Today's Agenda



A random slide

❏ Random numbers are widely used in testing ◆ Shown to be as good as any for general situations

❏ Could have used randoms instead of 1s and 0s in previous algorithm

◆ Have seen $urandom previously, returns a random value ◆ $urandom_range(1049, 17) returns value in the range

❏ Can declare SV types to take on random values ◆ rand — declares a variable that when randomized will take on

one of the 2n possible values, with replacement. ❏ each next value is chose from the whole range of 2n values

◆ randc — declares a variable that when randomized will take on one of the 2n possible values, without replacement

❏ think of drawing from a deck of cards ◆ No x or z values ◆ Synthesizable? Ha! Not at all

◆ Where would the RNG hardware be?

rand bit [3:0] val;

randc bit [4:0] valB;

Setting up an example

❏ We have a memory controller with 256 pages of 256 16-bit words

◆ We only have one memory (256 words) in the system

module memController #(parameter logic [7:0] page = 8'h02) (inout tri [15:0] addrData, input logic addrValid, rw, clk, reset);

During read, addrData driven

by memController in states B-E

During write, addrData driven by processor in

states B-E

Testbench needed

❏ Our testbench uses random numbers to determine: ◆ The address to write to and the values to write ◆ When to read or write, and whether to try the wrong page

❏ Start with a class definition — SV has classes! ◆ If there are rand(c) variables in the class, a randomize method is

available

class memPkt;rand bit [7:0] TBaddr;rand bit [15:0] TBvalue[4];randc bit [3:0] coinFlip;

constraint A {TBaddr inside {[0:252]};}endclass

memPkt pkt = new;

A random address to read/write

An array of 4 values to write

readwrite coin flip

You know, like in

software

A Constraint specification❏ Uses a set operator

◆ inside {…} ◆ What’s specified is a range [0:252] ◆ Could have specified {3, 17, [35:44]} as an alternate

❏ When the random numbers are generated, the constraint(s) is applied ◆ Why 252? So that an address doesn’t cross a page boundary


constraint A {TBaddr inside {[0:252]};}endclass

memPkt pkt = new;A set

operatorThe set

Our write_memory task (sort of)

task wMem;logic [7:0] myAD, i, pageNum;if (~pkt.randomize())

$display("oops, bad random numbers");if (pkt.coinFlip > 4'b0011)

pageNum = 8'h02;else pageNum = pkt.coinFlip;

…

Get a new set of random

numbers for all values above


constraint A {TBaddr inside {[0:252]};}

Most of the time access

page 2

Otherwise access pages 3-15 randomly

write_memory task, continued

❏ This could also keep track of what’s written ◆ Check if correct when read back ◆ Reads and writes randomly

❏ But what does it check? ◆ Go back to slide “What might go wrong here”

@(posedge clk);busDrive <= 1;ad <= { pageNum, pkt.TBaddr };addrValid <= 1;rw <= 0;

@(posedge clk);for (int i = 0 ; i < 4; i++) begin

addrValid <= 0;ad <= pkt.TBvalue[i];@(posedge clk);

end …

send address, including the

random address in first step of

protocol

then send the four data elements

Yet another random slide

❏ Can constrain random variables at each usage

◆ Here’s where randomize could fail

class vectors rand bit [3:0] valA; randc bit [4:0] valB; endclass

vectors tv = new; … if (~tv.randomize() with {valB < valA}) $display (“OOPS, randomize didn’t work”);

valB must take on all values, some aren’t less than

valA

“with” allows you to add constraints to those specified in the class

Multiple series of random numbersmodule randStuff (); bit [3:0] myT, myR; class TimeGen; rand bit [3:0] numWaits; constraint EVEN {numWaits[0] == 0;} function [3:0] getT (TimeGen T); if (T.randomize() == 1) getT = T.numWaits; else $display("Randomize had constraint problems."); endfunction

endclass initial begin TimeGen T = new, R = new; repeat (16) begin myT = T.getT(T); myR = R.getT(R); $display ("time = %d, %d", myT, myR); end end endmodule

time = 14, 6 time = 2, 0 time = 4, 12 time = 8, 0 time = 2, 14 time = 6, 10 time = 4, 4 time = 2, 8 time = 12, 2 time = 6, 6 time = 2, 14 time = 4, 8 time = 12, 12 time = 0, 2 time = 6, 8 time = 14, 14

Two series of random numbers

Random with enumerate

❏ Picks labels out of enumerations

typedef enum bit[2:0] {R, O, Y, G, B, I, V} roy_g_biv_t;

class greatColors; rand roy_g_biv myRainbow; endclass

greatColors gc = new;

Now when you call randomize, it will pick from

the enumerated labels

Today's Agenda



Testbenches for FSM-Ds

❏Tests ◆ not typically

characterized by single test vectors

❏Need to ◆ Send packets of info ◆ Follow protocols

❏ Use the old standby ◆ Assert go ◆ Send data to add ◆ When data == 0, set

done

Random info to send to sumItUp

❏Need to create packets of numbers to add ◆ This will send between 2 and 10 items to add ◆ and generate the values in array item[ ]

class sumPkt;rand bit [15:0] howMany;rand bit [15:0] item[];

constraint N {howMany inside {[2:10]};}

constraint arraySize {item.size == howMany;}endclass

A class with methods (tasks, fns)class testSumThread;

local bit unsigned [15:0] total;local sumPkt pkt = new;

task sendPktToThread; total = 0;if (~pkt.randomize()) $display("oops");for (byte i = 0; i < pkt.howMany; i++) begin

@(posedge ck);inA <= pkt.item[i];total += pkt.item[i];go_l <= (i == 0) ? 0 : 1;

end@(posedge ck);inA <= 0;

endtask

function checkTotal (bit unsigned [15:0] value);return value == total;

endfunctionendclass

Create instance of pkt to send

Fill in random numbers

Send values, keep track of total

Send 0 at end

Here’s a way to check the FSM-D

❏ Instantiate instance of testSumThread ◆ Send 100 packets to it – call sendPktToThread ◆ Check if total is correct

initial begin testSumThread t = new;for (int i = 0; i < 100; i++) begin

t.sendPktToThread();wait (done);if(~t.checkTotal(sum))

$display("OOPS"); end$finish;

end

Tasks and functions — revisited

❏ Both are subroutines ◆ Tasks can have timing operators

(#, @, wait), functions cannot ❏ Basic operations

◆ Arguments can be defined ◆ When called, inputs and inouts are copied into the

task/function ◆ On return, outputs and inouts are copied back to

calling site ◆ Function return value can be used

in an expression (e.g., in assign = …) ❏ Functions

◆ function type name (args)…endfunction ◆ If recursive, include “automatic” before type ◆ Functions can access vars not in port list (ouch, ugly

side effects!)

function bit[2:0] incB(input bit [2:0] b);incB = b + 1;

endfunction

return type name

value returned

assign a = incB(y) + z;

Tasks and functions — revisited

❏ Tasks don’t return a value ◆ except through output or inout arguments

❏ Since tasks can have timing operators… ◆ a single task can be called from more than one

process (initial block) ◆ In this case, they would share any variables

declared in the task ◆ A task can be declared as automatic

❏ Then each process has its own variables ❏ Inputs can be declared as ref

◆ Like passing a pointer to the function or task ◆ No difference at calling site ◆ function or task can change the variable by

reference too ◆ often used to pass arrays

task name (args);…

endtask

function bit[2:0] incB(input ref bit [2:0] b);incB = b + 1;

endfunction

b is passed by reference (pointer)

“automatic”

Hierarchical name refs

Hierarchical Names

❏ White box testing ◆ You can see

inside ❏ It’s all there but how

to access? ❏ Hierarchical name

references ◆ You can directly

access the leaf cells

◆ Can search “up” and then back “down”

◆ Lots of details left out

testbench design

top


logic a

a

b

c

$display (“aDownThere=%h”, top.design.a.b.c.a);

display(…);

You can do more than just print

❏ You can actually override values within the design

❏ Called force and release

❏ Use with hierarchical names to debug pieces

testbench design

top


Debugging with Force/Release

❏ These are used to temporarily override a net or register ◆ If sum is a “wire”, e.g., output of a gate primitive, assign, or

always_comb ❏ then force overrides the other drivers of the wire are

ignored after the force. ❏ When release is executed, the other drivers take over

◆ If sum is a “register,” e.g., output of an always_ff ❏ then force causes sum to hold the new forced value. It

can’t be overwritten by = or <=. ❏ When release is executed, this new value is held until

the register is written again. Acts like an async reset in a typical always_ff initial begin

… force sum = 0; … release sum; end

Debugging with Force/Release

❏ Can do this in “procedural continuous assign”

◆ That is, using assign in procedural block ◆ You may have accidentally tried this ◆ Essentially acts like force and release ◆ Don’t ! ◆ Will probably be deprecated in the future.

❏ Details — Back to force and release ◆ Force and release are procedural

❏ found in initial blocks of TBs only ◆ Can reference hierarchical names

❏ override anything anywhere! ◆ They are not synthesizable ◆ Can be executed on some simulator

command lines

initial begin … assign sum = q + r; … deassign sum; end

always_comb begin … sum = a + b; … end

Force/Release Scenario❏ Instead of injecting faults, isolate logic

◆ Isolate some logic without having to figure out how to set some of the other logic.

◆ e.g., it may be difficult to figure out the inputs needed to set B, and C to these values. — so just do it with force/release

◆ Debugging only — these don’t synthesize

logic

Lots of ugly logic logic

A B C

Many layers of instantiationmodule testbench initial begin… force B = 1; force C = 1; … // now A and // inputSig control // “logic”; … release B; release C; …

inputSig

Displaying information

Printing task When it prints$display Right now, with the values of the vars

currently existing$monitor At the end of the time step after all

values have settled to their final value$strobe Called like $display — from in any

procedural context — but prints after values have settled to their final value. (at the “monitor events” time)

always_ff @(posedge clk) begin a <= a + b; $display (“a=%h, b=%h, a, b); $strobe (“a=%h, b=%h, a, b); end

prints value before non-blocking

update

prints value of a after non-blocking

assignment resolved

File I/O

❏ Looks a lot like file I/O in C ◆ Although has a multichannel file descriptor, not shown here ◆ Some in book, all in manual

int fileDescriptor;

initial begin fileDescriptor = $fopen(“file.txt”, “w”); if (fileDescriptor == 0) $display(“oops, disk crashed, your files gone, ha ha!”); … $fdisplay (fileDescriptor, “a=%h, b=%d”, a, b); // could also $fmonitor (), $fstrobe () … $fclose (fileDescriptor); end

foreach loops

❏ Who needs counter variables? (who hates to type?) ◆ foreach — used to iterate over array elements ◆ Can be multidimensional

module tryForeach; string speak [2] = '{ "hello", "logic"}; logic [13:0] tvectors [2] = ' {14'b11110000111100, 14'b00001111000011}; logic [13:0] dutInput;

initial begin $monitor ($stime, " dutInput = %b", dutInput);

foreach (speak [j]) $display(speak[j]);

foreach (tvectors [i]) begin dutInput = tvectors[i]; #1; end for (int j = 0; j <= 1; j++) $display(speak[j]); end endmodule: tryForeach

hello logic 0 dutInput = 11110000111100 1 dutInput = 00001111000011 hello logic

final block

❏ It runs after everything else has finished, or $finish has been called

❏ Like an always_x or initial block, introduces procedural context ❏ Simulation only — all must finish in a single simulation cycle

◆ i.e., no #, @, or wait

final begin //print summary data $display (“executed %d instructions”, numInst); … //do other cleanup things like: $fclose (fileDescriptor); end

Today's Agenda



how do we test it? process/cmu... · white box testing white box testing access through ports and...

Documents