Slide 1

Coverage Discounting: Improved Testbench Qualification by Combining Mutation Analysis with Functional Coverage Nicole Lesperance, Peter Lisherness, and Tim Cheng Univ. of California, Santa Barbara September 9th, 2013 1

Slide 2

Verification Questions 1.Have the test vectors adequately stimulated the design? 2.Is the testbench able to identify errors? 2

Slide 3

Motivation 3 Functional Coverage Mutation Analysis Coverage Discounting + Functionally Meaningful + Evaluates design activation -Ignores propagation and checker quality + Evaluates propagation and checker quality - Mutants hard to analyze + Functionally Meaningful + Evaluates design activation, propagation and checkers Functional Coverage Mutation Analysis Coverage Discounting

Slide 4

Coverage Discounting: The Main Idea Use fault insertion to discount (revise downward) coverage scores 4 Detected + undetected mutants Functional Coverage Mutation Analysis Coverage Discounting Undetected Mutants Coverage Score 100% 0% Discounted Coverage Score

Slide 5

An Existing Solution: Mutation Analysis DUT Tests Checker Undetected Fix Add/Fix Detected Benefits Evaluates quality of testbench checkers Indicates if tests fail to propagate potential errors to the checker or fail to activate mutants 5

Slide 6

Mutation Analysis Drawbacks 1.Long runtime for simulation 2.Many man-hours required to analyze results: 2.1 Mutants are synthetic - difficult to identify testbench improvements to target mutant detection 2.2Redundant mutants never detectable: Some undetected faults are redundant if (a > 0) b=1+a; else b=1-a; if (a >= 0) b=1+a; else b=1-a; Example: 6

Slide 7

Premise of Coverage Discounting 1) A mutant may change design functionality 2) A coverpoint, covered by testbench in the original design, may be suppressed (i.e. no longer be covered) in the mutated design 3) If the mutant suppressing the coverpoint is not detected, then the coverpoint is no longer considered covered in the original design 7

Slide 8

Coverage Discounting Flow 8 DUT Tests Checker Detected Undetected Add more tests No Yes Fix Coverage Changed?

Slide 9

Experimental Results 9

Slide 10

OpenRISC CPU Experiment #1: Measure test quality Tests: Instruction-Set v. Random Coverpoints: Opcodes Experiment #2: Identify checker weaknesses Tests: SoC Regression Suite Coverpoints: CPU top-level signals 10

Slide 11

Measure Test Quality Functional Tests Original Discounted Random Tests 11

Slide 12

Identify Checker Weaknesses Tests 12

Slide 13

Identify Checker Weaknesses 13

Slide 14

UART Illustrates success of coverage discounting applied to a sophisticated testbench Simulation Infrastructure: Design RTL w/OVM Testbench SystemVerilog coverpoints (ModelSim) Certitude fault insertion Tests: OVM Sequences Coverpoints: Hand-written spec-based 14

Slide 15

UART Results Reduces debug effort from analysis of 146 mutants to examining 3 functional coverpoints 1588 Mutants: 7 not activated, 106 not propagated, 33 not detected Total 146 mutants demand attention 846 Coverpoints: 4 uncovered 3 discounted All discounted relate to specific unchecked functions Loopback, timeout interrupt identification register 15

Slide 16

Results Examined Coverage discounting: Identified "good" tests (propagation) Diagnosed checker problems Identified coverpoints vacuously covered However: Runtime of mutation analysis still an issue Technique sensitive to the quantity and quality of faults 16

Slide 17

Confidence Metric Aims to answer: 1.Is a coverpoint adequately challenged by the set of mutants? 2.When have enough faults been inserted (when can simulation be stopped)? 3.What is the optimal simulation ordering of mutants for coverage discounting? 17

Slide 18

Quicker Confidence CPUUART 18

Slide 19

Quicker Discounting CPUUART 19

Slide 20

Technology Transfer Working closely with Freescale to apply discounting technique to industry design Jayanta Bhadra and Vivekananda Vedula Publications Lisherness, P., Lesperance, N., and K. T. Cheng, Mutation Analysis with Coverage Discounting, Design Automation and Test Europe (DATE), March 2013. Lisherness, P., and K-T. Cheng, "Improving Validation Coverage Metrics to Account for Limited Observability", Asia and South Pacific Design Automation Conference, Feb. 2012. 20

Slide 21

Summary Analyzed information gained from coverage discounting for two designs Developed a confidence metric to gauge mutant effectiveness and an ordering heuristic to reduce runtime 21