staying competitive by evolving your fpga verification ... · vhdl verilog synopsys vera systemc...

Alex Grove

Staying competitive by evolving your FPGA verification methodologies

European Application Engineer

Mentor Graphics

October 2016

www.mentor.com

© Mentor Graphics Corp. Company Confidential

Industry Trends and Challenges for FPGA Development— What’s the state of FPGA development within industry— What’s driving the need for better verification practices

Verification to the Rescue— Improving quality with SystemVerilog and UVM— Lowering the barrier to entry for UVM— Improve productivity with Questa Verification IP— Where are other FPGA users going with their verification

practices

Final Thoughts

Agenda


www.mentor.com


INDUSTRY TRENDS AND CHALLENGES FOR FPGA

DEVELOPMENT

www.mentor.com


FPGA Completion to Original ScheduleMajority of FPGA Projects Miss Schedule

Behind ScheduleAhead of schedule

Source: Wilson Research Group and Mentor Graphics, 2016 Functional Verification Study H Foster, WRG Functional Verification Study, November 20144

0%

5%

10%

15%

20%

25%

30%

35%

40%

More than 10%EARLY

10% EARLY ON-SCHEDULE 10% BEHINDSCHEDULE

20% 30% 40% 50% >50% BEHINDSCHEDULE

De

sig

n P

roje

cts

Actual FPGA design completion compared to project's original schedule

2012

2014

2016

2012: 67% behind schedule2014: 59% behind schedule2016: 65% behind schedule


www.mentor.com


FPGA Verification Project TimeFPGA Verification Consumes Majority of Project Time

Source: Wilson Research Group and Mentor Graphics, 2016 Functional Verification Study

H Foster, WRG Functional Verification Study, July 20165

0%

5%

10%

15%

20%

25%

1%-20% 21%-30% 31%-40% 41%-50% 51%-60% 61%-70% 71%-80% >80%

De

sig

n P

roje

cts

Percentage of FPGA Project Time Spent in Verification

2012

2014

2016

2012: Average 43%2014: Average 46%2016: Average 48%


www.mentor.com

© Mentor Graphics Corp. Company ConfidentialSource: Wilson Research Group and Mentor Graphics, 2016 Functional Verification Study


0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

1 2 3 4 5 6 7 or more

De

sig

n P

roje

cts

Number of FPGA iterations in the lab

2014

2016

Number of FPGA Iterations in the LabBugs found in lab are 10x more expensive to resolve than if found in simulation

87% of FPGA design projectsrequire multiple iterations in the lab


www.mentor.com



0%

5%

10%

15%

20%

25%

30%

35%

0 1 2 3 4 5 6 or More

De

sig

n P

roje

cts

2016 FPGA Non-Trivial Bug Escapes to Production

78% of FPGA design projectshave non-trivial bugs that escape into productions

Number of FPGA Bug Escapes to ProductionBugs found late in the development cycle are exponentially more expensive


www.mentor.com


Driving the Need for Improved VerificationBusiness needs are pushing improved quality efforts

Safety Critical

• Safety integrity

• High reliability

• High security

Mission Critical

• High reliability

• High security

• High cost of failure

Business Critical

• Time to market

• Quality affects bottom line

• Reducing development cost

Demands better verification practices


www.mentor.com


Biggest FPGA Verification ChallengesSufficient testing and improving debug efficiency are the biggest challenges

Source: Wilson Research Group and Mentor Graphics, 2016 Functional Verification Study (Preliminary Results)

0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

CREATING SUFFICIENT TESTS TO VERIFY THE DESIGN(Coverage Closure)

KNOWING MY VERIFICATION COVERAGE

MANAGING THE VERIFICATION PROCESS

TIME TO ISOLATE AND RESOLVE A BUG

TIME TO DISCOVER THE NEXT BUG

DEFINING APPROPRIATE COVERAGE METRICS

Other

Study Participants

Big

ge

st

Ve

rifi

ca

tio

n C

ha

lle

ng

es

2014

2016


www.mentor.com


FPGA Productivity GapVerification must evolve to keep pace with design innovations

FPGA vendors continue with design innovations— Hardened blocks – Reuse — IP - Reuse— SOC – Reuse— Graphical system builders - Automation— HLS – Abstraction

“Design Productivity” alone ≠ faster “Time to Market”— Designs must also work and be of high quality to gain adoption— TTM depends on maximizing design and verification productivity

Verification must similarly evolve to keep pace— Maximize reuse— Improve automation— Raise the level of abstraction

Ca

pa

cit

y &

Co

mp

lex

ity

Design Productivity

VerificationProductivity

Pro

duct

ivity

Gap


www.mentor.com


Protocol IP Enables Faster DevelopmentBut how do you verify your custom application…

Custom Logic Standard Protocol

FPGA

GT

GT

GT

GT

I/O

I/O

I/O

I/O

Ethernet

PCIe

USB

DisplayPort

SPI

NOR

GT

GT

GT

GT

I/O

I/O

I/O

I/O Custom

UART

100GbE

HMC

XAUI

SATA

NAND

AXI

AXI

AXI

AXI

AXI

Custom

AXI

AXI

Custom

Custom

BRAM

Custom

Custom

Custom

Custom

Custom

FIFO

Custom

ReedSolomon

BRAM

DSP48

FFT

Custom

DSP48

MIG

MIG

MIG

AXI

DMA

AXI

AXI

Custom

AXI

Custom

AXI

DRAM

AXI

Custom

Custom

Custom

Custom

Custom

Custom

How do I test my custom application in this sea of

complicated IP?

Majority of traffic on/off today’s

FPGA is through protocol IP


www.mentor.com


What Makes Functional Verification Difficult?


Single, sequential data streams— Floating point unit— Graphics shading unit— DSP convolution unit — MPEG decode— . . .

Compressed

Audio

Sequential data streams1x number of bugs

Concurrent data streams10x number of bugs

Multiple, concurrent data streams— Cross bar— Bus traffic controller— DMA controller— Standard I/F (e.g., PCIe)— . . .

www.mentor.com


Outdated FPGA Verification PracticesAre They Now Impacting You?

Lab testing is slow and ineffective

Directed testing is not sufficient

Need reuse to stay productive

Lack complex IP protocol expertise

Late to market

Poor quality

Struggling with safety

critical

High development

costs

65% of FPGA projects are behind

schedule

78% of designs have production

bugs

50% of all designs will be safety

critical

Bugs found in lab are 10x more costly

than simulation


www.mentor.com


VERIFICATION TO THE RESCUE

www.mentor.com


Improving By Evolving Verification Methods

Advantages

More testing

Improved quality

Find bugs early

Greater insight

Faster debug

Greater reuse

Shorter cycles

SystemVerilog

Constrained Random

Coverage

Assertions

Object Oriented

UVM

UVM Framework

Reuse

Methodology

Formal

CDC

Exhaustive Testing

Time Reducing Apps

Verification IP

Accelerated Development

Easy Stimulus

Protocol Assurance

Reuse


www.mentor.com


How Do You Improve Verification Productivity?


Where do spend your time for FPGA verification?

Maximize Debug EfficiencyMore simulations

= more testing

Minimize Time on Non-Value Add Development

13%20%

21%

43%

3%

Test Planning

Testbench Development

Creating Test and Running Simulation

Debug

Other


www.mentor.com


Benefits of SystemVerilog and UVMImproving Quality and Schedule with SV and UVM

Constrained random

• “Automatic” stimulus accelerates closure

• Tests things you didn’t think of

Functional coverage

• Provides insight into what is being tested

• Measures testing completeness and quality

Assertions

• Improves debug efficiency

• Designers can write to augment verification

Enables Benefits of SystemVerilog

• Reduces scope of SV

• Enhances SV functionality

Provides Methodology

• Guides how to build a testbench

• Industry standard

Promotes Reuse

• Reuse within and across projects

• Reduces TB development effort

SystemVerilog UVM


www.mentor.com


FPGA Verification Language Adoption TrendsSystemVerilog is now the leading FPGA verification language



* Multiple answers possible

0%

10%

20%

30%

40%

50%

60%

70%

VHDL Verilog Synopsys Vera SystemC SystemVerilog Specman e C/C++ OTHERTestbench

De

sig

n P

roje

cts

Languages Used for Verification (Testbenches)

2012

2014

2016

Next Year


www.mentor.com


FPGA Testbench Methodology Adoption TrendsUVM is the clear leader in FPGA testbench methodologies



* Multiple answers possible

0%

10%

20%

30%

40%

50%

60%

Accellera UVM OVM Mentor AVM Synopsys VMM Synopsys RVM Cadence eRM Cadence URM None/Other

De

sig

n P

roje

cts

FPGA Methodologies and Testbench Base-Class Libraries

201220142016Next Year


www.mentor.com


FPGA Dynamic Verification Adoption TrendsAdoption of advanced verification techniques continue to grow for FPGA verification



0% 10% 20% 30% 40% 50% 60% 70%

Constrained-Random Simulation

Functional coverage

Assertions

Code coverage

FPGA Design Projects

2012

2014

2016


www.mentor.com


Mentor UVM Framework (UVMF) for Easier UVMMaking the transition to UVM possible

UVM Jumpstart

• Immediately productive while learning UVM

• Layer on top of UVM that hides UVM details

• Allows team to focus on verifying product features

• Promotes reuse increasing productivity

UVM Testbench Generators

• Code generators to create a UVM testbench in minutes

• Saves 3-4 weeks of effort on every project

• UVMF is open source and no cost – Delivered in Questa 10.5

Helped over 30 companies adopt UVM

• 75% are FPGA customers

• Over half are in mil-aero industry

• Majority use VHDL for design and have no SystemVerilog experience


www.mentor.com


Class library— Defines reuse methodology— Component base classes— Package structure for reuse

Scripts— Auto generation of components and test bench— Makefiles with common tool flow operations

Examples— Block and chip level benches— Technology integrations

Documentation— Users Guide

The UVM Framework Contents

UVMF generator


www.mentor.com



0%

5%

10%

15%

20%

25%

30%

1 2 3 4 5 6 or more

De

sig

n P

roje

cts

Number of FPGA Designs Completed by Project per Year

Number of Designs Projects Completed per YearUVM promotes verification reuse which improves productivity across projects


www.mentor.com


Verification IP: Off The Shelf Protocol Test Environments“Protocol Expertise” Reduces Time, Effort & Risk for IP Centric Designs

What is Verification IP?— Reusable test environments— Built around standard protocols

Reduces Verification Effort— Provides completed TB at standard interfaces— Provides functionality to create and run tests— Less time creating TB and more time testing

Eliminates Protocol Expertise— Deep protocol knowledge not needed for TB creation— Provides out of the box protocol stimulus and checking— Enhanced debug productivity with transactions and assertions

VIP Reduces Risk— Re-usable VIP building blocks— Common design architectures & protocols— Complete protocol assurance

Transactions

Configuration

Agent

Test

Plans

Coverage

Models

Test

Suites

Protocol

Debug

Interface

UVM Agent

DUT


www.mentor.com


Applications for Verification IPEnabling rapid testbench development and higher quality

Easy protocol stimulus to FPGA

Qualifying vendor IP in safety critical

applications

Custom IP development


www.mentor.com


Verification IP Enables Rapid Testbench DevelopmentEnabling verification of your custom application without deep protocol expertise or effort

Custom Logic Standard Protocol

FPGA

GT

GT

GT

GT

I/O

I/O

I/O

I/O

Ethernet

PCIe

USB

DisplayPort

SPI

NOR

GT

GT

GT

GT

I/O

I/O

I/O

I/O Custom

UART

100GbE

HMC

XAUI

SATA

NAND

AXI

AXI

AXI

AXI

AXI

Custom

AXI

AXI

Custom

Custom

BRAM

Custom

Custom

Custom

Custom

Custom

FIFO

Custom

ReedSolomon

BRAM

DSP48

FFT

Custom

DSP48

MIG

MIG

MIG

AXI

DMA

AXI

AXI

Custom

AXI

Custom

AXI

DRAM

AXI

Custom

Custom

Custom

Custom

Custom

Custom

HMCModel

SATAVIP

10GbEVIP

FlashModel

MemoryModel

MemoryModel

MemoryModel

PCIeVIP

EthernetVIP

FlashModel

SPIModel

UARTModel

Verification IP

100GbEVIP

USBVIP

DPVIP


www.mentor.com


Transaction recording— Debug at a higher level with integrated

transactions— Quickly understand and analyze bus activity— Automatically links transactions to signals

Protocol assertions— Integrated assertions automate protocol checking— Protocol assertions immediately pinpoint source of

failure— Quickly understand integration of your design with IP

Highlighting shows which signals and

WHEN

Relationship works BOTH

waysAssertion shows

exact time of violation at the

interface

Assertion details why it failed and what it was

expecting to pass

Debug Faster with Questa Verification IPQVIP Improves debug abstraction and automation to boost productivity


www.mentor.com


A Complete Solution for FPGA and ASIC DesignsMentor Graphics Questa Verification IP Library

Questa VIP Library

Serial

Family

Ethernet

Family

100G

40G

10G

1G

100M

10M

MIPI ®

Family

USB

Family

USB 3.0

USB 2.0

JTAG

SmartCard

Display

Family

I2C

I2S

SPI 4.2

SPI

UART

Unipro

LLI

CSI-2 / CSI-3

HDMI 2.0

HDMI 1.4

DSI

DigRF

HCDP

HDMI 1.3

USB 1.1

eHCI

CEC

oHCI

xHCI

SSIC

3.1 Serial

3.1 Pipe

MPHY

MPHY

I3C

UFS25/50G

HSI

USB PD

eDP

V-by-One

Automotive

CPHY

DPHY

AMBA ®

Family

AHB5

AXI3

AXI4

PCIe ®

Family

PCIe 3.0

PCIe 2.0

MPHY

PCIe 1.0

ACE

AHB MRIOV

PIE8

RMMI

PCIe 4.0

AHCI

NVMe

APB3 DisplayPort

I3C

Interlaken

400G

Questa Memory Library

DRAM

Family

DDR2

DDR4

LPDDR4

LPDDR3

LPDDR2

DDR3

DFI

WIDEIO

HBM2

HMC

Flash

Family

SDCard 4.2

eMMc 5.1

ONFI 4.0

UFS

SDIO 4.1

DIMM

Toggle

Automotive

Family

CAN

AMBA 5

CHI

CHI 5

Mil-Aero

Family

Spacewire

1553b

Hyperbus

Hyperram

5G

Family

JESD204B

Hyperflash

ParallelNOR

PCI


www.mentor.com


FINAL THOUGHTS

www.mentor.com


Proven Benefits from Advanced VerificationFPGA customers realize the benefits of evolving their verification methodologies

• Constrained random for better testing

• Functional coverage for better insight

• Assertions to improve debug efficiency

SystemVerilog

• Enables benefits of SystemVerilog

• Provides consistent methodology

• Promotes reuse for productivity boostUVM

• Protocol expertise

• Reduces TB development effort

• Risk reductionVerification IP

FPGA Customer Successes

400% ROI after adopting UVM and VIP

5 straight FPGA’s without an issue missed in verification

Zero bugs found in lab since moving to UVM

No longer need expensive lab equipment

100x faster to debug with UVM vs lab

Won more contracts due to shortened design cycles


www.mentor.com


Challenges: Evolving FPGA verification

“It is too difficult…”

UVM Framework

Verification Academy

SystemVerilog ecosystem

Questa Verification IP

“It is too expensive…”“400% ROI after moving to

UVM”

“Won more contracts because of shortened design cycles”

“Shifted resources from lab to simulation”

Simulation debug is 10x more effective than lab debug

“It is overkill for my needs…”

50% of all designs will be safety or security critical

Proven effective for teams of as small as 1 or 2 engineers

78% of designs have bugs that escape into production

Improved quality and productivity drive bottom line


www.mentor.com


Where Can You Go To Learn More?

Verification Academy — https://verificationacademy.com/— Most comprehensive resource for verification training

in industry— Focuses on methodologies rather than tools

Functional Verification at www.mentor.com— https://www.mentor.com/products/fv/— Learn more about tools to enable methodologies— Simulation, SV/UVM, Formal, Verification IP


https://verificationacademy.com/

https://www.mentor.com/products/fv/

www.mentor.com


www.mentor.com

staying competitive by evolving your fpga verification ... · vhdl verilog synopsys vera systemc...

Documents