Automated traceability to assist

DO-254 certification

Andy Nicol, Principal Firmware Engineer

2

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

Agenda

• Company Overview.

• Firmware Engineering group overview.

• DO-254 and the problem of traceability.

• What is ReqTracer and how does it help?

• Example outputs.

• Example use case.

• Pros and cons of ReqTracer.

• Summary.

• Questions.

3

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

Finmeccanica Airborne & Space Systems Divison

Overview

Integrated Networking Solutions for Netcentric Capabilities.

Sensors & Systems for Homeland Protection, Homeland Defence, ATC/ATM, VTMS.

Mission Critical Systems and Defensive Aids Systems.

4

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

Firmware Engineering at Finmeccanica Edinburgh

• The Firmware Engineering discipline is responsible for providing FPGA

expertise to a wide range of projects and products within the Radar and

Advanced Targeting business based in Edinburgh.

• Designs include:

• Radar antenna control.

• Control of aircraft self-protection systems.

• Implementation of radar and image processing algorithms.

• Control of electro-optic turrets.

• Projects have traditionally required designs ‘to the principles of’ DO-254.

• Increasing numbers needing full compliance and certification.

5

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

DO-254 Compliance

• Increasing number of customer requests for DO-254 ‘compliance’/certification.

– ‘Compliance’ is a vague term.

• What do we focus on ?

– Difficulties for ongoing or legacy projects.

• High cost/effort for projects where DO-254 wasn’t a customer

requirement at start-up.

• DO-254 focuses on:

– Requirements traceability.

– Verification/validation.

• Automation may offer part of solution on both fronts:

– Reduce errors.

– Cut timescales.

6

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

Flow-down / Traceability Problem

Customer Requirements

System Requirements

FPGA Requirements

FPGA Test Requirements

FPGA Testbench

FPGA Design

Test Results

7

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

Traceability Problem

• Traceability of requirements documentation well understood.

• Tools such as IBM Rational DOORS manage direct and derived

requirement flow-down from customer requirements to sub-system

requirements.

• Allows (relatively) simple traceability from sub-system requirements back

to customer requirements.

• Traceability from requirements to code or to simulation results has

traditionally been more difficult.

• Problem becomes extremely complex for anything more than the simplest

designs.

• Manual processes are very time consuming and error prone.

8

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

What is ReqTracer?

• Mentor Graphics tool which provides an interactive method to trace and

analyse requirements.

– Complete traceability is achieved by linking high level specifications

through to design implementation and verification results.

– Linkage is achieved across a number of different document types.

– Produces a range of customisable reports throughout the design process.

– Interfaces to other key Mentor tools such as HDS.

– Can be used to facilitate DO-254 certification.

9

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

How Does ReqTracer Help?

• Requirements traceability:

– From source requirements (DOORS/Word etc.).

– To implementation (HDS/VHDL etc.).

– To verification (Modelsim/Questa).

– Coverage reports.

– Simulation assertions.

• Removes need for compliance spreadsheet:

– Manually intensive.

– Error prone.

– Applied to varying degrees depending on project.

10

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

How Does ReqTracer Help?

• ReqTracer reports can be used as evidence of:

– Total coverage, derived requirements, uncovered requirements.

– Impact analysis of individual requirement changes.

– Overall ‘quality’ of the project.

• Helps meet DO-254 by:

- automating tracability.

- reporting and analysing data.

- tracking changes in requirements (the standard mandates changes are

managed effectively).

- ensuring links between requirements, the implemented code and the

verification tests and results.

11

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

ReqTracer Environment – Management View

12

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

ReqTracer Environment – Project Overview

13

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

ReqTracer Environment – Coverage Analysis View

14

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

ReqTracer Environment – Impact Analysis View

15

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

Verification Plan

• A spreadsheet listing all testing to be carried out, providing the link between

the requirements from DOORS to the coverage from ModelSim/ Questa.

• Created with the aid of the Mentor Graphics Questa Plug-in for OpenOffice.

• Links are directly to assertions/other coverage options present in the UCDB

file.

16

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

Questa/ModelSim – UCDB Files

• It is possible to demonstrate that the requirements have been implemented

correctly by saving the simulation results as a UCDB file.

• This file type captures any source of coverage data produced by verification

tools.

– In Questa & ModelSim this is used to store code coverage, assertion data

and functionality coverage.

• Multiple UCDB files can be merged together.

– This means results from multiple small testbenches can be merged to form

a system level coverage result.

– This could vastly cut down the simulation time for a top level testbench.

• Simulation results can also be merged with a verification plan to show that the

testing criteria has been met.

17

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

Questa/ModelSim – UCDB Example

• Example of a previously saved UCDB file, code coverage is active,

assertions are present in testbench.

• Assertion hits are 100%.

• In this state, we can look at the success rate of code coverage, but it

does not show these results in the context of the project and its

requirements.

18

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

Questa/ModelSim – UCDB Example cont.

• Merging the spreadsheet testplan together with the simulation results

produces the following UCDB:

• Coverage data now has context – shows how much of the testplan has been

successfully verified by the simulation behaviour.

19

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

Reports During Design Development

• ReqTracer can provide a range of reports throughout the design & verification

process including:

• Analysis Results.

Complete summary of project & coverage of all documents.

• Traceability Matrix.

Lists upstream-downstream coverage links between two or more

documents.

• Downstream Impact Analysis.

Provides traceability information for specific high level requirements.

Particularly useful for illustrating the impact of changing requirements

on the design.

20

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

ReqTracer Pros

• Automates linking of multiple file types.

• Significantly reduces the level of manual effort required.

• Provides multiple graphical views of results.

• A picture is worth a thousand words when showing results to

management…

• Adds value at multiple stages of the life cycle, not just at the close-out /

certification stage.

• Generates a number of custom reports which show the “quality” of your

project, quoting linkage, derived reqs, any attributes you’ve added.

• Can create regular snapshots of a project showing how it has changed

over time.

• show requirements added/modified/deleted.

• addition of new design and test code.

21

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

ReqTracer Cons

• Links are static.

• Once a link is created, the requirement could change without flagging the

design and/or test as being out of date.

• Regular expression tool is clunky and unintuitive.

• Relatively high learning curve required to get started.

• Improved integration with Mentor’s HDL Designer (our main development

environment) would be helpful.

22

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

Summary

• Clear tracking of requirements to hardware implementation is vital to safety

critical design and helps improve design quality of any complex FPGA / ASIC

design.

• ReqTracer automates and significantly simplifies this process.

• ReqTracer’s various visualisations and reports add value at all project stages.

• Tracks project quality during development.

• Gathers full traceability information during design close down.

• Most importantly – provides required information for DO-254 certification.

23

©

2016 S

ele

x E

S L

td –

All

rights

reserv

ed

Any Questions?