verifying safety- related systems - t&vs

Mike Bartley

TVS, Founder and CEO

Verifying Safety-

Related Systems

Copyright TVS Limited | Private & Confidential |

Agenda

Some background on your speaker and safety related systems

Safety standards and compliance

Requirements traceability

EDA considerations


Your speaker: Mike Bartley

PhD in Mathematical Logic MSc in Software Engineering MBA

Worked in software testing and hardware verification for over 25

years • Verification of safety-related hardware and software • Formal verification of both software and hardware

Started TVS in 2008 • Software testing and hardware verification products and services • Offices in India, UK, France and Germany


TVS - Global Leaders in Test and Verification

India - 2011

UK - 2008

Germany - 2011

France - 2012

Singapore - 2014

China South Korea

Continuous

geographical

expansion…

USA - 2014

0

100

200

Q3-13 Q4-13 Q1-14 Q2-14 Q3-14 Q4-14 Q1-15 Q2-15 Q3-15 Q4-15 Q1-16 Q2-16(Est.)

Q3-16(Est.)

Q4-16(Est.)E

MP

LO

YE

ES

CALENDAR YEAR

Number of Employees by quarter

Japan 2015


What is Safety?


Why is Safety (and Security) important?

IC Insights research • The automotive industry is set to drive chip demand over the coming years. • IC Insights research suggests the demand from automotive is expected to

exhibit average annual growth of 10.8% into at least 2018. • Demand will come from safety features that are increasingly becoming

mandatory, such as backup cameras or eCall, and the near-ubiquitous driver-assistance systems.

IoT • Drones (avionics), autonomous cars, robots, …. • Connected devices have potential security threats

TTTech • By 2020 50% of all ICs will be safety-related • By 2020 50% of all ICs will be connected


Safety standards Industrial and Energy: IEC 61508 (2010)

• 61508 – core one Functional Safety of Electrical/Electronic/Programmable Electronic Safety-related Systems

Nuclear: IEC 61513 (2010) • 61513 Nuclear power plants. Instrumentation and control important to safety. General requirements for

systems

Avionics: DO-254 (2005) & DO-178C (2012) • 254 Design assurance guidelines for airborne Electronic Hardware • 178C Software Considerations in Airborne Systems and Equipment Certification

Rail: EN 50128 (2011) • Software for railway control and protection systems

Medical: IEC 60601-1-11 (2010) • for the safety and effectiveness of medical electrical equipment 50128

Automotive: ISO 26262 (2011 [2018]) • Functional Safety standard – next one incl. motorbikes and 3.5 + tonnes


Safety Functions and Integrity

Functional safety is achieved by: Safety function requirements

• From Hazard Analysis

Safety integrity requirements • From Risk Assessment

Any system that carries out safety functions is a Safety-related System

May be separate from a control system or control system itself may be a safety-related system

Higher levels of safety integrity => greater rigour in developing a system


Safety Integrity

Risk based approach

Depends on: - severity of injur(ies)

- frequency/duration of exposure to hazard

- controllability of hazardous event by driver or other traffic participant

Degree of certainty necessary that safety function(s) will be carried out


ISO 26262 Risk Graph

C1 C2 C3

S1

E1 QM QM QM

E2 QM QM QM

E3 QM QM ASIL A

E4 QM ASIL A ASIL B

S2

E1 QM QM QM

E2 QM QM ASIL A

E3 QM ASIL A ASIL B

E4 ASIL A ASIL B ASIL C

S3

E1 QM QM ASIL A

E2 QM ASIL A ASIL B

E3 ASIL A ASIL B ASIL C

E4 ASIL B ASIL C ASIL D

Severity

Exposure (probability)

Controllability


Sample Differences in Safety Integrity Levels

Dynamic analysis and testing

Technique SIL 1 SIL 2 SIL 3 SIL 4

Structural test coverage (entry points) 100% HR HR HR HR

Structural test coverage (statements) 100% R HR HR HR

Structural test coverage (branches) 100% R R HR HR

Structural test coverage (conditions, MC/DC) 100% R R R HR

Test case execution from boundary value analysis R

HR HR HR

Test case execution from error guessing

R R R R

Test case execution from error seeding - R R R

Test case execution from model-based test case generation R R HR HR

Performance modelling R R R HR

Equivalence classes and input partition testing R R R HR


How Systems Fail

Random failures • Can usually predict (statistically)

• Can undertake preventative activities

Systematic failures • Specified, designed or implemented incorrectly

• Can’t usually predict

Systemic failures • Shortcomings in culture or practices

Copyright TVS Limited | Private & Confidential | Page 14

ISO2626 Overview


V Model with Safety Extension


Key Processes

Plans & Standards Requirements Design Specifications Reviews and Analyses Testing (against specifications)

• At different levels of hierarchy

Test Coverage Criteria Requirements Traceability Independence


Key Deliverables

Verification Plan Validation and Verification Standards Traceability Data Review and Analysis Procedures Review and Analysis Results Test Procedures Test Results Acceptance Test Criteria Problem Reports Configuration Management Records Process Assurance Records


Requirements Management “The management of safety requirements includes managing requirements, obtaining

agreement on the requirements, obtaining commitments from those implementing the requirements, and maintaining traceability.”

1.117 semi-formal notation • description technique whose syntax is completely defined but whose semantics

definition can be incomplete • EXAMPLE System Analysis and Design Techniques (SADT); Unified Modeling

Language (UML).

Stage one: What


Stage two: How will we prove it

Verification

“6.4.3.3 An appropriate combination of the verification methods listed in

Table 2 shall be applied to verify that the safety requirements comply with the requirements in this clause and that they comply with the specific requirements on the verification of safety requirements within the respective parts of ISO 26262 where safety requirements are derived.”


Stage Three: Proof of Implementation

Requirements stages • Of good quality • Correctly refined • Implemented • Proven to be implemented

How to prove • By test • By review • By justification • By documentation


Traceability in Practice

Intent to implement

Intent to verify

Stakeholder Requirements (Customers and internal)

Product Requirements

Product Architecture

Verification & Test Plans

Proof of implementation

Verification & Test Results

Requirements

Product Specification and Features

Shows a mapping from features to verification and test plans


Measuring Requirements Progress

Req1 Feat1 Feat1.1 Goal1 Directed Test

Code Coverage

Functional Coverage

Feat1.2 Goal3

Feat3 Req2 Property Proved

Assertion Passing Feat1.3 Goal5

Goal6 Assertion Coverage

Software Running Feat2 Goal7

Goal8 Lab Results

Verification Metrics

Feat2.1

Feat2.2

Goal9

Goal2

Goal4

Use a bi-directional mapping to track backwards

Use an SQL database to hold the mappings and results

75%

50%

0%

85%

70%

0%

Regression 2

Regression 1

84%

76%


asureSIGNTM at the heart of HW/SW V&V

Requirements - Excel - Doors - Jira - Etc Word via XML

Hardware Simulation • Coverage Cadence • Assertions Mentor, Aldec • Etc.

Directed test results asureSIGNTM

Matlab

Formal Verification • OneSpin

UCIS API

Run API

Automated SW Test Tool

SW Test Tools

Manual API

Lab Results

Requirements Engineering tools

SystemC Simulation

www.mentor.com © Mentor Graphics Corp. Company Confidential

A Typical Semiconductor Flow

26

Capture Requirements

Create Design Concept

Create RTL

Design

Synthesize Design

Insert Test

Structures

Perform Place & Route

Manufacture ASIC

Program FPGA

Evaluate HW

Chip Development Flow

Verification and Validation

Requirements Tracing

Requirements Validation

Modeling

Analysis & Reviews

Simulation

Formal Methods

Simulation

Formal Methods

Simulation

Static Timing Analysis

Test Harness

Lab Equipment

Emulation & Prototyping

Analysis & Reviews

Manufacture Testing

ISO 26262 requires a Software Tool Criteria Evaluation Report for each tool or tool chain used


Determining Tool Confidence Level

27

TD1 TD2 TD3

TI1 TCL1

TI2 TCL1 TCL2 TCL3

Tool Impact

Tool Error Detection

Tool Confidence Level

See ISO 26262-8, Section 11


Determining Tool Confidence Level

28

TCL1 – No further tool qualification activities needed

TCL2 / 3 – Formalized tool qualification required

TD1 TD2 TD3

TI1 TCL1

TI2 TCL1 TCL2 TCL3

Tool Impact: Can the software tool malfunction such that it introduces or fails to detects errors of safety requirements?

Tool Error Detection: Degree of confidence that the software tool’s malfunction and its corresponding erroneous output will be prevented or detected?

No

Yes

High Medium Low / None

Tool Confidence Level: Level of confidence that the software tool malfunctions will not lead to the violation of safety requirements.

No Qual Needed Qual Required


Tool Qualification Strategies

Increased Confidence from Use (1a) — This requires extensive use with the same (or very similar) version, constraints, uses cases, environment, etc. — Any tool issues should be documented, monitored, and appropriately worked around if needed

Evaluation of Tool Development Process (1b) — The tools must be developed to comply with an appropriate standard (e.g., Automotive SPICE, CMMI, ISO

15504) — Process should be evaluated and proper application of the assessed development

process shall be demonstrated

Validation of Software Tool (1c) — Verifying the tool performs as expected against its requirements — Typically done by testing functional and non-functional aspects

Development in Accordance with a Standard (1d) — The tool itself is developed in such a way as to be compliant to the

relevant aspects of a safety standard (e.g. ISO 26262, IEC 61508 or RTCA DO-178)

— NOTE: This is very rare

Note that a combination of 1b and 1c is the most common approach by far for ASIL C & D

30


ISO 26262 requires a Software Tool Criteria Evaluation Report for each tool or tool chain used

This report must include

1. Description of the tool

2. Planned usage of the tool 1. Version, configuration, and environment of the tool 2. How its used in the flow (with specific use cases) 3. The maximum ASIL of the project it will be used on

3. An evaluation of TI, TD, and ultimately the TCL classification

Establishing Tool Confidence Levels

31


Summary

Safety is not an add on methodology • Your development flows MUST comply

Safety will affect most of your development

Requirements tracing and proof of implementation will impact your verification

Your tools must also comply

verifying safety- related systems - t&vs

Documents