functional safety beyond iso26262 for neural networks in ... · functional safety beyond iso26262...
TRANSCRIPT
![Page 1: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/1.jpg)
Functional Safety beyond
ISO26262 for Neural Networks in
Highly Automated Driving
Autonomous Driving Meetup #5
MAN Track Forum, Munich
27th of March 2018
André Roßbach, Tim Heinemann, Florian Bogenberger
![Page 2: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/2.jpg)
Motivation
Copyright © exida.com 2000-201827/03/2018 2
ISO26262
CNN, AI, ML
![Page 3: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/3.jpg)
Agenda
• What is beyond ISO26262 incl. 2nd Edition? (today)
• How can NNs become "safe"?
• „Probably correct“ – Is this sufficient?
• How to measure „Safety“? – Metrics for NNs
• What is beyond NNs? (today)
27/03/2018 Copyright © exida.com 2000-2018 3
![Page 4: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/4.jpg)
Structure of the Implementation
27/03/2018 Copyright © exida.com 2000-2018 4
Classic Development Neural Networks
Paradigm ShiftStructure homomorph
to System
Structure disparat
to System
![Page 5: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/5.jpg)
Using „ISO-Methods“ for NNs?
Basic Aspect for
ISO26262
Static SW-
Algorithms
Parameter
Learning
Learning of Rules
(NNs, etc.)
Structur of
Implementation
Similarity to logical-
functional structure ☺
Fault-Injection Test ☺..
Statische Analysis ☺ ☺..
Prove of „Non-Behavior“ ☺ ☺..
Requirements Coverage
Requirements Tracing ☺..
Structural Coverage ☺
Training
Some methods according to ISO26262 work
no longer for neural networks
27/03/2018 Copyright © exida.com 2000-2018 5
![Page 6: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/6.jpg)
Tool Chain Aspects
Copyright © exida.com 2000-201827/03/2018 6
Tool chain complexity & degree of
automation will exceed by far today´s
development tool chains
ISO26262-8 11 “Confidence in the
use of Software Tools”
ISO26262-7 “Production & Operation”
... requires enhancements
![Page 7: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/7.jpg)
Enhancement: Tool Safety Concept
27/03/2018 Copyright © exida.com 2000-2018 8
![Page 8: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/8.jpg)
Agenda
• What is beyond ISO26262 incl. 2nd Edition? (today)
• How can NNs become "safe"?
• „Probably correct“ – Is this sufficient?
• How to measure „Safety“? – Metrics for NNs
• What is beyond NNs? (today)
27/03/2018 Copyright © exida.com 2000-2018 9
![Page 9: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/9.jpg)
Approach and Systematics
Copyright © exida.com 2000-201827/03/2018 10
![Page 10: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/10.jpg)
ISO26262 Mapping Matrix
27/03/2018 Copyright © exida.com 2000-2018 11
ISO26262
Structure
Extensions and
adaptations
map
map
![Page 11: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/11.jpg)
Machine Learning Safety Concept
27/03/2018 Copyright © exida.com 2000-2018 12
Machine learning
safety concept
Specification of safety
requirements
for machine learning
Dataset safety requirements
• ML Algorithm
• Performance Measures
• Avoidance of unintended Behavior
• Detection of Implausibility
• Error Detection & Mitigation
• …
![Page 12: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/12.jpg)
... a simplifying abstraction of real error effects ...
... intended to enable systematic analysis
in reality fault propagation is quite complex, but ...
... frequently different faults lead to similar errors
sometimes seem more pessimistic than reality ... but ...
reality is much more „creative“ than the human brain can
foresee
27/03/2018 Copyright © exida.com 2000-2018 14
Fault Model – What it is ...
Fault Models are Key for Safety Analysis
(Static & Dynamic)
![Page 13: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/13.jpg)
27/03/2018 Copyright © exida.com 2000-2018 16
Fault Model – Applied for NNs
Von Chrislb - Erstellt von Chrislb, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=224561
• Weight too high
• Weight too low
• Weight corrupt
• …
„Simple“ Approach: Fault Model on Structural Level
![Page 14: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/14.jpg)
Usage of different solutions to prevent, reduce or
detect failures
homogenous - assumes using an identical safety
element multiple times ( the „same“ twice)
heterogenous - assumes using different safety
elements to fulfill the same safety requirements
Why we need Diversity ...
27/03/2018 Copyright © exida.com 2000-2018 17
Diversity is one of the key methods to detect faults
beyond the limits of “single brain” human
reasoning.
![Page 15: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/15.jpg)
ML: Diversity for “free” ...
27/03/2018 Copyright © exida.com 2000-2018 18
Voting on results
Input
Difference
Machine Learning is well suited to realize diversity ...
... different ...
• architectures
• training data
• ML approaches
... come “for free”.
![Page 16: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/16.jpg)
Agenda
• What is beyond ISO26262 incl. 2nd Edition? (today)
• How can NNs become "safe"?
• „Probably correct“ – Is this sufficient?
• How to measure „Safety“? – Metrics for NNs
• What is beyond NNs? (today)
27/03/2018 Copyright © exida.com 2000-2018 19
![Page 17: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/17.jpg)
Failure without Fault?
ISO26262 simple model: Fault, error, failure chain
27/03/2018 Copyright © exida.com 2000-2018 20
… could there be a failure without a fault?
… are human faults just "stupidity"?
???
Key Insight: We need to understand how ”Hypothesis” work ...
essential for human intelligent behavior
... ISO26262 has zero idea about this
Extend your thinking beyond “faults” ...
... recognize the power of hypotheses ... improve hypothesis
... likewise
for NNs ...
![Page 18: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/18.jpg)
Agenda
• What is beyond ISO26262 incl. 2nd Edition? (today)
• How can NNs become "safe"?
• „Probably correct“ – Is this sufficient?
• How to measure „Safety“? – Metrics for NNs
• What is beyond NNs? (today)
27/03/2018 Copyright © exida.com 2000-2018 21
![Page 19: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/19.jpg)
Structural Coverage 1
ISO26262: Metric to identify shortcomings in test
cases, inadequacies in requirements or
unintended functionality
Meaning
– Find unneeded code
– Find needed code … but missing test
– Find needed code … but missing requirement
27/03/2018 Copyright © exida.com 2000-2018 22
![Page 20: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/20.jpg)
Structural Coverage 2
Structural Coverage for SW (ISO26262):– Statement Coverage
– Branch Coverage
– Modified Condition/Decision Coverage (MC/DC)
… for Neural Networks:– Measure Degree of Neuron Activations
– Possible Formula:
𝑆𝑡𝑟𝑢𝑐𝑡𝑢𝑟𝑎𝑙 𝐶𝑜𝑣𝑒𝑟𝑎𝑔𝑒 % =𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑐𝑡𝑖𝑣𝑎𝑡𝑒𝑑 𝑁𝑒𝑢𝑟𝑜𝑛𝑠
𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑁𝑒𝑢𝑟𝑜𝑛𝑠∗ 100
𝐴𝑐𝑡𝑖𝑣𝑎𝑡𝑒𝑑 𝑁𝑒𝑢𝑟𝑜𝑛 = max 𝑎𝑖) − min(𝑎𝑖 > 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑
27/03/2018 Copyright © exida.com 2000-2018 23
![Page 21: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/21.jpg)
Structural Coverage 3 - Example
Adapted for Neural Networks
– Observation of activation of nodes during test run
– Example
27/03/2018 Copyright © exida.com 2000-2018 24
…
Variance
(example only)
![Page 22: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/22.jpg)
Dangerous Confusion Metric (DCM) 1
27/03/2018 Copyright © exida.com 2000-2018 25
Real Class
Pedestrian Car Bus Cyclist Normal Street
PredictedClass
Pedestrian 568 0 0 0 0
Car 50 1056 0 25 0
Bus 150 10 746 15 0
Cyclist 30 0 0 198 0
Normal Street 200 300 700 800 3254
ASIL QM
ASIL A
ASIL B
ASIL C
ASIL D
Apply Safety Weight Factors
according to the assigned
ASIL of a misclassification
Traditional Confusion Matrix (example)
Safety Confusion Matrix
Weighted mis-
classification based on
impact (violation of a
safety goal)
Real Class
Pedestrian Car Bus Cyclist Normal Street
Predicted Class
Pedestrian 568 5 2 3 3
Car 5 1056 32 5 5
Bus 15 5 746 3 7
Cyclist 6 7 3 198 12
Normal Street 2 3 7 8 3254
Real Class
Pedestrian Car Bus Cyclist Normal Street
Predicted Class
Pedestrian 568 5 2 3 3
Car 5 1056 32 5 5
Bus 15 5 746 3 7
Cyclist 6 7 3 198 12
Normal Street 2 3 7 8 3254
0
2
5
10
100
![Page 23: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/23.jpg)
Dangerous Confusion Metric (DCM) 2
27/03/2018 Copyright © exida.com 2000-2018 26
Real Class
Pedestrian Car Bus Cyclist Normal Street
Predicted Class
Pedestrian 568 0 0 0 0
Car 50 1056 0 25 0
Bus 150 10 746 15 0
Cyclist 30 0 0 198 0
Normal Street 200 300 700 800 3254
𝐷𝐶𝑀𝐴𝑆𝐼𝐿 𝑆𝐺 Dangerous Confusion Metric for a given ASIL or safety goal respectively.
𝑛𝐹𝑁𝐴𝑆𝐼𝐿 𝑆𝐺Number of false neg. classifications per class and ASIL or safety goal respectively.
𝑛𝑇𝑃𝐴𝑆𝐼𝐿 𝑆𝐺 Number of true pos. classifications per class and ASIL or safety goal respectively.
𝐷𝐶𝑀𝐴𝑆𝐼𝐿 𝑆𝐺 = 1 −σ𝑛𝐹𝑁𝐴𝑆𝐼𝐿 𝑆𝐺
σ𝑛𝑇𝑃𝐴𝑆𝐼𝐿 𝑆𝐺 +σ𝑛𝐹𝑁𝐴𝑆𝐼𝐿 𝑆𝐺
True Pos. and False Neg.
Classifications per SG
(example for ASIL B SG)
![Page 24: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/24.jpg)
Agenda
• What is beyond ISO26262 incl. 2nd Edition? (today)
• How can NNs become "safe"?
• „Probably correct“ – Is this sufficient?
• How to measure „Safety“? – Metrics for NNs
• What is beyond NNs? (today)
27/03/2018 Copyright © exida.com 2000-2018 27
![Page 25: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/25.jpg)
Unintended Behaviour
Unintended or unknown behaviour could lead to
violation of a safety goal/requirement
For neural networks unintended behavior
is often very close
(see adversarial attacks)
Identify the potential unintended behaviours
Testing won´t solve the problem ...
... unintended behaviour must be “excluded by design”
27/03/2018 Copyright © exida.com 2000-2018 28
![Page 26: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/26.jpg)
Error Detection and Mitigation
27/03/2018 Copyright © exida.com 2000-2018 29
unexpected pedestrian – safety
anomaly unless visible before
Wrong decision
Correct decision
Key Insight: 100% correct decision not always possible
=> Calculate multiple variants in parallel
![Page 27: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/27.jpg)
Essential: Plausibility Checks
27/03/2018 Copyright © exida.com 2000-2018 30
implausible location of
traffic light – anomaly
must be detected by NN
![Page 28: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/28.jpg)
Conclusion: We need PROGRESS ..
... for ISO26262 ... for NNs & ML
Concepts
ML Safety Concept Systematics to handle
Functional SafetyTool Safety Concept
Link to Safety Requirements
Algorithms
& Design
Probability-based Algorithms Avoid unintended Behaviour
by Design
Hypothesis-based Prediction & Decision
Verification
& Validation
Detect Implausibility
Error Detection & Mitigation
Stress & Boundary Testing
AnalysisSafety Analysis Methods (Static & Dynamic)
Fault Models for Structure & Behavior
MetricsSafety Metrics suitable for
NNs, ML & probability-based Algorithms
27/03/2018 Copyright © exida.com 2000-2018 32
![Page 29: Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN](https://reader034.vdocuments.site/reader034/viewer/2022042302/5ecd217a51f84567ae76a805/html5/thumbnails/29.jpg)
What do think? ....
Copyright © exida.com 2000-201827/03/2018 34