![Page 1: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/1.jpg)
©20
08 T
he M
athW
orks
, Inc
.
® ®
New Concepts and Tools for Effective Verification and Validation Based on Model AnalysisMaster Class
![Page 2: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/2.jpg)
2
® ®
Today’s Agenda
� Quick Demo� Challenges� Methods for Early Verification and Validation
� Robustness Testing
� Automatic Test Generation
� Property Proving
� Questions and Answers
![Page 3: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/3.jpg)
3
® ®
Poll
� Do you test your models?� Do you have coverage requirements?
� How hard is it to reach 100% coverage?
![Page 4: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/4.jpg)
4
® ®
Requirements
FPGA ASIC
Digital Electronics
VHDL, Verilog
Implement
Integration
DSP
Embedded Software
C, C++
MCU
Address the Entire Development Process
Design
Physical Components
Environment
Algorithms
Requirements
Gen
erat
e Generate
Integration TestingSoftware Integration TestingHardware-in-the-Loop Testing
Hand-Generate
System V and VRequirements ValidationRobustness TestingModeling Standards Checking
Component V and V
Code VerificationCode CorrectnessProcessor-In-The Loop Testing
Design VerificationModel TestingCoverage & Test Generation Property Proving
![Page 5: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/5.jpg)
5
® ®
Requirements
FPGA ASIC
Digital Electronics
VHDL, Verilog
Implement
Integration
DSP
Embedded Software
C, C++
MCU
Address the Entire Development Process
Design
Physical Components
Environment
Algorithms
Requirements
Gen
erat
e Generate
Integration TestingSoftware Integration TestingHardware-in-the-Loop Testing
Hand-Generate
System V and VRequirements ValidationRobustness TestingModeling Standards Checking
Component V and V
Code VerificationCode CorrectnessProcessor-In-The Loop Testing
Design VerificationModel TestingCoverage & Test Generation Property Proving
Master Class
![Page 6: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/6.jpg)
6
® ®
Verification and Validation Challenges
� Management of tests and test assets
� Writing tests for 100% coverage of control logic is hard
� Some requirements are difficult to test
![Page 7: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/7.jpg)
7
® ®
Testing in Simulation
FPGA ASIC
Digital Electronics
VHDL, Verilog
Implement
Integration
DSP
Embedded Software
C, C++
MCU
Design
Physical Components
Environment
Algorithms
Functional Requirements
Gen
erat
e Generate
Hand-Generate
�Design Verification�SystemTest™�Simulink® Verification and Validation™�Simulink Design Verifier™
�Code Verification�Real-Time Workshop® Embedded Coder™�Embedded IDE Link™ products�Target Support Package™ products
Verify that design meets requirements
Verify that the behavior of source code and object code matches the model
![Page 8: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/8.jpg)
8
® ®
Requirements
Early Validation and Robustness Testing
Design
Physical Components
Environment
Algorithms
Requirements
System V and VRequirements ValidationRobustness TestingModeling Standards Checking
![Page 9: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/9.jpg)
9
® ®
System V and V - Example
� Evaluation of robustness of a DC Motor model� Assessment of model accuracy in predicting performance variability
of real systems
![Page 10: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/10.jpg)
10
® ®
Compute clusterCompute cluster
CPU
CPU
CPU
CPU
MATLAB® Distributed Computing Server™MATLAB® Distributed Computing Server™
Scheduler
Result
Result
Result
Result
Client MachineClient Machine
Task
Task
Task
Task
Worker
Worker
Worker
Worker
ParallelComputing Toolbox™
TOOLBOXES
BLOCKSETS
Result
Job
System Test with Distributed Computing
![Page 11: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/11.jpg)
11
® ®
Management of Tests and Test AssetsSystemTest™
� Authoring� Creating tests from requirements
� Importing existing test data from Excel
� Generating tests with Simulink Design Verifier
� Execution and Reporting� SystemTest plots and test report
� Benefits� Automate test execution
� Build consistent test execution environment for repeatable results
� Create baselines of design behavior and run them in regression
� Continuously improve quality of models and generated code
� Export tests and test results for testing on hardware
![Page 12: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/12.jpg)
12
® ®
Test Generation Workflow
FPGA ASIC
Digital Electronics
VHDL, Verilog
Implement
Integration
DSP
Embedded Software
C, C++
MCU
Design
Physical Components
Environment
Algorithms
Functional Requirements
Gen
erat
e Generate
Hand-Generate
Code Harness
C
Detailed modelsComponent
Source Code
C
AnalysisModel
Test Application
CodeGeneration
�Design Verification
�Code Verification
![Page 13: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/13.jpg)
13
® ®
Model CoverageSimulink Verification and Validation
� Structural metric � Measure of test completeness
TT, FTTT, TF
TT, FTTT, TF if (X & Y)
Z = 1;else
Z = -1;end
Example MC/DC Coverage
MC/DC Coverage� each condition independently changes the decision outcome
Decision
Condition
![Page 14: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/14.jpg)
14
® ®
Model Coverage ToolSimulink Verification and Validation
� Model Coverage tool reports coverage metrics� User must provide input data for the simulation
![Page 15: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/15.jpg)
15
® ®
Objectives for Test GenerationSimulink Design Verifier
if (X & Y)Z = 1;
elseZ = -1;
end
Affects (X & Y)to be T and F?
Affects (X & Y)to be T and F?
Affects (X & Y)to be T and F?
Affects (X & Y)to be T and F?
TT, FTTT, TF
TT, FTTT, TF
![Page 16: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/16.jpg)
16
® ®
Test Generation for CoverageSimulink Design Verifier
� Generating tests to reach coverage objectives
Test generation harness with the copy of the original model Test inputs that ensure complete
coverage
Test Generation
![Page 17: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/17.jpg)
17
® ®
Test Generation Results – Harness ModelAn interface block builds up vectors and cast signals to the needed data types
Test Cases are captured in a Signal Builder block
Input data sequences drive system from its initial configuration
Original model copied to the harness
![Page 18: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/18.jpg)
18
® ®
Code Testing with Generated SignalsSimulink
� Software-in-the-loop� On the host
� Processor-in-the-loop� On the target processor
� Independent code testing environment� Generated signals and model outputs
are saved as a .mat data file� Exported input signals drive code
tests� Exported model outputs become
expectation values for code testing
![Page 19: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/19.jpg)
19
® ®
Processor-In-The-Loop TestingEmbedded IDE Link™ TS (for Altium® TASKING®)
PIL also provides execution profiling, code coverage reports, and interactive debugging
Simulink:
Real-Time Workshop® and TASKING:
ECU:
� Model in simulation and code on the processor running in parallel
![Page 20: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/20.jpg)
20
® ®
Demonstration
� Demonstration of test generation with Simulink Design Verifier
![Page 21: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/21.jpg)
21
® ®
Thrust Reversers
![Page 22: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/22.jpg)
22
® ®
Thrust Reversers Should not be Deployed During Flight
![Page 23: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/23.jpg)
23
® ®
Thrust Reverser Deployment Requirements
� The following requirements shall be met prior to deploying the thrust reversers:� Weight on Wheels
� Each main gear, each redundant
� Wheel Speed Sensors� Each main gear
� Airspeed Limit� Redundant Sensors
� Throttle Positions� Each throttle, each redundant
![Page 24: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/24.jpg)
24
® ®
Proving
FPGA ASIC
Digital Electronics
VHDL, Verilog
Implement
Integration
DSP
Embedded Software
C, C++
MCU
Design
Physical Components
Environment
Algorithms
Gen
erat
e Generate
Hand-Generate
� Proving Design PropertiesSimulink Design Verifier
� Proving Code CorrectnessPolySpace™ Server for C/C++
Prove that design meets the key functional requirements
Prove that code meets non-functional runtime requirements
RequirementsRequirements
![Page 25: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/25.jpg)
25
® ®
Property Proving Workflow
FPGA ASIC
Digital Electronics
VHDL, Verilog
Implement
Integration
DSP
Embedded Software
C, C++
MCU
Design
Physical Components
Environment
Algorithms
Gen
erat
e Generate
Hand-Generate
RequirementsFunctional Requirements
AugmentedComponent Model
AnalysisReport
Model Harness
Counterexample
![Page 26: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/26.jpg)
26
® ®
Property Proving – OverviewSimulink Design Verifier
� Design (Structure) ->
� Design (Behavior) ->
![Page 27: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/27.jpg)
27
® ®
Demonstration
� Demonstration of Property Proving with Simulink Design Verifier
![Page 28: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/28.jpg)
28
® ®
Modeling Functional RequirementsSimulink Design Verifier
Functional Requirement:� If 2 or more thrust sensors are >0, the thrust reverser will not
deploy
![Page 29: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/29.jpg)
29
® ®
Modeling Functional RequirementsExpressing requirements with temporal aspects
After condition ABC is true for X sample periods the controller shall enter mode XYZ within Y samples.
![Page 30: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/30.jpg)
30
® ®
Proving Design PropertiesSimulink Design Verifier
Property Proving Harness augmented with design properties
Detailed report and violations
Property Proving
![Page 31: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/31.jpg)
31
® ®
Property Proving - Counterexample
� Leads to improvement of design and/or requirements
Counterexample!
![Page 32: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/32.jpg)
32
® ®
Improvements
� After some quality design time…
![Page 33: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/33.jpg)
33
® ®
Wheel Speed Check Errors
� Forgot the “=“ case
![Page 34: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/34.jpg)
34
® ®
Throttle Logic Significantly Flawed
� What if 1 throttle is higherthan the threshold, and1 is lower?
![Page 35: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/35.jpg)
35
® ®
Proving Properties – WorkflowsSimulink Design Verifier
1. Authoring� Highly Iterative� Leads to improvement in
design and in specifications
2. Execution and Reporting� Automated
� Part of the regression testing harness
� Benefits� Leads to precise definition of low
level functional requirements
� Once established properties represent a model of design behavior
� Minimizes a chance of implementing undesired behavior
![Page 36: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/36.jpg)
36
® ®
©20
07 T
he M
athW
orks
, Inc
.
Closing Remarks
![Page 37: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/37.jpg)
37
® ®
SystemTestSimulink Verification and ValidationxPC Target
xPC TargetData Acquisition ToolboxInstrument Control Toolbox
PolySpace productsEmbedded IDE Link products
Target Support Package products
Simulink Verification and ValidationSimulink Design Verifier
SystemTest
Requirements
FPGA ASIC
Digital Electronics
VHDL, Verilog
Implement
Integration
DSP
Embedded Software
C, C++
MCU
Verification and Validation Tools
Design
Physical Components
Environment
Algorithms
Requirements
Gen
erat
e Generate
Integration TestingSoftware Integration TestingHardware-in-the-Loop TestingHardware Connectivity
Hand-Generate
System V and VRobustness TestingModeling Standards Checking Requirements Validation
Component V and V
Code VerificationCode CorrectnessProcessor-In-The Loop Testing
Design VerificationModel TestingCoverage and Test GenerationProperty Proving
![Page 38: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/38.jpg)
38
® ®
Do I Need To Implement All / Some of the New Verification and Validation Methods?
� Traditional Verification and Validation Methods� Hardware Integration Testing
� Software Integration Testing
� Unit Testing of Code
� Ad-hoc Testing in Simulation
� Methods for Early Verification and Validation� Traceability
� Modeling and Coding Standards Checking
� Model Testing
� Proving Design Properties and Code Correctness
![Page 39: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/39.jpg)
39
® ®
Motorola Creates Electric Vehicle Battery Management Controller with Real-Time Workshop Embedded Coder
ChallengeTo develop battery management controller software within a tight deadline
SolutionUse integrated tools for Model-Based Design and code generation from The MathWorks to design, test, and manage requirements for the controller
Results� Automatic generation of efficient C code � Optimized memory resources � Ability to detect design flaws before
generating code
To validate the design against the
customer’s requirements, the engineers
associated the model components to the
written requirements with the Requirements
Management Interface. “Internal reviews
were then easy, and we could demonstrate
to our customer that all the requirements
had been met.”
Salam Zeidan
Software Manager
Motorola Automotive
To validate the design against the
customer’s requirements, the engineers
associated the model components to the
written requirements with the Requirements
Management Interface. “Internal reviews
were then easy, and we could demonstrate
to our customer that all the requirements
had been met.”
Salam Zeidan
Software Manager
Motorola Automotive
The Motorola electronic control unit
![Page 40: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/40.jpg)
40
® ®
Model-Based Design for Safety-Critical Applications Success Stories
Alstom Generates Production Code for Safety-Critical Power Converter Control Systems • Defect-free, safety-critical code generated and certified• Development time cut by 50 percent
“the railway application was the first with automatically generated code to receive TÜV certific ation.”
Honeywell Generates DO-178B Certified Code• 1,000,000+ lines of code certified in a single year• 6.3 sigma quality achieved
Institute for Radiological Protection and Nuclear Safety Verifies Nuclear Safety Software with PolySpace™ Products for C/C++
![Page 41: Verification and Validation Master Class€¦ · Verification and Validation Challenges Management of tests and test assets Writing tests for 100% coverage of control logic is hard](https://reader034.vdocuments.site/reader034/viewer/2022042809/5f943e1e3d0372216917fe62/html5/thumbnails/41.jpg)
41
® ®
Summary
� Model-Based Design is a platform that enables you to start verification and validation of designs and embedded software early
� When building a verification environment for your models and thegenerated code there are several different methods you can use to increase confidence in your designs� Traceability
� Modeling and Coding Standards checking� Testing� Proving
� The MathWorks consulting and training teams can help you create a plan for the optimization of your verification and validation process