V0.1 | 2019-05-10

Embedded Testing for the Complete Application

Vector India Conference 2019

2

➢ VectorCAST/QA and Functional Test

➢ Change Based Testing at Black Box

➢ Gray Box Testing using VectorCAST/Probe

➢ Demo

➢ Q&A

Agenda

3

VectorCAST/QA and Functional Test

4

What is Functional Or Black Box Test?

VectorCAST/QA and Functional Test

Functional testing is a way of checking software to ensure that it has all the required functionality that's specified within its functional requirements.

Black box testing is defined as a testing technique in which functionality of the application under test is tested without looking at the internal code structure, implementation details and knowledge of internal paths of the software.

5

REQUIRMENTS and CODE are two different entities and validating just one of these would be unfair to the other one.

Test Coverage = Requirement Coverage + Code Coverage

Requirement coverage is ensured by tracing the requirements to your test cases and ensuring user needs are met but what about the traceability between the code and the test. In other words, what happens if there is code which is not exercised by your test and results into unexpected behaviour at customer’s end.

Test Coverage

VectorCAST/QA and Functional Test

6

Test Coverage

VectorCAST/QA and Functional Test

Test measurement allows better product quality:

So, how do you measure test effectiveness? Counting number of lines covered by tests or taking a functional approach and see if all the features are covered.

Test Coverage:

Test Coverage aims to measure of the effectiveness of testing in a qualitative manner. It determines whether the test cases are covering entire functional requirements. You can think of it as a kind of black box testing, where test cases are not written based on code but based on user requirements or expected functionality.

7

Who should use VectorCAST/QA

VectorCAST/QA and Functional Test

▪ Recurring quality issues

▪ Measure effectiveness of functional test

▪ Wanting to do more with their QA resources in less time

▪ Starting a new project and want to reduce risk as it grows and changes

▪ Taking over a legacy product and need to minimize change risk

▪ Trying to improve the entire application life-cycle management process

▪ Wanting to increase communication between developers and QA for faster cycle times

▪ Outsourcing development and wanting to verify quality of testing

8

Why VectorCAST/QA? (1\2)

VectorCAST/QA and Functional Test

▪ Built specifically for teams and organizations

▪ Easy to set up and configure- whether end

environment is embedded target or x86 host

▪ Adopted to existing workflow with minimal

changes

▪ Can be used by both development and QA

teams making consistent reporting simple

▪ Works with any type of testing from manual

testing to automated

9

Why VectorCAST/QA? (2\2)

VectorCAST/QA and Functional Test

▪ Support wide variety of compiler

▪ Proven tool in safety critical domain like

DO-178B/C (Avionics)

ISO-26262 (Automotive)

IEC-61508 (Industrial)

EN-50128 (Railway)

IEC-62304 (Medical)

▪ Provides all coverage metrics such as statement, Branch, MC/DC, Function, Function Call, Basis Path

▪ Web based dashboard for key metric analysis which, combined with code coverage, allow for quick targeting of high risk, problem areas for refactoring, increased testing, or both

▪ HTML/Text reports are sharable

10

VectorCAST/QA Key Features

VectorCAST/QA and Functional Test

VectorCAST/QA enables teams to implement consistent and repeatable processes for managing test activities and reporting key quality metrics.

▪ Code Coverage Analysis

▪ Continuous Testing

▪ Test Collaboration

▪ System Test Automation

▪ Change-Based Testing

▪ Change Impact Analysis

▪ Parallel Testing

▪ Web-based Quality Dashboard

▪ Quality Trend Analysis

11

VectorCAST/QA Environment

VectorCAST/QA and Functional Test

12

VectorCAST/Analytics

VectorCAST/QA and Functional Test

13

Change Based Testing at Black Box

14

Running the complete set of tests for software applications can take days. Obviously running a full set of tests prior to a Production Release is prudent, but how about for intermediate builds as changes are made.

Change Based Testing

Change Based Testing at Black Box

15

Automatically Identify the Minimum Tests for Each Code Change

Change Based Testing at Black Box

For most software changes, you would ideally like to choose a sub-set of tests to run which will still adequately test the changes being made.

Choosing which tests to run is not as simple as finding tests that invoke a function directly. How about all of the tests that call functions that call that function, or that are called by the changed function, they are likely to be impacted too.

In fact most regressions occur because of these transitive relationships.

16

Change Impact Report

Change Based Testing at Black Box

17

Build Execute-Incremental

Change Based Testing at Black Box

18

Gray Box Testing using VectorCAST/Probe

19

▪ Gray-box testing is a combination of white-box testing and black-box testing. The aim of this testing is to search for the defects if any due to improper structure or improper usage of applications

▪ Gray box testing, also know as gray box analysis, is a strategy for software debugging in which the tester has limited knowledge of the internal details of the program. A gray box is a device, program or system whose workings are partially understood.

▪ Gray-box testers require both high-level and detailed documents describing the application, which they collect in order to define test cases.

What is Gray Box Testing?

Gray Box Testing using VectorCAST/Probe

20

▪ One of the challenges with resolving defects in a fully integrated system is how to capture the data needed to understand the root cause of the problem. Using a debugger often changes the timing of the system in a way that masks the bug or prevents the system from running properly.

▪ VectorCAST/Probe addresses these challenges using our existing code instrumentation and target support technology to make the process of adding trace code as simple as possible.

VectorCAST/Probe

Gray Box Testing using VectorCAST/Probe

21

Features

Gray Box Testing using VectorCAST/Probe

▪ Dynamically instruments device software to isolate defects

▪ Inserts probes during Unit, Integration, or System Testing

▪ Captures internal data values

▪ Records detailed control flow

▪ Injects Faulty Values to test error handling

▪ Debugs hard to trigger race conditions

▪ Supports all embedded targets

22

VectorCAST/Probe Example

Gray Box Testing using VectorCAST/Probe

23

Demo

24

Q&A

25

Author:

Yadav, Sunil

Vector India