realtime-unittest on object code...

Get the Most out of Development and Testing with the Maker of the Blue Box

V11.01

RealTime-UnitTest

On Object Code Level


V11.01

The Blue Box


V11.01

Motivation

• More and more standards highly recommend intensive testing

• Model based development and testing explicitly mentioned

• Tool qualification to gain confidence in the use of software tools

IEC 61508

ISO 26262 DO178B/C IEC 62304 EN 5012x …


V11.01

RealTime-UnitTest - What‘s in it for you?

• Characteristics

― Object Code Level

― No test driver

― Integrated in the development environment

• Requirements

― For Code Coverage analysis a micro controller with trace support is a must

― The target application should already run on the target

― Some stack space is needed

― Compilers should be EABI (Embedded Application Binary Interface)

compatible*

* specifies standard conventions for file formats, data types, register usage, stack frame organization, and function parameter passing


V11.01

RealTime-UnitTest - What will it do for you?

• Benefits

― Quick turn-around times

• No additional compile, link and download cycle while testing

― Flexibility in use

• Combine trace, performance analysis, code coverage and I/O stimuli with test runs

• Technology may be used for unit, integration and system test

― Test environment corresponds as closely as possible to the target environment


V11.01

How does it work?

• Application is running on a target

― Initialization done

― Pieces of the application already implemented

• A test case is setup and will be executed on the target

― Run until main{}

― Jump to the function under test, push test parameters on the stack

― Execute the funtion under test with parameters

― Compare return values with expected values and jump back to main{}

• How to setup test cases and execute them on a target?

― Remote Control and Test API Architecture


V11.01

isystem.connect

isystem.connect debug

isystem.connect test

iSYSTEM IDE & Debugger winIDEA

isystem.connect for Python wrapper isystem.connect for Python wrapper

isystem.connect for Python wrapper

Remote Control and Test API Architecture


V11.01

C/C++ API

isystem.connect



V11.01

Verwendung isystem.connect debug debug.download(); debug.setBP("myFunction"); debug.run(); debug.waitUntilStopped(0, 200); debug.deleteBP("myFunction"); Use of isystem.connect m_pIConnectDebug.RunControl(IConnectDebug::rDownload, 0, 0); m_pIConnectDebug.SetBreakpoint(IConnectDebug::bSymbol | IConnectDebug::bSet, 0, 0, "myFunction", 0); m_pIConnectDebug.RunControl(IConnectDebug::rRun, 0, 0); IConnectDebug::SStatus status; do { m_pIConnectDebug.GetStatus(IConnectDebug::tCPU, &status); Sleep(200); } while (status.m_byStatus != IConnectDebug::stStopped); m_pIConnectDebug.SetBreakpoint(IConnectDebug::bSymbol | IConnectDebug::bClear, 0, 0, "myFunction", 0);


C/C++ API


V11.01

C/C++ API

• Write your own C/C++ application and remote control winIDEA core

functionality

• Use already built abstraction layers for

― National Instruments LabVIEW (VI library), The Mathworks Matlab (JAVA)

― Microsoft Automation (e.g., to read/write data from/in Excel, use C#, Jscript,

ASP, VBScript, ...)

― ...

• Write your own abstraction layer

• To simplify the use of this API and to implement RealTime-UnitTest we

added another abstraction layer

― isystem.connect debug

― Script language wrappers


V11.01

Debug Abstraction and Script Language Layer

isystem.connect






V11.01

Use of isystem.connect debug debug.download(); debug.setBP("myFunction"); debug.run(); debug.waitUntilStopped(0, 200); debug.deleteBP("myFunction"); Use of isystem.connect m_pIConnectDebug.RunControl(IConnectDebug::rDownload, 0, 0); m_pIConnectDebug.SetBreakpoint(IConnectDebug::bSymbol | IConnectDebug::bSet, 0, 0, "myFunction", 0); m_pIConnectDebug.RunControl(IConnectDebug::rRun, 0, 0); IConnectDebug::SStatus status; do { m_pIConnectDebug.GetStatus(IConnectDebug::tCPU, &status); Sleep(200); } while (status.m_byStatus != IConnectDebug::stStopped); m_pIConnectDebug.SetBreakpoint(IConnectDebug::bSymbol | IConnectDebug::bClear, 0, 0, "myFunction", 0);

Debug Abstraction and

Script Language Layer



V11.01

Read variable 'iCounter' on the target and print its value

until it reaches value of 100000.

import isystem.connect

from isystem import icutils

from isystem.connect import IConnectDebug

# First we obtain controller objects

icFactory = icutils.getFactory()

dbg = icFactory.getDebugFacade()

ide = icFactory.getIdeCtrl()

import time

# Initialize

dbg.download()

dbg.deleteAll()

dbg.setBP('main')

dbg.run()

dbg.waitUntilStopped()

# Set global variable 'isDebugTest' to 1 and run the target

dbg.modify(IConnectDebug.fMonitor, 'isDebugTest', '1')

dbg.run()

py_iCounter = 0

# Read value of 'iCounter' until condition is met

while py_iCounter < 100000:

evalResult = dbg.evaluate(IConnectDebug.fRealTime, 'iCounter')

py_iCounter = evalResult.getInt()

print py_iCounter

time.sleep(1)

dbg.stop()

print "Target stopped, 'iCounter' reached the limit:", py_iCounter

raw_input('Press ENTER to exit')


V11.01

Conclusion

• Generic API enables you to setup an automation scenario for

― Development

― Testing

• Effort doing it in C/C++ is a bit higher than using abstraction layers as well

as script languages

• It still has not much to do with functional testing approach


V11.01

Test API

isystem.connect







V11.01

Use of isystem.connect test Use of isystem.connect debug debug.download(); Use of isystem.connect m_pIConnectDebug.RunControl(IConnectDebug::rDownload, 0, 0);

Test API


V11.01

Use of isystem.connect test testCase.itestCpp("func: [funcTestGlobals]\n" Use of isystem.connect debug debug.download(); Use of isystem.connect m_pIConnectDebug.RunControl(IConnectDebug::rDownload, 0, 0);

# Simple test of function: int funcTestInt2(int a, int b, int c) func: [funcTestInt2, [-30000000, 2, 1], retVal] expect: - retVal == -29999996

YAML 1.1 – yaml.org • Similar to XML • Data structures like in Perl, Python, C • Human readable • de.wikipedia.org/wiki/YAML • isystem.com/downloads/isystem.connect/api/yaml_spec.html

Test Specification Language


V11.01

import isystem.connect as ic

import isystem.itest as it

def verifyResult(results):

for result in results:

if result.isError():

out = ic.CStringStream()

emitter = ic.EmitterFactory.createYamlEmitter(out)

emitter.startStream()

emitter.startDocument(False)

result.serializeErrorsOnly(emitter, None)

emitter.endDocument(False)

emitter.endStream()

print out.getString()

else:

print "OK"

cmgr = ic.ConnectionMgr()

cmgr.connectMRU('')

dbg = ic.CDebugFacade(cmgr)

# initialize the target - this may differ in your environment

dbg.download()

dbg.runUntilFunction("main")

dbg.waitUntilStopped()

testCase = it.PTestCase(cmgr)

# Simple test of function: int funcTestInt2(int a, int b, int c)

testCase.itest(

"""

func: [funcTestInt2, [-30000000, 2, 1], retVal]

expect:

- retVal == -29999996

""",

None, True)

print 'funcTestInt2 test result:',

verifyResult(testCase.getTestResults())

# Test with coverage

testCase.itest(

"""

func: [funcTestGlobals]

coverage:

runMode: start

document: itestCoverage.ccv

openMode: w

isSaveAfterTest: yes

exportFormat: XML2

exportFile: coverageTest.xml

statistics:

- func: funcTestGlobals

""", None);

print 'funcTestGlobals test result:',

verifyResult(testCase.getTestResults())


V11.01

isystem.connect






iSY

ST

EM

Test G

UI te

stID

EA

Remote Control and Test API Architecture


V11.01

testIDEA GUI


V11.01

testIDEA Standard

• Integrated in winIDEA

• Free of charge

• Online help and samples

• Descriptive tooltips

• TestID, tags, description

• Local test variables

• Stubs

• Trace, Profiler, Code Coverage support

• Derived/sub test case creation

• Debugging of test cases

• Report Generation in XML and HTML

• Stylesheet for HTML Report Generation


V11.01

testIDEA Professional

• testIDEA standard features

• A license fee applies

• Python script generation by mouse click

• I/O module support

• Script execution before and after a test run

• Excel/CSV import and export

• Test filter based on tags

• Test specification can be saved and loaded into/from C/C++ source code

• Dry run – precalculation of expected return values by test execution

• Automatic wrong parameter detection

• Tag for specifiying the sequence of functions called during test

• Breakpoints inside functions under test

• Real-time stubs (stubs executed in real-time)

• Support/update and upgrade service


V11.01

env:

initBeforeRun: false

downloadOnInit: true

resetOnInit: false

runOnInit: true

stopFunction: main

deleteBreakpointsOnInit: false

reportConfig:

reportContents: full

outFormat: xml

fileName: 'D:\isystem.testIDEA\TestResults\PPC554

ITMPC554 GCC int RAM Test Report.xml'

xsltFull: '<built-in> itestResult.blue.xslt'

isIncludeTestSpec: true

useCustomTime: true

testInfo:

tester: 'Werner'

date: '09.11.2010'

time: '11:33:29'

software: 'SW'

hardware: 'HW'

description: 'Desc'

testSpecificationFile: 'D:\isystem.testIDEA\My YAML

Files\PPC5554 ITMPC5554 GCC

Int RAM.iyaml'

wiWorkspacePath:

'D:\winIDEA\2011\Examples\OnChip\PowerPC\MPC55xx\ITM

PC5554\GCC\IntRAM'

tests:

- id: 'StubTest'

desc: 'Stub Test'

init:

iCounter: 0

func: [Address_DifferentFunctionParameters]

stubs:

- func:

- Func2

- retV

assign:

retV: 2

expect:

- iCounter == 1

- id: 'GlobalVariableTest'

desc: 'Global Variable Test'

init:

iCounter: 0

func: [Address_DifferentFunctionParameters]

expect:

- iCounter == 1

Saved Test Specification from within testIDEA


V11.01


V11.01

eMOTE *E*mbedded *Mo*del-based *Te*sting

• Funded by BMWi Germany • Project partners: FZI Karlsruhe, sepp.med GmbH • Project Number: KF2076903SS9


V11.01

Regression Test Tool Suite


V11.01

isystem.connect



iSYSTEM Development Environment & Debugger winIDEA


testID

EA

Reference Target Reference Application

+ Consulting/Integration Services

Test GUI

Test Specification, Test Cases,

Test Reports Reference-Target

Safety Manual

Tool Qualification

realtime-unittest on object code...

Documents