a system testing framework

A System Testing Framework

RTECC, 6. May 2003 Dr. Ferdinand Gramsamer

[email protected]

www.bbv.ch

System Testing Framework RTECC, May 6 2003

bbv Software Services Corp. 2

Abstract While a large variety of test frameworks for unit testing of software systems exists, similar frameworks for testing hardware systems and distributed components are hardly found. We introduce ‘TestClamp’, a scalable, modular, and customizable system-testing framework provided with a single common user-interface ‘clamping’ on the hardware I/Os. Its benefits are automated regression testing, administering test suites and test cases, and reporting.

Summary TestClamp, developed as an open source application by bbv Software Services Corp. for a customer in the building technology arena, is a framework for system testing. It is not only a workbench to easily write test cases, but also

• Middleware to integrate standard or proprietary interfaces, such as RS232, CAN, and BACNet.

• Interface to simulation platforms such as MATLAB • Test-manager to administer tests and generate test reports • Tool box for often commonly required functionality, such as logging,

schedulability, timing control, etc. TestClamp realizes these design goals:

1. Scalable. TestClamp is covered by the LGPL 1.2 or higher licence, which opens the door to self-written add-ons and changes to the main application. It is therefore scalable by the needs of the user or the requirements of the system under test. The scalability feature is guaranteed by the underlying software architecture, which rigorously implements common design patterns [Gamma97].

2. Modular. TestClamp is modular. Middleware provides functionality to the

main application. The main application brings all the required functionality together and delivers the framework for the test case developer. All modules, middleware, the application and test cases are written in one language, which is Java to provide platform-independence. This opens the possibility to utilize open-source development tools. TestClamp has been designed for someone not knowledgeable in Java or the application, to be able to write test cases after two days of coaching.

3. Customizable. TestClamp is customizable by its command-line options and

configuration capabilities. Whereas command-line options set up the application in the desired mode of operation, configuration files contain set-up information for attached subsystems. Furthermore, TestClamp can run in master, configuration, or server mode, which allows TestClamp to be used in cooperation with third-party tools.

All in all, TestClamp is a framework to perform system testing. It not only has its strengths in its design, but has also matured through the close cooperation of customers with bbv Software Services Corp.




A System Testing Framework

RTECC, Zurich, May 6Dr. Ferdinand Gramsamer


Situation

You have successfully implemented your real-time embedded application.

It is running on a piece of hardware.

You now want to test it.




Agenda

• Motivation• Problem Description• Solutions - Examples• State of the Art• Summary


Motivation

• Systems grow larger and are more complex• Too many different tools - expensive• Proprietary interfaces supported?• Increased need for simulation • Amount of test data needs to be dealt with• Repetitive tasks for testing

Need of framework for system testing




Typical Test Environment

• Components– Subsystems (adapter cards,

uP subsystem), peer-systems

• Instruments– Logic Analyzer,

Oscilloscope, BERT, etc.

• Simulation– MatLab/Simulink, Filter

• SUT: System Under Test

PC

Component

Simulation

Instrument

Component Instrument

SUT


Goal

Develop endurable system testing framework:

– Platform independent (OS)– I/O adaptable (new standards)– Configurable (ever more complex cards, and boards)– Test administration: register and execute test cases, store

results, replay, schedulable– Open source (no license fee, customizable)– One programming language– Flexible to integrate third party tools




Focus

• PC• Software architecture?

PC

Component

Simulation

Instrument

Component Instrument

SUT


Logical View

Testscript

Ablauf-steuerung

TesthilfsmittelSchnittstellen

TesthilfsmittelRS232

TesthilfsmittelIO-Emulator

Testhilfsmittel….

Middle-ware

Test-resultate

TestclampTestscript-Generator

Testscript-Debugger

Testscript-Editor

GraphischesUser Interface

TestresultatAnalyser

TesthilfsmittelIO-System

(Applikation)




Distributed Architecture

Application

ComponentMgrExecution

RepositoryMgr

TestMgr

ConfigurationMgr

CommandLineMgr

Tools


Elements

• CommandLineMgrParses command line:

• Database• Directory• Tests • Configuration file• others

• ConfigurationMgrkey-value pairs

• TestMgrRegistered tests to execute

Application

ComponentMgr

Execution

RepositoryMgr

TestMgr

ConfigurationMgr

CommandLineMgr

Tools




Elements cont’d• RepositoryMgr

Create directory structure• ComponentMgr

Download file, or configuration to subsystem(s)

• ExecutionRun tests

• Tools– Matlab (simulation)– I/Os, interfaces– Source control/database

Application

ComponentMgr

Execution

RepositoryMgr

TestMgr

ConfigurationMgr

CommandLineMgr

Tools


Functionality• Timer:

– measure elapsed time; delays– Example: Timer t = new Timer(); t.start(); t.stop();

t.getElapsedTime();

• Scheduler: – execute tests at a specific time of the day– Example: addTestSuite(“TestSuite1”, “2003:05:06:10:12:00”);

• Comparator: – test event in a configurable time window– Example: Comparator c = new Comparator(x, minusdelta, plusdelta);

boolean result = c.isInLimits(y);

• Logger: – progress of test execution (view/file)– Example: getDefaultLogger().addInformativeLog(“Hello”, 3);




TestClamp Design• Creation, Start-up, Shutdown

– Basic services: i.e. kernel, memory management– Creation: create parts– Initialization: set-up relationship between elements– Startup: put system in full operational mode– Shut-down: stop system before leaving application

• Post-Office– One element requests service from another-one– Central object rather than hard-coded in elements– Expl: DemoAppl a=(DemoAppl)Application.getDefault();

a.getScheduler().getSuiteResultFile().writeln(“Hello”);

• Composite for tests (next slides)


Test HierarchyTestItem

TestSuite TestCase

TestContainer

TestCampaignTestProject

Entity holding alltests and direct-ory

Common managementcapabilities

Aggregation of TestItemswith parameters

All tests for a sub-SystemaddTestSuite();

All tests fora unitaddTestCase();

A scenariorun();




TestItem

• Name• Author• Version• Modification Date• Description• General administrative information• Example:

author("Gramsamer (bbv Software Services Corp.)");modificationDate("May 6th 2003");version("1.0");


Test Composition

ApplicationTestManager

TestProject

0 .. 1

1

TestCase1 .. n

1

TestSuite

TestCase

1 .. n

0 .. n

1

For each subsystemA TestProject

optional

1 .. n

1

TestCampaign

TestSuite

TestCase

1 .. n

1

0 .. n

1

0 .. n

TestContainer




TestCase Example


Commercial Tools

• LabView/Windows (NI)– Focus: system tests, standard interfaces

• Rational Test RealTime (IBM)– Focus: rt-tests of code as it executes on target

• TestDirector (Mercury)– Focus: test management– Execution of tests through third party tools, scripts.

WinRunner designed for GUI-testing




Summary

• Flexibility by design• One language from low to high-level programming• Test management• Automated test execution• Open source

12 May 200312 May 2003


References• www.ni.com• www.rational.com• www.mercury.com• www.bbv.ch• E. Gamma et al. “Design Patterns – Elements of Reusable

Object-oriented Software,” Addison-Wesley, 1997




Thank You!

See you at the bbv boothfor a demonstration!

a system testing framework

Documents

test bbv software services

system testing framework

test cases

system testing frameworkrtecc

test suites

system testingframeworkrtecc

testing hardware systems

test data needs