model-based testing of a software bus - applied on core flight executive
Post on 14-Jul-2015
194 Views
Preview:
TRANSCRIPT
1© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Model-based Testing of a Software Bus
applied on Core Flight Executive
Dharmalingam Ganesan, Mikael Lindvall
Dave McComas (NASA GSFC)
Presented at Flight Software Workshop, Pasadena, CA, 2014
2© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Traditional Test Automation
Only test execution is automated
E.g. Junit, CuTest, etc.
Test cases are manually constructed
Effort intensive
Some issues with traditional testing:
Insufficient coverage of off-nominal behaviors
Tests are too-detailed with low-level details
Not easy to test multi-tasking architecture
3© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Model-based Testing (MBT)
The tester develops a model (a.k.a. testing models)
instead of writing suite of test cases
The model becomes the test oracle
Test cases are auto-generated from the model
based on requirements, API documentations
Key benefits:
Tester works at a high-level of abstraction
Innumerable number of test cases derived from the model
Triggers many, if not all, off-nominal behaviors
Precise specification
4© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
System under test (SUT)
• The Core Flight Software System (CFS)• a mission-independent, platform-independent, Flight
Software (FSW) environment integrating a reusable core flight executive (cFE)
• The CFS is a product-line developed by the NASA Goddard Space Flight Center (GSFC)
• CFS is implemented in C
• cFE layer of the CFS is open source
5© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
cFE/CFS Context Diagram
Inter-task Message Router (SW Bus)
TranspondersCommands
Real-time Telemetry (UDP)
Comm Cards
File downlink
(CFDP)
Summit Chip
Mass
Storage
System
CFDP File
Transfer
FileManager
Local
Storage
DataStorage
Event
ServicesExecutive
Services
Time
Services1553 Bus
Support
Software
BusCommand
Ingest
Telemetry
Output
Table
Services
EDACMemory
Scrubber
Self
Test
Memory
DwellInstrument
ManagerChecksum
Memory
ManagerGN&C
Applications
(4)
Mission Apps
cFE core App
CFS Applications
Stored
Commanding
Software
Scheduler
Health &Safety
Manager
House-
keeping
Limit
Checker
6© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
MBT of Software Bus
cFE has a software bus (SB) Apps communicate indirectly using the SB
Publish-Subscribe architectural style
SB has unit-tests (developed by NASA GSFC) Good coverage but
not taking multi-tasking into consideration
Goals: find defects related to multi-tasking (difficult!)
Generate test cases for SB
Generate the “bubbles” (the apps on previous slide)
Demonstrate the applicability of MBT
Developed an approach of testing SB
Allows testing of multi-tasking architectures
7© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Scope of the current model
Modeled the following behaviors Create Pipes (to hold messages)
Delete Pipes
Subscribe to Messages
Send Messages
Receive Messages
Multiple Apps (dynamically instantiated)
Innumerable test cases (in C) auto-generated
Model based on Microsoft’s Spec Explorer Tool
Will get back to this
8© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Challenges of testing a SB
To test a SB we need apps that publish-subscribe
Apps are runtime tasks that communicate using the
SB
Apps publish/subscribe to messages
Each app cannot decide on its own the correctness
Correctness depends on the global state of the system
E.g: subscribe(msg), RecvMsg() may not work if no other
task is publishing any message
The order of execution of tasks also matters
Need a test architecture!
9© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Test Architecture – Key Ideas
Parent/Child architecture for testing
Each test case is a parent
Each test case runs as an app
At run-time, one or more child tasks are spanned by
the parent
Model controls the behavior of the parent
All test assertions are decided by the parent
All child tasks share the codebase
How the parent and children communicate?
Why not just use the software bus itself?
10© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Test Architecture
CFE
Paren
tChild
1
Child
2
Child
n
Command Pipe
All child tasks share the codebase
11© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Test Architecture – Key Ideas
Each child task subscribes to all commands, such as Create Pipe, Delete Pipe, Subscribe, etc.
Parent broadcast commands to all child tasks Communication uses CFE infrastructure
Task id of the child is also part of the message struct
Only the target child can perform a certain command
Child tasks perform the command and send the return code back to the parent Child tasks send out a result msg
Parent task subscribes to the result msg
The parent verifies test assertion
Asserts are generated from the model
12© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Spec Explorer – Brief Background
Tester develops a model (a.k.a. model program)
Spec Explorer runs as a plug-in to MS Visual Studio
Model programs are written in C# like syntax
The model program is a simplified version of the
SUT
Spec Explorer generates state machines from
models
Also checks whether model satisfies invariants
Helps in validating the model
Test cases are automatically derived from state
machines
SUT’s behavior is automatically compared with
model
Tests failure: Deviation between model and SUT
13© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Abstractions for the model program
Model program is another implementation of the
SUT
But we do not want to create two implementations
No one wants to maintain two implementations
No one wants to develop a system two times
How did we create a simplified version of the SUT?
Key idea: Apply abstractions
Model of GPM, which uses CFE
14© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Sample abstractions for a model
Model is agnostic to multi-tasking complexity of the
SB
Model has a very simple message structure
Message is modeled as an int (not C structures)
Message queues/pipes are also abstracted
Finite depth
Message queues are modeled as simple sequences
instead of using shared memory
No pointers, threads, semaphores at the model level
Very simple data structure using very basic data types
int, boolean, maps
15© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Structure of the model program
E.g.: Number of apps, pipes
Represent the state of the bus
Used for excluding the uninteresting states
Which states are good for terminating tests
Models the actual logic of the software bus
Preconditions for enabling the rule methods to fire
Generates values for rule methods
Utilties for rule methods and guards
17© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Fragments of the model program
Rules are enabled only if Condition.IsTrue returns true
19© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Slicing the model for specific tests
20© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Generated from the model program
We generate
the model!
In “regular”
MBT you have
to manually
create the
model.
21© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Generated test sequences - sample
Each chain is a test case
22© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
SUT Adapter
Adapter wraps the SUT
Converts data/commands from the model into SUT’s
syntax
Adapter simplifies modeling complexity
Methods of the model should map to the adapter
Our adapter is in C#
We “print” test code from our adapter in C
Converts C# tests into C tests
Recall that CFE’s SB interface is in C
23© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Our abstracted interface for Testing
int32 InitApp_w(int32 appName);
int32 CreatePipe_w(int32 appName, int32 pipeName, int32
pipeDepth);
int32 DeletePipe_w(int32 appName, int32 pipeName);
int32 Subscribe_w(int32 appName, int32 msgId, int32 pipeName);
int32 UnSubscribe_w(int32 appName, int32 msgId, int32
pipeName);
int32 RcvMsg_w(int32 appName, int32 pipeName, int32*
actualMsgId);
int32 SendMsg_w(int32 appName, int32 msgId);
24© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Sample generated test casevoid Parent_TestAppMain( void ) {
int32 status;
uint32 RunStatus = CFE_ES_APP_RUN;
Parent_TestAppInit();
status = InitApp_w(APP_0);
assert(status == CFE_SUCCESS);
status = CreatePipe_w(APP_0, PIPE_0, 1);
assert(status == CFE_SUCCESS);
status = Subscribe_w(APP_0, MSG_0, PIPE_0);
assert(status == CFE_SUCCESS);
status = Subscribe_w(APP_0, MSG_1, PIPE_0);
assert(status == CFE_SUCCESS);
status = UnSubscribe_w(APP_0, MSG_1, PIPE_0);
assert(status == CFE_SUCCESS);
CFE_ES_ExitApp(RunStatus);
}
25© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Advantages of using Model-Based Testing
The model focuses on the domain (easier to
understand)
Instead of being source code oriented (harder to
understand)
We automatically generate an endless number of
executable test cases (high coverage)
Instead of manually writing individual test cases
The information is in one place: in the model, easy to
maintain
Instead of being spread out (hard to maintain)
The test cases can easily be run over and over again
26© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Advantages of using Spec Explorer
Generated tests are pretty readable
This is due to the ability to slice models into smaller
models
Data parameters are well handled
E.g., Model can be configured to test multiple apps
Models are programs
Ideal for programmers (who prefer coding)
Models can be formally verified
Invariants encoded in the model help to validate the
model
27© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Challenges with Spec Explorer
Modeling errors can lead to infinite state machine
Need to be careful even for small models (e.g., int
parameters)
Syntax for slicing the model is powerful but not that
easy
Easy to misuse some of (algebraic) operators for slicing
Completeness of our slices
Did we miss any combination of behaviors during slicing?
Model debugging. For example:
Why a new state was generated?
Where/Why the invariants are violated?
28© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Applicability to other flight software
The same approach is applicable to other types of
sw
Requirements are that
The software has an interface (e.g. API, GUI)
Through which commands (stimuli) can be sent
Through which results (responses) can be received
Need (some) specification
Optional: Sample test cases, API usage examples
29© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Conclusion
MBT works well for testing of multi-tasking
architecture
In this case of a software bus
Parent/Child test architecture facilitates testing
Individual tasks cannot decide correctness of their own
Parent coordinates with children and asserts correctness
Models and generated state machines: a good spec!
Innumerable number of test cases from the model
Test cases are agnostic to cFE syntax but still executable
Need to be careful in managing the model’s
complexity
Abstraction is important
Otherwise the model will be as complex as the system under
test
30© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Acknowledgement
Jan-Philip Quirmbach (Fraunhofer Intern)
cFE is open source – not an issue for foreign interns
Alan Cudmore (NASA GSFC)
OSMA SARP:
Martha Wetherholt (NASA HQ)
Ken Rehm (NASA IV&V)
Ricky A. Forquer (NASA IV&V)
This work was partly funded by SARP
31© 2014 Fraunhofer USA, Inc.
Center for Experimental Software Engineering
Questions
Dharma Ganesan (dganesan@fc-md.umd.edu)
top related