test oracle generation procedures and executable specifications 4 th annual international software...
TRANSCRIPT
Test Oracle Generation Procedures and Executable Specifications
4th Annual International Software Testing Conference 2004
Authors: Rupak Das and Sita RamakrishnanPresented by: Dr Sita Ramakrishnan
4th International Software Testing Conference Pune, India 20-21 Feb 2004
2
Overview Oracle – an example A Generic Test Environment Test Oracles and their Importance Problems Assertion-based Implementation Oracles Specification Oracles Real Specification Languages Testability and Diagnosability with industry
standards AsmL Oracles and the Future Conclusions
4th International Software Testing Conference Pune, India 20-21 Feb 2004
3
Oracle – an example
The Oracle in Matrix Is a program Predicts choices and
outcomes based on input from the Zion mainframe
Test Oracles …
The Oracle offers Candy to Neo, the following dialogue ensues …Neo: D'you already know if I'm going to take it?The Oracle: Wouldn't be much of an Oracle if I didn't.
4th International Software Testing Conference Pune, India 20-21 Feb 2004
4
A Generic Test Environment
4th International Software Testing Conference Pune, India 20-21 Feb 2004
5
A Generic Test Environment
3 main aspects to the test environment:
1. The test model, which is the application model is augmented to include testing requirements
- By adding application specific constraints
to the component interface protocol
2. The test model (state model) is fed to
the test design software
4th International Software Testing Conference Pune, India 20-21 Feb 2004
6
A Generic Test Environment
This software tool uses the critical properties of testing requirements built in the test model to generate automated test sequences
3. The test execution phase is the diagnosis phase. Accepts test inputs from SUT,
and test is evaluated as pass or fail.
4th International Software Testing Conference Pune, India 20-21 Feb 2004
7
Testing Environment in VISWAS
o Our Testing Environment in VISWAS
supports such an automated process
o Includes the Test Sequence Generation phase and the System under Test
4th International Software Testing Conference Pune, India 20-21 Feb 2004
8
Testing Environment in VISWAS
o Diagnosability was not the focus of concern in VISWAS, and test execution phase of test environment was not automated in VISWAS
o Next we look at Test Execution phase and Test Oracle Generation Procedures
4th International Software Testing Conference Pune, India 20-21 Feb 2004
9
Test Oracles – What are they?
A method to establish that the system under test has produced expected outputs corresponding to given test inputs.
All testing techniques assume the existence of an oracle, be it a human `eyeball’ oracle, or a software oracle with valid input/output pairs.
Valid input/output pairs (automatically) generated from specifications helps to verify `intended behaviour’ of the system.
4th International Software Testing Conference Pune, India 20-21 Feb 2004
10
Active and Passive Oracles
Active Oracles mimic the behaviour of the software under
test mainly built for accuracy rather than
efficiency
Passive Oracles Verifies software behaviour without
replicating it
4th International Software Testing Conference Pune, India 20-21 Feb 2004
11
The Problems
Reliable test set problem Whole input domain needs to be taken into
account Otherwise it becomes impossible to create a
reliable exhaustive test set Oracle problem
Grey areas between intended behaviour, expected behaviour and actual behaviour
Some opine that `truly general oracles’ may not be achieved
4th International Software Testing Conference Pune, India 20-21 Feb 2004
12
Assertion-based Implementation Oracles
Assertions are ways to specify what a system is supposed to do, and not how it is to do it
Embedded assertions in source code can be used to create implementation oracles
Not a dedicated specification mechanism
Pre-processing of code is often required
4th International Software Testing Conference Pune, India 20-21 Feb 2004
13
Assertion-based Implementation Oracles …
Assertions
AsmL specification of a stack interface
4th International Software Testing Conference Pune, India 20-21 Feb 2004
14
Assertion-based Implementation Oracles …
Anna (ANNotated Ada) APP (Annotation Pre-Processor) for C Nana (Assertion library for C/C++) Eiffel
Built in assertion support Integrated with exception-handling
Java Multitude of assertion systems iContract, JaWa, Handshake, JMSAssert,
jContractor and JUnit
4th International Software Testing Conference Pune, India 20-21 Feb 2004
15
Assertion-based Implementation Oracles
Dependency on control points Assertions are treated as program statements
with pre-defined execution points
Caching of values Pre- and Post-Condition pairs relate the program
states before and after method execution
Expensive for complex/large data structures
Existential and Universal Quantification Translating input to reflect quantification in terms
of `loops of execution’ can be cumbersome
4th International Software Testing Conference Pune, India 20-21 Feb 2004
16
Specification Oracles
Translating a specification into an oracle irrespective of implementation
Oracles can be generated automatically from specifications or documentation
Can extract relevant input/output pairs of test data from specification to check intended behaviour
4th International Software Testing Conference Pune, India 20-21 Feb 2004
17
Specification Oracles …
Program Testing Assistant Defines test cases interactively Uses layer correspondences to generate test
cases automatically from a library of templates
Test cases execute when appropriate Success criteria defined by specifier Requires at least one successful execution
to generate valid test data sets
Chapman 1982, A Programmer’s Testing Assistant
4th International Software Testing Conference Pune, India 20-21 Feb 2004
18
Specification Oracles …
Test Oracle Generator (TOG)Automatically generates test oracles
from a relational specification that uses tabular expressions
Test harness (oracle) takes input/output pair from Program Under Test (PUT)
Relation is evaluatedReturns true/false
Peters and Parnas 1994, Test Oracle Generator
4th International Software Testing Conference Pune, India 20-21 Feb 2004
19
Specification Oracles …
Documentation components required by TOG Program Specification Auxiliary Predicates and Functions User Definitions
A relational specification is created If result of TOG execution is present in
the relational specification table, oracle procedure execution is successful
4th International Software Testing Conference Pune, India 20-21 Feb 2004
20
Specification Oracles
TOG oracles are only as good as the specification
Access to data structures required to evaluate expressions and predicates
Complexity of test harness reduces readability of the oracle
Developed further by Software Engineering Research Group at McMaster University, Ontario
4th International Software Testing Conference Pune, India 20-21 Feb 2004
21
Real Specification Languages
Actual specifications are translated to oracles for validation and verification
Generate test oracles from specifications that are executable in their own right
B, Z, AsmL (Abstract State Machine Language)
Deriving test oracles require an ``executable subset’’ of the original specification
4th International Software Testing Conference Pune, India 20-21 Feb 2004
22
Real Specification Languages
Automatic Test Result Evaluation approach – translates Z schema to an evaluation function
Test Template Framework derives ``oracle templates’’ from model based specification written in Z
Multiple oracles from multi-language specifications composed as a unit
4th International Software Testing Conference Pune, India 20-21 Feb 2004
23
Testability and Diagnosability with industry standards
o Diagnostic components constructed according to standards facilitate competition in the market place in terms of risks, cost and quality
o Standards imply a maturity in the underlying technology, thus adding to the level of confidence. TTCN-3 is the only standardised language for the specification & implementation of test cases.
4th International Software Testing Conference Pune, India 20-21 Feb 2004
24
Testability and Diagnosability with industry standards
o UML is a widely used industry standard for software component modelling.
o Graphical format for TTCN-3 (GFT) is based on Message Sequence Charts (MSCs) and UML.
o GFT extends TTCN-3 with test specific concepts such as verdicts & defaults.
o GFT is the basis for the definition of UML 2.0 test profile.
4th International Software Testing Conference Pune, India 20-21 Feb 2004
25
Testability and Diagnosability with industry standards
o UML 2.0 Test Profile (UTP) has the notion of an arbiter, which is a test component aimed at separating test behaviour from test evaluationo Evaluates test results and assigns
verdicts of a test case
o A verdict is the outcome of a test case being pass, fail, inconc or error as defined in TTCN-3
4th International Software Testing Conference Pune, India 20-21 Feb 2004
26
Testability and Diagnosability with industry standards
o A Test case may have 4 outcomes:o Pass, Fail, inconclusive or abort
o During the execution of a test case, a test trace is generated and stored in a test log
o UML 2.0 Test Profile, GFT, !TeLa!
have similar diagnostic features
4th International Software Testing Conference Pune, India 20-21 Feb 2004
27
AsmL Oracles
Abstract State Machine approach has been proven suitable for large-scale executable specifications
AsmL is a high-level language that implements the ASM paradigm
AsmL specifications can be simulated Microsoft Research - Foundations of
Software Engineering Group
4th International Software Testing Conference Pune, India 20-21 Feb 2004
28
AsmL Oracles …
Specification is a non-deterministic ASM program written in AsmL that has a fixed initial state
Individual finite states are grouped together to form `hyperstates’
Results in a finite automaton – used to generate test suite
Extraction of an FSM from an ASM specification by extracting algorithm
4th International Software Testing Conference Pune, India 20-21 Feb 2004
29
… and the future
Non-deterministic finite state machines cannot be generated from ASMs by using the extraction algorithm
Combination of the Test Template Framework and Automatic Test Result Evaluation approach for Z specifications could be used
AsmL as a specification and implementation language can be used to generate exhaustive executable oracles
AsmL oracles can be translated to oracles for other .NET aware languages
4th International Software Testing Conference Pune, India 20-21 Feb 2004
30
Conclusions
Producing a comprehensive, precise and understandable specification is difficult enough
Adding computability to specifications to generate oracles makes it even more harder
Intrinsic dependence of testing on oracles make them essential
Pros and Cons exist for all approaches Any of the existing approaches is better and less
expensive compared to a human oracle More research and development of automated
oracle generation tools will make the V & V process much easier and productive
4th International Software Testing Conference Pune, India 20-21 Feb 2004
31
Contact Details
Authors: Rupak Das and Sita Ramakrishnan Presenter: Dr Sita Ramakrishnan
School Computer Science & Software EngineeringMonash UniversityAustralia
[email protected]: 61-3-99052485 fax: 61-3-99031077