software testing and quality assurance: inspection reading assignment: –john mcgregor and david a....

Software Testing and Quality Assurance: Inspection

• Reading Assignment: – John McGregor and David A. Sykes, A

Practical Guide to Testing Object-Oriented Software, Addison-Wesley, 2001, ISBN: 0-201-325640.

• Chapter 4: Testing Analysis and Design Models

Objectives

• To learn how to inspect the semantics of UML models.

• To be able to setup an inspection session.

• To learn a technique for testing a model for extensibility.

Topics covered

• Introduction

• Guided inspection technique

• Testing specific types of models– Requirements model– Analysis models– Design models

• Testing models for additional qualities

Introduction• Developers model the software they are

constructing because:– It assists in understanding the problem they are solving

– It helps manage the complexity of the system being developed.

• High quality of models make a valuable contribution to the project

• Guided inspection: an enhanced inspection technique for verifying the models as they are created and for validating the completed models against project requirements.

Introduction (cont...)

• Reviews focus on what is in the model rather than what should be in the model.

• Guided inspection applies the testing perspective early in the development process.

• Guided inspection requires valuable resources, and the time and attention of project personnel.

Introduction: the V Model

System Models

System Implementation

Architecture Models

Subsystem Integration

Component Models

Component Implementation

Guided Inspection

Code-b

ased

Tes

ting t

echn

iques

Place of Guided Inspection in the Development Process

• The last activity in each phase in the development process should be a verification of the required qualities of the developed product.

• Differences between succeeding models:– The scope of the content

– The level of abstraction

Guided Inspection in the Development Process: Models and Phases

Architectural design model and standard design patterns and algorithms

Each interface in the Architecture is implemented by one or more components

Detailed design

Standard architectural patterns and creativity of designers

Basic structure of interfaces and their interactions

Architectural design

The domain analysis model and the requirements model

Concepts needed to explain the specific problem; standard algorithm

Application analysis

The minds of the domain experts

Domain concepts, standard algorithms

Domain analysis

Transformed fromContentPhase/model

The Basics of Guided Inspection

• The guided inspection technique provides a means of objectively and systematically searching a work product for faults by using explicit test cases.

• The basic steps in guided inspection– Define the test space

– Select values from the test space using a specific strategy

– Apply the test values to the product being tested

– Evaluate the results and the percentage of the model covered by the test

The Basics of Guided Inspection (cont...)

• The previous steps are specialized to the following steps:– Specify the scope and depth of the inspection– Identify the basis models from which the model under test (MUT)

was created– Develop test cases for each of the evaluation criteria to be applied

using the contents of the basis model as input.– Establish criteria for measuring test coverage.– Perform the static analysis using the appropriate checklist.– “Execute” the test cases.– Evaluate the effectiveness of tests using the coverage measurement.– If the coverage is insufficient, expand the test suite and apply the

additional tests, otherwise, terminate the testing.

The Basics of Guided Inspection : Example

• Initial stages of developing Brickles

• Following work products have been produced:– Design-level class diagrams [Ref. Figure 2.18]

– State diagrams [Ref. Figure 2.19]

– Sequence diagrams [Ref. Figure 2.20]

• Test the design model before starting coding

The Basics of Guided Inspection : Example (cont...)

• Assign the inspection team, the team includes– Two developers

– A system tester from the company

– Company’s process person (Moderator)

• The tester will develop a set of test cases from the use case diagram.

• The developers will show how the classes in the design model handle each test case.

• The moderator will define the inspection boundaries, schedule the guided inspection sessions, etc

• Steps of guided inspection


• Step 1: Specify the scope and depth of the inspection– The scope is defined by a set of use cases.

• We will cover all use cases

– The depth is defined by identifying levels of containment in the composition hierarchies.

• Test only those objects that represent the state of the match (BricklesDoc object)


• Step 2: Identify the basis models from which the model under test (MUT) was created– Design Model

• Design-level class diagrams [Ref. Figure 2.18]

• State diagrams [Ref. Figure 2.19]

• Sequence diagrams [Ref. Figure 2.20]


• Step 3: Develop test cases for each of the evaluation criteria to be applied using the contents of the basis model as input.– Tester will write all test cases based on the Use case diagrams like

Figure 2.11, 2.12 (see next slide)– Each test case should be in the following format

• The use case: A player stops a Brickles match by selecting QUIT from the File Menu

• Preconditions: The Player has started the Brickles, has moved the paddle, and has broken some bricks

• Test input: The player selects Quit• Expected output: All game action freezes and the game window

disappears

The Basics of Guided Inspection: Example― Use Cases for Brickles (cont...)

Use cases for Brickles


• Step 4: Establish criteria for measuring test coverage.– All use cases should be tested. Coverage will be

100 % for use cases and 60 % for composition hierarchies.


• Step 5: Perform the static analysis using the appropriate checklist.– Developers complete the design model

checklist [Ref. Figure 4.3]– Developers compare class diagram from

analysis model [Ref. Figure 2.13] with the class diagram in the design model.

The Basics of Guided Inspection : Example— Design Model Checklist (cont...)


• Step 6: “Execute” the test cases.– Moderator schedules a meeting and makes the

relevant material available to all involved– During the meeting, the developers show how

test cases will be handled by the classes in the design model


• Step 7: Evaluate the effectiveness of tests using the coverage measurement.– The moderator will note down the problems

found during the symbolic execution of test cases.

– Founded bugs will be analyzed by the team and proper description for each bug will be developed.


• Step 8: If the coverage is insufficient, expand the test suite and apply the additional tests, otherwise, terminate the testing.– The moderator will prepare and submit a report

to the project manager that contains:• The problems found during the execution of the test

cases.• Where execution terminated.• The sequence diagram used to record the test

execution.

Guided Inspection Issues• Should test cases be available to the developers prior the

inspection test?– No, developers should not have the scenarios prior the

inspection test.

• Should testers only use test cases for the current increment in an inspection test?– No, running a test scenario from a previous increment as a

regression check on the mode is a useful idea.

• Coverage in models– Higher the level of abstraction, the higher the level of coverage.

Guided Inspection: Evaluation Criteria• Correctness: is a measure of the accuracy of the model:

– In analysis: it is the accuracy of the problem description.– In design: it is how accurately the model represents the solution of the

problem– The model is correct with respect to a set of test cases if every test case

produces the expected result.

• Completeness: is a measure of the inclusiveness of the model (are any necessary, or at least useful, elements missing from the model)?– In iterative incremental process, completeness is considered relative to how

mature the current increment is expected to be.– Do all objects needed for the sequence diagram come from classes in the

class diagram?– The model is complete if the result of executing the test cases can be

adequately represented using only the content of the model.

Guided Inspection: Evaluation Criteria (cont...)

• Consistency: is a measure of whether there are contradictions within the model or between the current model and the model upon which it is based.– Consistency can determine whether there are contradictions or conflicts

present either internal to a single diagram or between two diagrams.– The model is inconsistent if there are different representations within the

model for similar test cases.

• Other qualities like performance goals: define a number of system attributes that the development team might wish to verify.– The guided inspection test cases can be used as scenarios for testing

performance.

Organization of the Guided Inspection Activity

• Basic roles: – Domain expert: the source of the expected results– Tester: conduct the analysis necessary to select

effective test cases.– Developer: the creators of the models under test.

• Individual inspections: testers complete a checklist specific to the type of model being inspected.– This process can be automated.

Preparing for the Guided Inspection: Specifying the Inspection

• Scope of an inspection is defined by specifying a set of use cases, a set of packages, or abstract classes/interfaces.

• Depth of an inspection is defined by specifying layers in aggregation hierarchies under which messages are not sent.

Preparing for the Guided Inspection: Realistic Models

• Layered approach: more individual diagrams but each diagram is sufficiently modular to fit within the scope of a specific inspection

Preparing for the Guided Inspection: Realistic Models— Layered Approach

Preparing for the Guided Inspection: Realistic Models— Layered Approach (cont)

Layering of sequence diagrams

• The diagram terminates at an interface or abstract class

• A sequence diagram is then constructed for each class that implements the interface or specializes the abstract class

Preparing for the Guided Inspection: Realistic Models— Layered Approach (cont)

Preparing For The Guided Inspection: Realistic Models— Layered Approach (cont)

Preparing for the Guided Inspection: Selecting Test Cases for the Inspection• Test cases can be selected to ensure that

specific types of coverage are achieved or to find specific type of defects.

• Test case selection methods:– Orthogonal defect classification: most likely to

identify defects by covering the different categories of system actions that trigger defects.

– Use profiles: give confidence in the reliability of the product by identifying which parts of the program are used the most,

– Risk as a test case selector

Preparing for the Guided Inspection: Selecting Test Cases for the Inspection (cont)

• Orthogonal defect classification (ODC)– The activities that caused a defect to be

detected are classified as triggers.– The guided inspection technique uses several of

these triggers as a guide to select test cases.– By structuring the guided inspection process so

that as many of these triggers as possible are encountered, you ensure that the tests that guide the inspection are more likely to trigger as many failures as possible.


Some review and inspection triggers– Design conformance is addressed by comparing the

basis model to the MUT

– Concurrency is a trigger that will be visible in the design model and scenarios can be generated that explicitly explore thread interactions

– Lateral compatibility is activated by the trace of scenarios between objects on sequence diagrams.

– E.g. rare situation (examining unexpected behavior of the system), backward compatibility (comparison to previous products).


• Use profiles:– A use profile for a system is an ordering of the

individual test cases based on a combination of the frequency and criticality values for the individual use cases.

– The traditional operational profile used for procedural systems is based strictly on frequency-of-use information

– Ex, Logo, attaching local db server–


• Risk as a test case selector:– Risk can be used as the basis for test case selection– It is useful during the development– It is not appropriate after development when we are

trying to achieve some measure of reliability of the software, at that time the use of profile technique is better because it supports the way software will be used.

– The frequency/criticality information is used instead of the risk information for guided inspection

Preparing for the Guided Inspection: Creating Test Cases

• Use case scenario: the path taken

• The alternative paths: several scenarios that differ from the use scenario but represent valid execution

• Exceptional paths: scenarios that result in error conditions

Preparing for the Guided Inspection: an Example of a Use Case

Use case # 1Actor: PlayerUse Scenario: The user selects the play option from the menu. The

system responds by starting the game.

Alternate Paths: If a match is already in progress, the selection is ignored.

Exceptional Cases: If the match cannot open the display, an error message is displayed and the game aborts.

Frequency: LowCriticality: HighRisk: Medium

Testing Specific Types of Models

• The level of detail in the model becomes greater as development proceeds.

• The amount of information also increases as development proceeds.

• The exact interpretation of the evaluation criteria can be made more specific for a specific model.

• The membership of the inspection team changes for different models.

Testing Specific Types of Models: Requirements Model

• Acceptance testing often finds faults that result from problems with the requirements (missing requirements, contradiction in requirements).

• There is no UML model on which the requirements are based, so comparison to the basis model refer to documents produced by marketing, system engineering or client organization.

Testing Specific Types of Models: Requirements Model (Cont...)

• Outline of testing the requirements models:

– Develop the ranking of use cases by computing combined frequency and criticality information for a use case.

– Determine the total number of test cases that can be constructed given the amount of resources available.

– Ration the tests based on the ranking– Write scenarios based only on the knowledge of those in

the domain expert’s role– Check for completeness, consistency, and correctness


UseFrequenc

yCriticalit

yCombined value Rank

Number of test cases

Start Brickles

Medium High High 1 3

Pause Brickles

Low Low Low 3 1

Stop Brickles

Medium Low Medium 2 2

Break brick

High High High 1 3

Wins Medium High High 1 3

Loses Medium Low Medium 2 2


• Criteria for requirement inspection:

Any system functionality is specified in the same manner everywhere it is described.

Consistency

Each use case accurately represents a requirement.

Correctness

The use cases represent all of the functionality needed for a satisfactory product. No use case is needed that is not required functionality.

Completeness

Interpretation for domain modelingCriteria

Testing Specific Types of Models: Analysis Models

• Domain analysis model

• Application analysis model

Testing Specific Types of Models: Analysis Models— Domain Analysis Model

• A domain model is a representation of the knowledge in a domain as seen through the eyes of a set of domain experts.

• Two groups are formed:– Group one: developers group ―creates the

domain model– Group two: testers and domain expert group

―serves as testers of that model. (this group creates the test cases of the model).

Testing Specific Types of Models: Analysis Models— Domain Analysis Model (Cont...)

• Criteria for domain analysis model inspection

Model elements should be consistent with company’s definitions and meanings

Consistency

The description of domain concepts are accurate; the algorithms will produce the expected results

Correctness

The concepts are sufficient to cover the scope of the content specified. Sufficient detail is given to describe concepts to the required path.

Completeness


Testing Specific Types of Models: Analysis Models— Application Analysis Model

• Multiple domain models contribute to the single application analysis model.

• Criteria for application analysis model inspection.

Where there are multiple ways to represent a concept of action, those ways are equivalent

Consistency

Experts agree with the attributes and behaviors assigned to each concept; on the steps in each algorithm; major states for each conceptual entity.

Correctness

The ideas expressed in each use case can be represented by the concepts and algorithms in the model. No design information is included in the model.

Completeness


Testing Specific Types of Models: Design Models

• In an OO project there are three levels of design:– Architectural

– Mechanistic

– Detailed

• We will focus on two basic design models that encompass those three levels:– Architectural design model: provided the basic structure of the

application by defining how a set of interfaces are related.

– Detailed class design model: provides the precise semantics of each class and identifies the architectural interface to which the class corresponds.

Testing Specific Types of Models: Design Models— Architectural Model

• The architectural model is the skeleton of the entire application.

• Non-functional requirements are blended with functional requirements.

• A software architectural model is the basic structure that defines the system in terms of computational components and interactions among those components.

Testing Specific Types of Models: Design Models— Architectural Model (Cont...)• Representations of the architecture:

– To represent an architecture:• Relationships• States• Algorithms

– Tool support: Rational Rose performs a variety of consistency checks on the static-relationship model.

• Consistency checks will prevent certain types of connection from being established. However, it is not sufficient for determining whether the functionality is correctly implemented

• Architecture description languages provide the capability to represent a system at a high level of abstraction (e.g. Rapide)

• Testing the architecture:– The Software Architecture Testing (SAT)

technique is a special type of guided inspection that requires the following steps:

• Test cases are constructed.

• The tests are conducted on the product

• The results of the test are evaluated for correctness.

Testing Specific Types of Models: Design Models— Architectural Model (Cont...)

• Constructing test cases:– Test cases are constructed from the use cases.– The test cases for the architecture are defined at

a higher level than more detailed design models.


• Criteria for the architectural design model inspection:

Each use of the system can be handled only in one set of interfaces.

Consistency

The architecture satisfies its constraints; uses the appropriate architecture patterns; represents the interactions between interfaces.

Correctness

A sufficient set of interfaces are defined to provide all of the services needed for the application’s functionality. The relationship between the interfaces allows for the flow of control and data necessary to realize all of the uses described in the use case diagram.

Completeness

Interpretation for the architectural design modelCriteria


• Test execution:

– Test cases are executed by constructing a message-sequence diagram.

– The diagram reflects preconditions for a test case.


• Verification of results:

– When the output from the test is in the form of diagrams, the resulting diagrams must be verified after each test execution by domain experts.

– When the output is the result of an execution, the test result can be verified by having those domain experts construct event sequences that would be produced by an architecture that performs correctly.


• Evaluating performance and scalability:– An architecture of a system should be evaluated

beyond correctness, completeness and consistency (performance and scalability).

– The test cases are symbolically executed and the message-sequence diagrams can be analyzed from a performance perspective.

– Sequence diagrams can also be used to evaluate scalability.


Testing Specific Types of Models: Design Models— Detailed Class Design Model• The detailed class design model populates the

architectural model with classes that will implement the interfaces defined in the architecture.

• This model typically includes:– A set of class diagram– The Object Constraint Language (OCL) pre- and

postconditions for every method of every class– Activity diagrams of significant algorithms – State diagram for each class

• The test cases at this level are very much like final system test cases.

Testing Specific Types of Models: Design Models— Detailed Class Design Model (cont)

• Criteria for the class design model inspection:

The behavior in the interface of each class provides either a single way to accomplish a task or, if there are multiple ways, they provide the same behavior but with different preconditions

Consistency

Each class accurately implements the semantics of an interface. For those classes that correspond to interfaces in the architecture, the class’ specification must correspond to the interface specified by the architecture

Correctness

Classes are defined for each interface in the architecture. The preconditions for each method specify sufficient information so that the user can safely use the method. The postconditions for a method show error conditions as well as the normally expected result.

Completeness

Interpretation for the architectural design modelCriteria

• As the test progresses:– The executers select methods that will be invoked– The state model of the receiver is checked to be certain the

target object can receive the message– The messages are then added to the sequence diagram and

the state models are updated to reflect changes in the state.– Sequence diagrams will have dead-end objects in which

testers will not attempt to examine the logic beyond that object.

– Implementation and code-based testing start after this point.

Testing Specific Types of Models: Design Models— Detailed Class Design Model (cont)

Testing Models for Additional Qualities

• Projects are chartered to achieve more aggressive objectives such as:

– The development of extensible designs

– The design of reusable frameworks

– Highly portable systems

• A change case is a use case that is not a requirement of the system, but it is an anticipated change to the system.

Testing Models for Additional Qualities (cont...)

• Explicitly state the objective that the change will address.– The design will be easily extensible to

accommodate new games.

• Construct a “change case” including a specific scenario that illustrates the objective.– The framework is to be used to implement pinball

games that are user configurable. The obstacles to be available include posts, flippers, and bumpers.

• Create test cases by sampling from the range permitted by the change case.– A pinball game is to be created. States may be

added to Brickles state machine.

• Enumerate the work needed to achieve the objective by specifying the differences in state and behavior required for the new objectives.– The stationarySprite class will be subclasses to

provide the new obstacle.


• Evaluate the current design relative to the design required to achieve the objective.– The necessary base classes and methods are

present. The needed attributs can be added without conflict with existing attributes.

• Repeat with additional test scenarios until all proposed changes are examined.

• The output of this process is a set of potential changes needed to achieve the desired system quality such as extensibility.


Guided Inspection Process Checklist• Decide how completeness, consistency, and correctness will

be judged for particular use in the model under test (MUT)• Determine which scenarios to sample from the use case

model to use as test cases.• Create test cases by supplementing the scenarios with specific

data• Select the model/notation that will record the results of each

execution• Conduct tests• Evaluate the results of the test executions to determine which

tests the model passed and which it failed

• Record the results for use in guiding the repair and testing processes in the next iteration.

Key Points

• The guided inspection technique provides a means of objectively and systematically searching a work product for faults by using explicit test cases.

• The basic steps in guided inspection– Define the test space– Select values from the test space using a specific strategy– Apply the test values to the product being tested– Evaluate the results and the percentage of the model covered by the

test• Evaluation criteria:

– Correctness– Completeness– Consistency– Other qualities

Key Points (cont...)• Test case selection methods:

– Orthogonal defect classification

– Use profiles

– Risk

• Testing the requirements models:– Develop the ranking of use cases by computing combined frequency and

criticality information for a use case.

– Determine the total number of test cases that can be constructed given the amount of resources available.

– Ration the tests based on the ranking

– Write scenarios based only on the knowledge of those in the domain expert’s role

– Check for completeness, consistency, and correctness

Key Points (cont...)

• A domain model is a representation of the knowledge in a domain as seen through the eyes of a set of domain experts.

• Multiple domain models contribute to the single application analysis model.

• Design models– Architectural design model– Detailed class design model

• Projects are chartered to achieve more aggressive objectives such as:– The development of extensible designs– The design of reusable frameworks– Highly portable systems

software testing and quality assurance: inspection reading assignment: –john mcgregor and david a....

Documents

project guided inspection

basics of guided inspection

place of guided inspection

test slide

basis models

succeeding models

completed models

inspection session