system test

7/28/2019 System Test

1/48

System Test

By: Dung Nguyen Date: June 10, 2007


2/48

Feb. 04, 2002 Confidential 2

Contents

IntroductionSystem Test OverviewSystem Test Plan ContentSystem Test Cases

Execution of System Test CasesReporting Test ResultsRegression Test CasesAppendix I - Example Test Plan

Appendix II - Example Test ExecutionReferences


3/48


INTRODUCTION

Purpose and Scope of this document:This document provides information on the requirementsof the system test phase and its deliverables.

The document is intended to be used as a guide for the preparation and execution of system testing.


4/48

System Test Overview


5/48


System Test Overview - Objectives

System Test is responsible for:Developing the System Test Plan and test casesExecuting the testing based on the test cases developedGenerating the test results and reporting weekly, monthly,

or by release depending upon the needGenerating and closing problem reports

The objective of System Test is to test the system

capabilities and behaviour of the product on different typesof platforms on which the product will be available.


6/48


System Test Overview - Scope of System Test

The System Test test areas are usually:Feature and Feature Interaction Tests,Performance and Memory Measurement,Capacity Limits Tests,

Load and Longevity Tests,System Recovery and Robustness,System Security Tests, System Operations (including newInstallation, Software Conversion, and System Expansion),

andEnd-User Documentation.


7/48Feb. 04, 2002 Confidential 7

System Test Overview - Scope of System Test (Cont)

Feature and Feature Interaction Tests : These test verifyon functionality and features that are the product release

Performance and Memory Measurement : These testsmeasure the system performance, memory usage, andvarious data throughput rates to compare these againstthose specified in the product specification and engineeringguidelines.

Capacity Limits Tests : These tests focus on system behavior when the system reaches its capacity limits asdefined in the product specification.




Load and Longevity Tests : Load tests will be run to check service standards related to serviceavailability, lost message rate, damaged message rate, droppedcalls, etc., as defined in the product specification.Longevity tests test the system behavior over time when subjectedto a specified load

System Recovery and Robustness : These tests will startwith inserting both software and hardware faults into thesystem and observing:

How the system detects faults,How the system recovers when faults are detected,How robust the system is in general, andHow well faults are reported.




System Security Tests : These tests will focus on thesecurity of the system from all kinds of terminals as:

Screen based, telset,Access (remote, local),Client/server, and

Different kinds of users, operators, and administrators.

A key goal of these tests is to ensure that system operationfunctions do not violate system security.




System Operations: System Operations for the productrefer to the system configuration utilities provided for the

product, including:Installation,Expansion,

Conversion, andA number of less general utilities.

Testing will cover all operations on a variety of hardware

platforms




End-User Documentation: End-user documentation procedural testing and review is part of the function of System Test. This can be achieved efficiently by usingend-user documentation when testing the various systemfunctions.


12/48

System Test Plan Content



System Test Plan: Responsibility

The system test plan is the responsibility of the SystemTest group


14/48


System Test Plan: Major Sections

The System Test Plan must minimally include the followingsections:Executive SummaryProduct OverviewSchedule

Acceptance CriteriaTest CoverageTest Environment NeedsAssumptions, Dependencies, and Risks

ReportingGlossaryReferences


15/48


System Test Plan - Executive summary

The executive summary provides an overview of the purpose of this document. Key points outlined hereinclude:

Resources,Schedule time-frames,

Assumptions,Dependencies, andRisks.


16/48


System Test Plan - Product Overview

The product overview provides a brief description of theservices offered in the product as well as the platforms andconnectivity of the product.

If the product is a new release of an existing product,discussion also describes new release capabilities as wellas existing base release capabilities.


17/48


System Test Plan - Schedule

This section describesSystem test milestones andDetailing the resource (staffing) requirements before, during, andafter the system test phase.

A detailed schedule is presented here for the actual testing phase including

Personnel, platforms, and

type of testingduring the planned actual testing period.


18/48


System Test Plan - Acceptance Criteria

The acceptance criteria provide details in the following terms:Entrance criteria,Exit criteria,Suspension criteria, and

Deliverables criteria to System Test.

Note:The use of entrance and exit criteria is to ensure that those responsible for

the preceding integration test phase and subsequent product verification phases are aware of the quality that is expected and will be delivered in the product.


19/48


System Test Plan - Test Coverage

This section provides a description of the testing that will be performed on the product for each of the system testingtest areas. Test areas might consist of the following:Features and Feature InteractionsPerformance and Memory MeasurementsCapacity LimitsLoad and LongevitySystem Recovery and Robustness

System SecuritySystem OperationsEnd-User Documentation


20/48


System Test Plan - Test Environment Needs

This section describes the hardware, and software requiredduring the system test phase (including all test tools required to

perform system testing).


21/48


System Test Plan - Assumptions, Dependencies, Risks

Assumptions, Dependencies, and Risks:Assumptions are things that are outside the scope of the project but thatmust happen for the system integration test cycle to be successful.

Dependencies are things that are inside the scope of the project over which

integration has no control.

Risks are statements about things for which the integration prime can plancontingencies. There must be at least one contingency for each risk.


22/48


System Test Plan - Glossary

Provide a detailed glossary of the terms and acronyms used withinthe system test plan document


23/48


System Test Plan - References

Provide a reference to other documents to which the system test plan relates


24/48


System Test Plan : Review

The test plan is reviewed subsequently by:System Test Group, andProduct Verification


25/48


System Test Plan: Modification

All changes to the test plan are then reviewed.


26/48


System Test Plan: Archival

The reviewed system test plan is archived as described inthe Product or Project Development Plan and followsversion control.


27/48

System Test Case


28/48


System Test Cases : Coverage

System Test cases focus upon testing the system capabilities of the product as well as individual features with the goal of ensuringthat the system conforms with the product and featurespecifications.

System Test test cases are intended to test:All new functionality on at least one applicable platform and

Regression test existing functionality.

Test cases will be traced to the feature specification requirements,to ensure proper requirement coverage.


29/48


System Test Cases : Creation

Test cases for System Test are developed from :Product and Feature Specifications.

Development of test cases can proceed after, and only after:The Product Specification and

Related Feature Specifications

have been reviewed and signed off.

Note :

Design Specifications may be used if the Product and Feature Specifications do not contain all necessary information.


30/48


System Test Cases : Creation (Cont.)

In cases where free-form test cases (i.e. test cases executed by atester for which no written test case exists) result in the detectionof a problem, the Test Prime may decide to formalize the free-form test case into a written one. The decision on whether to dothis will be made at the Process Post-Mortem meeting.

Note : In order to facilitate the formalization of a free-form test case, any problem report

resulting from the test case must describe all the steps leading to the product defect.


31/48


System Test Cases : Format

Appendix II Section 7.2 provides a recommended form for writing test cases.


32/48


System Test Cases : Automated

Wherever practical, automated test cases are developed for thesystem testing phase.

Certain test areas, such as capacity limits, load testing, and performance measurements, lend well to automation,

Whereas other test areas, such as system operations and systemsecurity testing require a high degree of human (tester) interaction


33/48


System Test Cases : Review

Group consisting of members from System Test, Design,and Product Verification is responsible for reviewing

a particular test area or test item (feature),


34/48


System Test Cases: Modification

Test cases may be added, deleted, or modified. New andmodified test cases are subject to review


35/48


System Test Cases: Archival

All reviewed system test cases are archived and followversion control


36/48


System Test Cases: Execution

Test cases are executed on the target platform during thesystem testing phase.

Prior to the beginning of test case execution, the productmust meet the System Test Entrance Criteria as outlined inthe System Test Plan.

System Test will execute test cases only on approvedreleases of the product.


37/48

System Test Results


38/48


System Test Results: Creation

System test results must provide the following information: Name of tester(s).Software Release.Date and time of test execution.Pass, Fail, or Inconclusive indication.

Problem Report number (if applicable).


39/48


System Test Results: Review

The test results are reviewed by System Team and ProductVerification group


40/48


System Test Results: Modification

System rest results are not modified.


41/48


System Test Results: Archival

System test results are stored in the Information ResourcesCentre.


42/48

Regression Test Cases


43/48


Regression Test Cases - Purpose

Regression testing as the name implies is used to determinewhether changes or additions to software or hardware havecaused degradation in the quality of existing functionality.

Regression tests therefore, can never be run on an initial release of software or hardware.

Regression test cases are chosen from the full suite of testcases which were written for the entity under test.

Once the regression test cases have been executed, thenumber of test cases which failed provides an indication of whether more extensive testing needs to be done.


44/48


Regression Test Cases Purpose (Cont.)

The system test plan must identify the required pass rate.

Note: Regression test suites may only be run on subsystems and on the full

system.


45/48


Regression Test Cases - Planning

The System Test Team is responsible for determining theamount of regression testing required.


46/48


Regression Test Cases - Selecting

The regression test suite is chosen from the full suite of integration test cases for the entity under test. Eachregression test case chosen must have already beenexecuted successfully.

The ideal regression test case is one which exercises major functionality.

The frequency, with which regression test cases areexecuted, makes regression test cases ideal candidates for automation.


47/48


Regression Test Cases - Executing

In the event that a software defect has to be corrected and a new

release is made, the full suite system test cases must be run,followed by the regression test suite

If the software changes are minor, a partial regression may be

executed.

Note : However, if the number or extent of the software changes are significant, then either a full

regression or the complete integration test suite for the subsystem(s) affected will beexecuted. If the entire system is similarly affected, then the complete integration test

suite is executed for the system.

This determination is made by the System Test Team inconjunction with the Product Verification


48/48

References

[1] Boris Beizer, Software Systems Testing and Quality Assurance ,

1984.[2] Boris Beizer, Software Testing Techniques , 1990.[3] Ross Collard, Systems Testing and Quality Assurance Techniques ,Volumes I & II, May26th, 1994.[4] Yuk-Wha Li , Meridian Mail Products - System TestResponsibilities Issue 1.0, January

system test

Documents