Project TestingA guide to types of tests for engineering projects

Thomas G. CleaverUniversity of Louisville

Department of Electrical and Computer Engineering

February 25, 2005

ReferencesThis presentation was developed using the following sources:• Suresh, “General Testing Types,” [online article] (2002, April 5)

Available at HTTP: http://www.software-engineer.org/article_read.php?article_id=20000349

• “Software Development and QA,” [online article] (2003) Available at HTTP: http://www.ximxim.com/software_qa_cycle_types_of_tests.html

• “Automated Test Concepts,” [online article] (2003) Available at HTTP: http://www.lw-tech.com/q1/ug_concepts.htm

• “Testing, Verification, Debugging CS577B Workshop,” [Adobe Acrobat pdf file] (2002, Spring) Available at HTTP: http://sunset.usc.edu/classes/cs577b_2002/EC/16/EC-16.pdf

• C. Kaner, “Architectures of Test Automation,” [online article] (2000, August) Available at HTTP: http://www.kaner.com/testarch.html

Types of Tests

• Unit Test/Component Test

• Integration Test• Interface Test• Paths Test• Functional Test• System Test• Performance Test• Alpha Test

• Beta Test• Acceptance Test• Regression Test• Smoke Test• Stress Test• Validation Test• White Box Test• Black Box Test• Field Test

Unit Test/Component Test

• Software:– A Unit Test or Component Test is a test of the

smallest functional segment of code. This is usually done by the developer at the time of coding. Tests usually consist of applying sample inputs and verifying that the outputs are as expected.

• Hardware:– A Unit Test or Component Test is a test of a small

hardware module. This is usually done by the engineer who designed the module. It frequently consists of applying a range of inputs, and then measuring the corresponding outputs for accuracy.

Integration Test

• Software:– Integration tests are designed to test integrated software

components to determine if they actually run as one program.– Integration tests follow unit tests.– Integration tests expose flaws that arise from interaction of

(presumably perfect) software components.

• Hardware:– Integration tests are designed to test the functionality of a

system of individually tested hardware modules when they are connected together.

– Integration tests follow unit tests.– Use the integration test to determine if subsystems are

interacting correctly.

Integration and Interface Tests

• Integration tests see whether components that work separately work together.

• Interface tests focus on the points of contact between two components.

Paths Test (software)

• A Paths Test is a test of various paths through the system, usually based on the paths that a user is expected to take.

• It is usually not practical to test all paths.• Test all of the most frequent and most

important paths.

Functional Test

• Functional tests ensure that the application (hardware and/or software) works the way the stakeholders intend it to work.

• Unit tests, integration tests, etc. are often functional tests.

Functional Test Features

• Valid Input - identified classes of valid input must be accepted.

• Invalid Input - identified classes of invalid input must be rejected.

• Functions - identified functions must be exercised.

• Output - identified classes of application outputs must be exercised.

• Systems/Procedures - interfacing systems or procedures must be invoked.

System Test

• System testing ensures that the entire integrated system (software and/or hardware) meets requirements. It tests a configuration to ensure known and predictable results.

• Integration tests and acceptance tests are examples of system tests.

• Between unit tests and system tests, there may also be module tests and subsystem tests.

Performance Test (software)

• Performance tests determine the runtime performance of the application.

• Performance testing is used to measure:– processing speed– response time– resource consumption– throughput

• Performance testing is sometimes coupled with stress testing.

Alpha Test

• Alpha and beta testing are terms used almost exclusively with software projects.

• In an Alpha Test, the software or hardware is exposed to a number of “friendly” users, who were not involved in development or previous testing.

Beta Test

• In a Beta Test, the hardware or software is assumed to have nearly-commercial quality so it is exposed to “non-friendly” users. The information gathered is fed into a final build of the system which becomes the commercial version.

• Beta testing is performed to uncover errors that only the end user seems able to find. The beta test is a "live" application of the system. The customer records all the problems that are encountered during beta testing and reports these to the developer. The developer makes modifications, and then prepares the release of the product to the entire customer base.

Acceptance Test

• Acceptance testing is performed before the application is implemented into the production environment.

• Acceptance testing is conducted to determine whether a system satisfies its acceptance criteria by validating requirements have been met.

• Based on the results of the Acceptance Test, the customer determines whether to accept or decline the system.

• Acceptance testing is frequently done by the customer.

Regression Test (software)

• In a software system, when one component is changed, it may cause unforeseen changes in other, seemingly unrelated, components.

• A Regression Test is basically a retest of the entire system after a modification is made.

• Regression tests assure that no adverse changes are introduced as a result of maintenance or upgrade of the software.

• Procedurally, regression testing applies a rigorously defined set of inputs and outputs, to ensure the new version has introduced no inconsistencies with old versions.

Smoke Test

• Software: This is a standard, relatively brief test suite. Every time there’s a new build, these tests are run to check whether there is anything fundamentally wrong with the software. If so, the build is rejected. Only builds that pass the smoke test get more thorough testing.

• Hardware:1. A stress test, or

2. An error made by a student that causes the project to emit smoke.

Stress Test

• A Stress Test deliberately applies overload conditions to a system to test limit conditions, scalability and fault tolerance.

• The system’s performance limits are stretched (and broken, usually).

• Stress tests are used to gather performance statistics, and to determine whether the system handles failure gracefully.

Validation Test

• A Validation Test determines how well a component corresponds to its specification in the design.

White Box Test (software)

• A White Box Test (aka Glass Box Test) is an examination of the code, looking for errors or ways to break the code.

• It is usually done as part of a unit test.• It is usually done by the programmer or

someone else familiar with the code.

• The tester must consider boundary conditions, errors, and limits.

Black Box Test

• The “black box” refers to a piece of software or hardware treated as a single unit.

• The contents of the “black box” is treated as unknown.

• Only the inputs and outputs of the “black box” are considered.

• A Black Box Test isn’t concerned with the system’s implementation, but only gauges whether it accepts the correct inputs and produces correct outputs.

Black Box Test (Continued)

Group inputs and outputs:

– group valid inputs together.

– group invalid inputs together.

– group questionable inputs together.

– group outputs and identify erroneous groupings.

Focus on boundary values, and inputs which may be strange or unexpected.

Field Test

• A Field Test is a test conducted “in the field,” that is, in the place where the hardware or the software is deployed.

• Sometimes a field test is done by the supplier upon delivery to the customer.

• Sometimes a field test is done in response to an error report by the customer.