carnegie mellon university notice · testing. the . execution. of an object under test (out) under...

1A Taxonomy of Testing TypesSEI Webinar© 2015 Carnegie Mellon University

Carnegie Mellon University Notice

This video and all related information and materials (“materials”) are owned by Carnegie Mellon University. These materials are provided on an “as-is” “as available” basis without any warranties and solely for your personal viewing and use.

You agree that Carnegie Mellon is not liable with respect to any materials received by you as a result of viewing the video, or using referenced websites, and/or for any consequences or the use by you of such materials.

By viewing, downloading, and/or using this video and related materials, you agree that you have read and agree to our terms of use (www.sei.cmu.edu/legal/).

© 2015 Carnegie Mellon University.


Copyright 2015 Carnegie Mellon UniversityThis material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense.

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

This material has been approved for public release and unlimited distribution except as restricted below.

This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at [email protected].

Carnegie Mellon® is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.

DM-0002727

© 2015 Carnegie Mellon University

A Taxonomy of Testing TypesSoftware Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213

Donald G. Firesmith


Topics

Relevant Testing ChallengesGoals of PresentationWhat is Testing?Presentation ScopeTesting TypesConclusion


Relevant Testing Challenges


Relevant Testing Challenges

Many testers are only aware of a minority of types of testing, let alone know how to perform them.Test managers and developers are aware of even fewer testing types.The test strategies, project test plans, and test sections of system/software development plans tend to identify only a very small number of types of testing (e.g., unit, integration, system, and acceptance testing).• Not planned → Not performed


Goals of the Presentation


Goals of the Presentation

Make it clear that:• There are many different types of testing.• Testing is a complex discipline with its own technical jargon.• There is a lot of overlap between different classes of testing types.

• Think multiple classification (object-oriented design)or multiple inheritance (object-oriented programming).

Get you to take a look at your:• Test strategies and test plans and ask yourselves “Are they sufficiently complete?”• Testers and other testing stakeholders and ask yourselves “Do they need additional

training in testing types.


Polling Question 1

How many different types of testing do you typically use on a project?O 1-5O 6-10O 11-15O 16+


What is Testing?


What is Testing?

TestingThe execution of an Object Under Test (OUT) under specific preconditions with specific stimuli so that its actual behavior can be compared with its expected or required behavior• Preconditions: pretest mode, states, stored data, or external conditions• Stimuli:

- Calls, commands, and messages (control flows)- Data inputs (data flows)- Trigger events such as state changes and temporal events

• Actual Behavior:- During Test:

– Calls, commands, and messages (control flows)– Data outputs (data flows)

- Postconditions: post-test mode, states, stored data, or external conditions


Presentation Scope


Presentation Scope Verification and Validation (V&V) MethodsQuality Control (QC)

Test Evaluation OtherIn Scope

T&E

Analysis

StaticAnalysis

DynamicAnalysis

Demonstration

Desk Checking

Inspection ReviewWalk-

Through

Peer Review

Formal Review

“Inspection” Certification Reuse Warantee

Legend

Static Dynamic


The Taxonomy of Testing Types


Types of Testing

A type of testing is:• A specific way to perform testing• A class or subclass of testing• Much narrower in scope than a testing paradigmThere are relationships between the various types of testing.Most testers know:• A lot about a few types of testing• A little about some additional types of testing• Very little about a sizable number of testing types


Polling Question 2

Have you ever seen a taxonomy of testing types(i.e., a hierarchical categorization of different ways to test)?O YesO NoO Not Sure


Types of Testing

16 Categories of Testing Types Answering the 5W+2H Questions:• What?• When?• Where?• Who?• Why?• How?• How Well?These supertypes are not disjoint (think multiple inheritance)!

Object Under Test (OUT)-Based Test Types

Quality-Characteristic-Based Test Types

Phase-Based Test Types

Built-In-Test (BIT) Types

Test Types

What-Based Test Types(What Gets Tested)

When-Based Test Types(When Testing Occurs)

Domain-Based Test Types

Why-Based Test Types(Why Testing Occurs)

Driver-Based Test Types

Reason-Based Test Types

Who-Based Test Types(Who Does Testing)

Collaboration-Based Test Types

Organization-Based Test Types

Role-Based Test Types

Where-Based Test Types(Why Testing Occurs)

Organization-Location-Based Test Types

Physical-Location-Based Test Types

How-Based Test Types(How Testing is Performed

Automation-Based Test Types

Level-of-Scripting-Based Test Types

Technique-Based Test Types

How-Well-Based Test Types(Quality Verified)

Lifecycle-Based Test Types

Temporal-Order-Based Test Types


Types of Testing

WHAT is Tested


What: by Object Under Test (OUT)

OUT-Based Testing

Model Testing

Software Testing

System Testing

Hardware Testing

Tool/Environment Testing

Data CenterTesting

What-Based TestingTesting


What: by Object Under Test (OUT) – Model Testing

OUT-Based Testing

Model Testing

Requirements Model Testing

Design Model Testing

Architecture Model Testing



What: by Object Under Test (OUT) – Hardware Testing

OUT-Based Testing

Hardware Testing

HW Qualification Testing


Power-off Testing

Highly Accelerated Life Testing (HALT)

Hardware Stress Testing

Continuity Testing


What: by Object Under Test (OUT) – Software Testing

OUT-Based Testing

Software Testing

SW Unit [Component] Testing

SW Application Testing

SW Integration Testing



What: by Object Under Test (OUT) – System Testing

OUT-Based Testing

System Testing

System Integration Testing

SoS Integration Testing

SoS Testing

Subsystem Testing

System Testing

Hardware-in-the-Loop (HIL) Testing

Processor-in-the-Loop (PIL) Testing

Human-in-the-Loop (HIL) Testing


Software-in-the-Loop (SIL) Testing


What: by Object Under Test (OUT) – Data Center Testing

OUT-Based Testing

Data Center Testing

Configuration Testing

Integrated System Testing (IST)

Failover and Restore Testing


Network Traffic Testing


What: by Object Under Test (OUT) – Tool / Environment Testing

OUT-Based Testing

Tool/Environment Testing

Development Environment

Testing

Development Tool Testing

What-Based Testing

Test Environment Testing

Test Tool Testing

Testing


What:by Domain

Domain-Specific Testing

Domain-Independent Testing

Domain-Based Testing

Flight Testing

Sea Trials

Live Fire Testing

Wind Tunnel Testing

Crash Testing

Exhaust Emissions Testing

Many Types of Domain-

Independent Testing



Types of Testing

WHEN Testing Occurs


When: by Temporal Order

Order-Based Testing

Order-By-Direction Testing

Layer-Based Testing

Order-By-Component Testing

Feature-Based Testing

When-Based TestingTesting

Top-Down Testing

Bottom-Up Testing

Outside-In Testing

Subsystem-Based Testing


When: by Lifecycle

Lifecycle-Based Testing

Incremental Testing

Waterfall Testing

DevOps Testing

Continuous Testing (CT)

Agile Testing


V-Model Testing

At-The-End Testing


When: by Phase

Beware ofSynonyms andAlmost Synonyms!

Developmental Testing (DT)

Acceptance Testing (AT)

Business Acceptance Testing (BAT)

Contract(ual) Acceptance Testing (CAT)

Operational Acceptance Testing (OAT)

Regulation Acceptance Testing (RAT)

Customer Acceptance Testing (CAT)

Site Acceptance Testing (SAT)

Factory Acceptance Testing (FAT)

Production Acceptance Testing (PAT)

Operational Testing (OT)

Initial Operational Testing (IOT)

Follow-on Operational Testing (FOT)

Qualification Operational Testing

(QOT)

Many Types of Developmental

Testing

Operational Effectiveness Testing

Operational Suitability Testing

User Acceptance Testing (UAT)

Data CenterTesting


Phase-Based Testing


When: byBIT Execution Time

Built-In Testing (BIT)

PeriodicBuilt-In Testing

(PBIT)

Power-UpBuilt-In Testing

(PupBIT)

Interrupt-driven Built-In Testing

(IBIT)

User-initiated Built-In Testing

(UBIT)

ShutdownBuilt-In Testing

(SBIT)

OngoingBuilt-In Testing

(OBIT)

Self-Testing via Heartbeat

Self-Testing via Prognostics and

Health Management(PHM) Subsystem

Self-Testing via Assertion Checking



Types of Testing

WHY Testing is Being Performed


Why: by Driver

Verifies

Driver-Based Testing

Requirements-Driven Testing

(tests what + how-well)

Design-Driven Testing

(tests how)

Architecture-Driven Testing

(tests how)

Needs-Driven Testing

(tests why)

System MeetsSystem Requirements

System Conforms to

Design

System Conforms to Architecture

System Meets Stakeholder

Needs

Regulatory-Compliance Testing (tests compliance)

System Complies with Laws or Regulations

Why-Based TestingTesting

Validates VerifiesVerifiesVerifies


Why: by Reason

Reason-Based Testing

Initial Testing

Regression TestingRetesting

Partial Regression

Testing

Complete Regression

Testing

Smoke Testing

Error Seeding

ReuseTesting

COTS Testing

Open Source Testing

Legacy Testing

Why-Based TestingTesting


Types of Testing

WHO Performs Testing


Who: byCollaboration

Collaboration-Based Testing

Individual Testing

Pair Testing

Buddy Testing

Group Testing

Flash Mob Testing

Who-Based TestingTesting


Who: byOrganization

Development Organization

Testing

Subcontractor Testing

Prime Contractor

Testing

Acquisition Organization

Testing

IndependentTest

Organization Testing

Operations Organization

Testing

User Organization

Testing

COTS Vendor Testing

DT Organization

Testing

OT Organization

Testing

Who-Based TestingTesting

Organization-Based Testing


Polling Question 3

Who performs testing on your projects? Check all that apply.

□ Project-internal Testers

□ Independent Testers

□ Developers

□ Specialty Engineers (e.g., performance, reliability, safety, security, human factors)

□ Quality Engineers

□ Others


Who: by Role Who-Based TestingTesting

Role-Based Testing

Security Engineer Testing

Safety Engineer Testing

Human Factors Engineer Testing

Programmer Testing

Requirements Engineer Testing

Architect Testing

Database Admin Testing

Network Admin Testing

Sys Admin Testing

Beta Tester Testing

User as Tester Testing

Closed Beta

Testing

OpenBeta

Testing

Alpha Tester Testing

Embeded Tester Testing

Independent Tester Testing

Buddy Testing

Pair Testing

Tester Testing

Operator Testing

Developer Testing

User Testing


Types of Testing

WHERE Testing is Performed


Where: by Organizational Location

Outsourced Testing

Insourced Testing

Organizational-Location-Based Testing

Where-Based TestingTesting


Where: by Physical Location

Physical-Location-Based Testing

Local Testing

Distributed Testing

Cloud Testing

Where-Based TestingTesting


Types of Testing

HOW Testing is Performed


How: by Level of Automation

Level-of-Automation-Based Testing

Manual Testing Automated Testing- Test Script Generation- Test Data Generation- Test Script Execution- Test Report Generation

Data-Driven Testing

Action-Keyword Testing

Script-Based Testing

Record-Playback Testing

Model-Based Testing

How-Based TestingTesting


How: by Level of Scripting

Unscripted TestingScripted Testing

Level-of-Scripting-Based Testing

Monkey Testing


Exploratory Testing


How: by Technique

Blackbox Testing

Whitebox Testing

Experience-Based Testing

Graybox Testing

Random Testing(Cat on the Keyboard)

Patterns-Based Testing

Test-Technique-Based Testing



How: by Technique- Blackbox Testing

Blackbox Testing

Cause and Effect Testing

Classification Tree Testing

Combinatorial Testing

Decision Table Testing

Requirements Testing Risk-Based

TestingScenario Testing State-Based

Testing

Syntax Testing

Manual Procedure

Testing

User Interface Navigation

Testing



End-to-End Testing


How: by Technique- Graybox Testing

Graybox Testing

Boundary Value Testing

Equivalence Class Testing




How: by Technique- Whitebox Testing

Whitebox Testing

Control Flow Testing

Data Flow Testing

Branch Testing

Condition Testing

Statement Testing

All Definitions Testing

All Uses Testing




How: by Technique- Experience-Based Testing

Experience-Based Testing

Error Guessing Testing

Exploratory Testing

Bug Hunt Testing

Galumphing




How: by Technique- Random Testing

Random Testing(Cat on the Keyboard)

Monkey Testing

Fuzz Testing



Stuck Key Testing

Shoe Testing


Types of Testing

HOW WELL Object Under Test Functions


How Well:by QualityCharacteristic

Based on the associatedquality characteristicand its associatedquality attributes:• Uncover related defects• Determine level of quality

Capacity Testing

Compatibility Testing

Consistency Testing

Data Migration Testing

Functionality Testing

Performance Testing

Configuration Testing

Reliability Testing

Robustness Testing

Safety Testing

Security Testing

Usability Testing

Quality-Characteristic-Based Testingby Quality Characteristic

Correctness Testing

Interoperability Testing

Scalability Testing

Backwards Compatibility Testing

Infrastructure Compatibility Testing

Flexibility Testing Internationalization Testing

PersonalizationTesting

How-Well-Based TestingTesting

Mobile Testing


How Well: by Quality– Capacity Testing

Capacity Testing

Load Testing

Stress Testing

Volume Testing

Quality-Characteristic-Based Testing



How Well: by Quality– Reliability Testing

Reliability Testing

Reliability Enhancement Testing

Reliability Growth Testing


Endurance (Stability) Testing

Reliability Mechanism Testing



How Well: by Quality– Robustness Testing

Robustness Testing

Error Tolerance Testing

Fault Tolerance Testing

Failure Tolerance Testing

Environmental Tolerance Testing

Acoustic Tolerance Testing

Shock/Drop Tolerance Testing

Acceleration Tolerance Testing

Electromagnetic Compatibility (EMC) Tolerance Testing

Temperature Tolerance Testing

Pressure/Leakage Tolerance Testing

Vacuum Tolerance Testing

Radiation Tolerance Testing

Temperature Cycling Tolerance Testing

Pressure Cycling Tolerance Testing


Fault Injection Testing

Failover and Recovery Testing

System Error Testing

Human Error Testing

Communication Error Testing

Hardware Error Testing

Software Error Testing



How Well: by Quality– Security Testing

Encryption Testing

EMSEC Testing

Blue Team Testing

Red Team Testing

Infrastructure Testing

Penetration Testing

Anti-Tamper Testing

Anti-Spoofing Testing

Access Control Testing

Security Testing




How Well: by Quality– Usability Testing

Usability Testing

A/B Testing

Accessibility Testing

Alpha Testing

Content Usage Testing

Beta Testing

(Flash) Mob Testing

Closed Beta Testing

Open Beta Testing




Conclusion


Conclusion

Most systems require quite a few different types of testing.Most testers are not aware of the majority of the different types of testing.• If you are not aware that it exists, then you don’t know whether you need it.These types of testing can be organized into a taxonomy by the 5W + 2H questions.This taxonomy has several uses:• Ensure the test strategy is sufficiently complete with no important type of testing

overlooked.• Organize testing types to make them and their relationships more understandable.• Augment test training materials.• Help categorize and understand limitations of testing tools.

carnegie mellon university notice · testing. the . execution. of an object under test (out) under...

Documents