pragmatic risk analysis and management - rbcs, inc risk analysis and management . ... 16-20 report...

Risk Based Testing Pragmatic Risk Analysis and Management

What is Risk Based Testing?


What is Risk Based Testing?

Risk: the possibility of a negative outcome Quality risk: the possibility that the system might not deliver key quality attributes, endangering quality outcomes Risk based testing uses an analysis of quality risks to prioritize tests and allocate testing effort Risk based testing involves key business and technical project stakeholders Risk based testing also means managing project risks

Risk Based Testing (v3.0)

www.rbcs-us.com

Copyright (c) 1999-2017 RBCS

What Are the Benefits?

Risk based testing includes the following benefits Running the tests in risk order gives the highest likelihood of discovering defects in severity order (“find the scary stuff first”) Allocating test effort based on risk is the most efficient way to minimize the residual quality risk upon release (“pick the right tests out of the infinite cloud of possible tests”) Measuring test results based on risk allows the organization to know the residual level of quality risk during test execution, and to make smart release decisions (“release when risk of delay balances risk of dissatisfaction”) If schedule requires, dropping tests in reverse risk order reduces the test execution period with the least possible increase in quality risk (“give up tests you worry about the least”)

All of these benefits allow the test team to operate more efficiently and in a targeted fashion, especially in time-constrained and/or resource-constrained situations


www.rbcs-us.com


The Foundation of Risk Based Testing

Quality risk analysis is the foundation of risk based testing Quality risk analysis includes three major elements

Identifying the quality risks Assessing the level of risk associated with each quality risk Capturing the quality risk information

A thorough, accurate quality risk analysis leads to thorough, well-aligned, properly sequenced risk based testing Let’s look at these activities more closely…


www.rbcs-us.com


Identification of Risk Items

Three main techniques In Agile lifecycles, risk analysis happens during iteration planning, after story grooming and before estimation For traditional lifecycles such as waterfall or RUP, we can have all stakeholder groups represented in a brainstorming session Alternatively, each stakeholder group representative interviewed by the lead analyst (usually a test lead or manager) in charge of the risk analysis

Any of these techniques can work, given stakeholder engagement


www.rbcs-us.com


Generic Quality Risk Categories (Part 1) Quality Risk Category What Kind of Problems Fit Into this Category

Competitive Inferiority Failures to match competing systems in quality.

Data Quality Failures in processing, storing, or retrieving data.

Date and Time Handling Failures in date-based and/or time-based inputs/outputs, calculations, and event handling.

Disaster Handling and Recovery

Failure to degrade gracefully in the face of catastrophic incidents and/or failure to recover properly from such incidents.

Documentation Failures in operating instructions for users or system administrators.

Error Handling, and Recovery

Failures due to beyond-peak or illegal conditions (i.e., knock-on effects of deliberately inflicted errors).

Functionality Failures that cause specific features not to work.


www.rbcs-us.com



Installation, Setup, Upgrade, and Migration

Failures that prevent or impede deploying the system, migrating data to new versions, including unwanted side-effects (e.g., installing of additional, unwelcome, unintended software such as spyware, malware, etc.)

Interoperability Failures that occur when major components, subsystems, or related systems interact.

Load, Capacity, and Volume

Failures in scaling of system to expected peak concurrent usage levels.

Localization Failures in specific localities, including languages, messages, taxes and finances, operational issues, and time zones.

Networked and Distributed

Failure to handle networked/distributed operation, including latency, delays, lost packet/connectivity, and unavailable resources.


www.rbcs-us.com



Operations and Maintenance

Failures that endanger continuing operation, including backup/restore processes.

Packaging/Fulfillment Failures associated with the packaging and/or delivery of the system or product.

Performance Failures to perform as required under expected loads.

Portability, Configuration, and Compatibility

Failures specific to different supported platforms, supported configurations, configuration problems, and/or cohabitation with other software/systems.

Reliability, Availability, and Stability

Failures to meet reasonable expectations of availability and mean-time-between-failure.

Security/Privacy Failures to protect the system and secured data from fraudulent or malicious misuse.


www.rbcs-us.com



Standards Compliance Failure to conform to mandatory standards, company standards, and/or applicable voluntary standards.

States and Transactions Failure to properly respond to sequences of events or to particular transactions.

User Interface and Usability

Failures in human factors, especially at the user interface.


www.rbcs-us.com


Capturing Risk Items

During brainstorming sessions or interviews, capture risk items on whiteboards, flipcharts, or notepads Afterwards, transfer them to a document As you capture risk items, deal with duplicate, overlapping, and miscategorized risk items If risk items arose from user stories or specifications, capture traceability information Capture the risks in a negative sense


www.rbcs-us.com


By-Products

Quality risk analysis typically produces three valuable by-products which you should capture

Project risks

Input document defects

Assumptions, issues, and questions

Capture these in the same spreadsheet as the quality risks for completeness


www.rbcs-us.com


What Is Risk Based Testing?

Exercise 1: Identifying quality risks


Exercise: Identifying Quality Risks

For a product you are currently working on (or have recently worked on), identify three to five quality risks

Use the quality risk categories or the quality characteristics as a checklist to help identify the risk item

Discuss your risk items and their categories or characteristics with the class


www.rbcs-us.com


Assessing Risk Items

For each risk item, assess two factors Likelihood: upon delivery for testing, how likely is the system to contain one or more bugs related to the risk item? Impact: if such bugs were not detected in testing and were delivered into production, how bad would the impact be?

Likelihood arises primarily from technical considerations, such as the programming languages used, the bandwidth of connections, and so forth Impact arises from business considerations, such as the financial loss the business will suffer, the number of users or customers affected, and so forth


www.rbcs-us.com


Likelihood Scale Likelihood Rating Criteria

Very likely 1 Almost certain to happen. (You’d be shocked if it didn’t happen.)

Likely 2 More likely to happen than not to happen. (You’d be surprised if it didn’t happen.)

Somewhat likely 3 About even odds of happening. (No surprise one way or the other.)

Unlikely 4 Less likely to happen than not to happen. (You’d be surprised if it did happen.)

Very unlikely 5 Almost certain not to happen. (You’d be shocked if it did happen.)

These ratings, while qualitative, are relatively objective. Technical stakeholders

should have little difficulty reaching agreement on the likelihood for any given risk

item, when discussing an existing system. (New systems based on unproven

technologies are harder to assess.)


www.rbcs-us.com


Impact Scale

Impact Rating High-level Criteria

Very high 1 Total disaster, with financial, reputational, security, and/or legal implications

High 2 Major short-term and long-term loss

Medium 3 Significant loss and damage

Low 4 Some loss and damage

Very low 5 Little to no loss or damage

These ratings, while based on clear criteria, are somewhat subjective. Business

stakeholders might have difficulty reaching agreement on the impact for a given risk

item.

It is also important to think of the typical bug related to a risk item that could occur,

rather than the most catastrophic bug that could occur.

You should create specific criteria for each impact rating, along with examples of

failures that fall into each rating


www.rbcs-us.com


Risk Priority Number

Calculated by multiplying the likelihood and impact Using scale shown earlier, RPN ranges from 1 (worst) to 25 (least bad) RPN measures aggregate level of risk associated with each risk item During test design and implementation, testers design test cases to cover risk items Test cases inherit the RPN of the risk item from which they descend Risk based test sequencing uses the RPN to determine the order of execution of the test cases


www.rbcs-us.com


Risk Based Test Effort Allocation

RPN determines the allocation of test effort Allocation can vary, depending on project parameters, constraints, and priorities In two coming slides, we look at two alternative examples

Quality-driven Schedule-driven

Other variations are possible The project management team may adopt either model


www.rbcs-us.com


Quality Driven Test Effort Allocation

RPN Range

Extent of Testing

Comments

1-10 Extensive Run a large number of tests that are both broad and deep, exercising combinations and variations of interesting conditions.

11-20 Broad Run a medium number of tests that exercise many different interesting conditions.

21-25 Cursory Run a small number of tests that sample the most interesting conditions.

Note: This has the effect that all risks item have some tests associated with them.


www.rbcs-us.com


Schedule Driven Test Effort Allocation

RPN Range

Extent of Testing

Comments

1-2 Extensive Run a large number of tests that are both broad and deep, exercising combinations and variations of interesting conditions.

3-5 Broad Run a medium number of tests that exercise many different interesting conditions.

6-10 Cursory Run a small number of tests that sample the most interesting conditions.

11-15 Opportunity Leverage other tests or activities to run a test or two of an interesting condition, but invest very little time and effort.

16-20 Report bugs only Do not test at all, but, if bugs related to this risk arise during other tests, report those bugs

21 - 25 None Neither test for these risks nor report bugs.

Note: This has the effect that some risk items have no tests associated with them.


www.rbcs-us.com


What Is Risk Based Testing?

Exercise 2: Assessing quality risks


Exercise: Assessing Quality Risks

For the three to five quality risks identified in the previous exercise, rate each risk’s likelihood and impact, using the given scale

Be ready to explain and justify the ratings

Multiple them together to obtain a risk priority number

Discuss your risk items’ likelihood, impact, and RPN ratings with the class


www.rbcs-us.com


Risk Based Testing In the Project


Sequential SDLCs

Risk based testing fits into the sequential software development lifecycles

Perform initial quality risk analysis during requirements Create test cases during design phase Adjust risk analysis periodically Sequence tests, report results, and, if necessary, triage test cases

Risk based testing is responsive to the schedule/quality trade-offs inherent in sequential SDLCs


www.rbcs-us.com


Iterative SDLCs

In iterative SDLCs, quality risk analysis is more iterative

Sequential: one large quality risk analysis effort during requirements

Iterative: High level quality risk analysis, followed by iteration-focused analyses

Risk based testing aligns with the philosophy behind iterative lifecycles


www.rbcs-us.com


Agile SDLCs

Agile SDLCs have short iterations and lightweight documentation

Repeated, focused quality risk analysis sessions for each iteration

Risk analysis results can guide unit test

Risk based testing aligns with the agile philosophies of change and documentation


www.rbcs-us.com


Agile Quality Risk Analysis

Agile quality risk analysis process (iteration planning)

Gather the agile team

List iteration backlog items

Identify functional, non-functional quality risks for each item

Assess identified risks: categorize each risk, determine risk level

Build consensus and ensure a good distribution of risk ratings

Use level of risk to choose extent of testing

Select appropriate test techniques for each risk item

Adjustments may occur during an iteration

Risk analysis may detect opportunities for early defect removal (e.g., problems in user stories)


www.rbcs-us.com


Handling Project Risks

In addition to use testing to manage quality risks, testing is also subject to risk

To discover risks to the testing effort, ask yourself and other stakeholders:

What could go wrong on the project that would delay or invalidate your test plan and/or estimate?

What kind of unacceptable testing outcomes do you worry about?

For each project risk, you have four options: Mitigation: Reduce the likelihood or impact through preventive steps

Contingency: Have a plan in place to reduce the impact

Transfer: Get some other party to accept the consequences

Accept (or ignore): Do nothing about it (best if both likelihood and impact are low!)

Insurance (a form of transference) is usually not available


www.rbcs-us.com


Sample Test-related Project Risks

Logistics or product quality problems block tests

Test deliverables won’t install

Excessive change invalidates results, requires test updates

Insufficient or unrealistic test environment(s)

Test environment support unreliable

Gaps in test coverage revealed during test execution

Slips in test start dates and/or deliverables to test

Budget and/or staffing cuts

Debugging in the test environment


www.rbcs-us.com


Risk Based Testing in the Project

Exercise 3: Managing project risks


Exercise: Omninet Testing Risks

For the same project as before, identify three to five project risks

For each risk, select a method for managing the risk

Discuss your risks and management methods with the class


www.rbcs-us.com


For More Information…



…Contact RBCS For over 20 years, RBCS has delivered consulting, training, and expert services to clients, helping them with software and hardware testing. Employing the industry’s most experienced and recognized consultants, RBCS advises its clients, trains their employees, conducts product testing, builds and improves testing groups, and hires testing staff for hundreds of clients worldwide. Ranging from Fortune 20 companies to start-ups, RBCS clients save time and money through improved product development, decreased tech support calls, improved corporate reputation and more. To learn more about RBCS, visit www.rbcs-us.com.

Address: RBCS, Inc. 31520 Beck Road Bulverde, TX 78163-3911 USA Phone: +1 (830) 438-4830 E-mail: [email protected] Web: www.rbcs-us.com Twitter: @RBCS, @MisterSDET, @LaikaTestDog Facebook: @TestingImprovedbyRBCS LinkedIn: https://www.linkedin.com/in/rex-black YouTube: https://www.youtube.com/user/RBCSINC

www.rbcs-us.com


pragmatic risk analysis and management - rbcs, inc risk analysis and management . ... 16-20 report...

Documents