qa club kiev #17 measuring quality by volodymyr prymakov

Measuring QualityRelease Readiness Criteria and Quality Reporting

11 Years general working experience in IT/QA (30+ projects)

----------

•QA Consultant at Ciklum Interactive

•QA Manager (Manual)

•Automated Testing Manager

•Load Testing Manager

•Project Manager

----------

•Hobbies: Job, Wife and 2 children, Sport

----------

Skype: vladimir.primakov

Linkedin: ua.linkedin.com/in/vladimirprimakov/

Email: v.v.primakov@gmail.com

Some Words About Me

Volodymyr Prymakov (Vladimir Primakov)

The presentation will cover the following items:

• Release Readiness criteria as a part of a master test plan, their types and details. (~ 30 min)

• Release readiness criteria appliance and areas of influence. (~ 5 min)

• Quality report and quality metrics + their connection to release readiness criteria. (~ 15 min)

• Concluding release readiness. (~1 min)

• Questions (~ 10 min)

Presentation’s Structure

Presentation Plan

Release Readiness Criteria as a part of Master Test Plan

Release Readiness Criteria

Master Test Plan

Categories of Release Readiness Criteria

Analyzing Quality

(RESULTS)Catching Quality

(ACTIVITIES)

1. New Functionality Testing

2. Regression Testing

3. Test Environments Testing

4. Unit & Integration testing

5. Special Types of Testing

Any Ideas?

Criteria for New Functionality Testing

Short Example

Test Environment Priority

Coverage of Test Environments

Tested Functionality Priority

Minimum Test Deepness

Minimum Functionality Coverage

MaximumFail Rate %

The Highest one 100% Features of Any Priority

Detailed Exploratory testing

100% 0%

High 100% (all high priority test environments)

All priorities Happy path Exploratory testing

100% 0%

Medium Please See test environments related criteria

Low Please See test environments related criteria

Short Example

MaximumFail Rate %

100% 0%

Short Example

MaximumFail Rate %

100% 0%

Short Example

MaximumFail Rate %

100% 0%

Short Example

MaximumFail Rate %

100% 0%

Short Example

MaximumFail Rate %

100% 0%

Short Example

MaximumFail Rate %

100% 0%

Expanded Example

MaximumFail Rate %

The Highest one 100% High (Blocker/Critical)

100% 0%

Medium (Major) Detailed Exploratory testing

100% 0%

Low (Normal/Trivial) Happy path Exploratory testing

100% 0%

Test Deepness

MaximumFail Rate %

100% 0%

Test Deepness

TEST DEEPNESS / DETALIZATION

Test Case Based Thought Out Test

Detailed Exploratory Test

Happy Path Exploratory test

Smoke test

Nothing

Test Coverage

MaximumFail Rate %

100% 0%

Test Coverage

Test Coverage is 71 %, But should be 100%

MaximumFail Rate %

100% 0%

Fail Rate %

Functionality PriorityMaximum Allowable Bugs Number

(Story, Improvement, New Feature)

BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs

High (Blocker / Critical) 0 0 0 * *

Medium (Major) 0 0 0 * *

Low (Normal/Trivial) 0 0 * * *

PASS CRITERIA

*if too many bugs are in place for the functionality – the bugs number can be considered for any specific case separately

Fail Rate %

BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs

PASS CRITERIA

Fail Rate %

BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs

PASS CRITERIA

Fail Rate %

BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs

PASS CRITERIA

Fail Rate %

BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs

Features of Any Priority 0 0 * * *

PASS CRITERIA (short and not so strict)

Fail Rate %

BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs

Status: FAIL RATE %

Blocked

Fail Rate %

BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs

Status: FAIL RATE %

Blocked

Failed Release Readiness Criteria

Fail Rate %

BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs

Status: FAIL RATE %

Blocked

Status: Pass Rate %

Failed With Acceptable Bugs

Fail Rate %

BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs

Status: FAIL RATE %

Blocked

Status: Pass Rate %

Fail Rate %

BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs

Status: FAIL RATE %

Blocked

Status: Pass Rate %

Fail Rate %

BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs

Status: FAIL RATE %

Blocked

Status: Pass Rate %

Passed

Fail Rate %

Test Coverage

Fail Rate is 29%, But should be 0%

Test Environment Coverage

MaximumFail Rate %

100% 0%

Test Environments Coverage

Test EnvironmentsType OS Browser/Device Resolution Priority

PC Win 7 Chrome 1920*1080 Highest

PC Win 8 (metro) IE 10 1920*1080

HighMac MacOS Firefox 1920*1080

iPad iOs7 iPad 4 2048*1536

Mac MacOS Safari 2880*1800

MediumPC Win 7 IE 11 1366*768

PC Linux Firefox 1366*768Low

PC Win 8 (metro) IE 10 1920*1080

There is only 1 highest

priority environment so the

coverage, which it gives for

the priority, is 100%.

PC Win 8 (metro) IE 10 1920*1080

1 environment of 3 high priority

environments gives ~ 33% coverage.

All these environments together

gives 100% coverage for the priority.

Test Environment Coverage - Example

What Else?

DEFINTION OF DONE (EXAMPLE)

User story reviewed & adjusted correspondingly

Unit-test created and passed

Integration test created and passed

Pear review passed

UX-review passed

Manual Test Passed

End User Test Passed…

Definition of Done

Criteria for Regression Testing

Test Deepness

Test Coverage

Test Deepness

Test Coverage

Test Deepness

Fail Rate

Test Coverage

Test Deepness

Fail Rate

Test Environments

Example

Tested Scenario Priority

MaximumFail Rate %

The Highest one 100% High Detailed Exploratory testing

100% 0%

Medium (That might be affected by recent product changes)

Happy path Exploratory testing

100% 0%

Other Medium Happy path Exploratory testing

50% 0%

High See test environments related criteria

Medium See test environments related criteria

Low See test environments related criteria

Example

MaximumFail Rate %

100% 0%

50% 0%

Example

MaximumFail Rate %

100% 0%

50% 0%

Criteria for Test Environment TestingPossible Test Environment Ingredients

Internet Speed Antivirus, Firewalls

Plugins

Integration with other

applications and API

User Rights

RuntimesExternal Devices

Screen Resolutions and Modes

Hardware and Software Platforms and Modes

Web-browsers

Mobile Software Platforms

External Libraries

Hardware

Characteristics and

Capacity

Robots (HD,

Software,

Capabilities)

Criteria for Test Environment Testing

Product-wide Test Deepness

Product-wide Test

Deepness

Full Regression

Full Regression for

functionality of certain

priority

Main product Flows test /

User Acceptance Test /

Cover test

Selective Regression

Smoke Test

Coverage of

Test Environments

Coverage of

Test Environments

Fail Rate

Test Environment Priority Target (functionality) Minimum Test Deepness

Minimum Coverage of Test environments

MaximumFail Rate %

Highest Please see the release readiness-criteria for new functionality and oldfunctionality

High New functionality Please see release readiness-criteria for newfunctionality

Old (Regression-related) functionality

Selective Regression and/or Main product Flows test

100% 0%

Medium New and Old (Regression-related) functionality

Product-wide Smoke test

75%* 0%

Low New and Old (Regression-related) functionality

50%* 0%

Example

*Not covered environments may be covered next release and so on

MaximumFail Rate %

100% 0%

75%* 0%

50%* 0%

Example

MaximumFail Rate %

100% 0%

75%* 0%

50%* 0%

Example

Criteria for Unit & Integration Testing

Test Coverage

Fail RateGUI-LESS

Functionality

Functionality*

Unit & Integration Testing

Test Coverage

Fail Rate

*Criteria may be not so strict, because GUI functionality can be also manually tested

Criteria for Unit & Integration TestingExample

Integration Testing or Unit-TestingPriority of functionality

MaximumFailRate %

High 100% 0%*

Medium 0% 0%*

*Depends on severity of the problems found in

integration/unit testing

Criteria for Special Types of testing

Type of testing* Sufficient?

(yes/no)

Passed? Conclusions

Load/Performance

testing (client/server

NOT READY

Security Testing NOT READY

UX-Testing READY

(Accessibility Testing) NOT READY

GENERAL CONCLUSSION: NOT READY

* Usually has separate release readiness criteria, but in any case it should be sufficient

and passed.

Special types of testing, which are obligatory to be conducted and passed

Release readiness criteria appliance and areas of influence

Release readiness criteria

Defines testing activities and product quality sufficient for release

Influences testing strategy, testing approaches, and activities needed

Defines QA Resources requirements

Defines Quality reporting structure and content

Helps making conclusion about release readiness

For those who don’t sleep yet

thank you for being cheerful!

Quality Reporting and its connection to release readiness criteria

Quality Reporting

1. Number of Bugs and their severity

2. New Functionality Testing & Results

3. Regression Testing & Results

4. Test Environments Testing & Results

5. Unit & Integration testing & Results

6. Special Types of Testing & Results

7. Concluding Release Readiness

Quality Report Structure

Quality Reporting

1. Number of Bugs and their severity

2. New Functionality Testing & Results

3. Regression Testing & Results

4. Test Environments Testing & Results

5. Unit & Integration testing & Results

6. Special Types of Testing & Results

7. Concluding Release Readiness

Quality Report Structure

Quality Reporting

All Not Closed Bugs

Number of Bugs and their severity

All Not Closed Bugs

All Not Closed Bugs – By Component

All Not Closed Bugs – Not-Assigned to any Sprint

Number of Bugs and their severityAll Not Closed Bugs – Not-Assigned to any Sprint

Bugs List (for All not-closed bugs) – That fails release readiness criteria

Assigned to sprints

Not-Assigned to sprints

Make a separate release readiness conclusion for this criterion type

New Functionality Statuses (From Tasks Management System)

New Functionality Testing & Results

Features Statuses (Current release)

Not-Resolved Features By Components(Current release)

Test Environments

Failed Functionalities Details (Optionally)

Regression testing results for the highest priority environment

Regression Testing & Results

By Components

Regression testing results for the highest priority environment

Failed Functionalities Details (Optionally)

Test Environments Testing & Results

Test plans for test environments of different priorities

Test Environments Testing & Results

GUI-less functionality

Unit & Integration testing & Results

Type of testing Coverage Fail Percentage Conclusions

Unit-test 50% 0% READY

Integration test 100% 0% READY

GENERAL RELEASE READINESS CONCLUSSION: READY

Special Types of Testing & Results

(yes/no)

Passed? Conclusions

Load/Performance

NOT READY

UX-Testing READY

GENERAL RELEASE READINESS CONCLUSSION: NOT READY

and passed.

(yes/no)

Passed? Conclusions

Load/Performance

NOT READY

UX-Testing READY

and passed.

(yes/no)

Passed? Conclusions

Load/Performance

NOT READY

UX-Testing READY

and passed.

(yes/no)

Passed? Conclusions

Load/Performance

NOT READY

UX-Testing READY

and passed.

Concluding Release Readiness

Criterion Status

1. Number of Bugs and their severityNOT READY

2. New Functionality Testing & ResultsREADY

3. Regression Testing & ResultsREADY

4. Test Environments Testing & ResultsNOT READY

5. Unit & Integration testing & ResultsREADY

6. Special Types of Testing & ResultsNOT READY

Concluding Release Readiness

Criterion Status

1. Number of Bugs and their severityNOT READY

2. New Functionality Testing & ResultsREADY

3. Regression Testing & ResultsREADY

4. Test Environments Testing & ResultsNOT READY

5. Unit & Integration testing & ResultsREADY

6. Special Types of Testing & ResultsNOT READY

Instruments Facilitating Quality Reporting

As Gadgets, iFrames,

Screenshots, Links

plugins

Thank you! ;)

QUESTIONS?

qa club kiev #17 measuring quality by volodymyr prymakov

test environments testing4

new functionality testing2

short examplecriteria

highest one100

special types of testing

quality metrics

qa consultant

following items

Education

kiev, i love you - kiev for kids

qa club kiev #18 - test management in google sheets

volodymyr lisin teasers

national symphony orchestra of ukraine volodymyr …national...

volodymyr prykhodko

volodymyr magas

kiev 4 kiev 4a

qa club kiev #20. mobile testing. tips & tricks

kiev, i love you - art in kiev

192 volodymyr velykyy

sap forum kiev · Максимизация выгод от...

th2011 levchenko volodymyr

volodymyr orlov

volodymyr ilchyshyn portfolio

from chernobyl to fukushima - €¦ · from chernobyl to...

volodymyr yakubov 1 standardsregs - inogate

volodymyr lisin portfolio

volodymyr hucul

diz 2010 lynnyk volodymyr

university of engineering and economics in rzeszów lviv...