1.basic metrics and measurement
TRANSCRIPT
-
8/6/2019 1.Basic Metrics and Measurement
1/19
SQE Vol. 1 R.L. Probert, 2001
MODULE 1: Quality,
Process and Management
SOFTWARE QUALITY
MANAGEMENTManagement Objectives
Measurements & Metrics
(Ch. 3, 4. Of Kan)
Process Metrics
Product Metrics
Cost of Quality
INDUSTRIAL SCALE TESTING
Best Principles
Best PracticesBest Automation Strategies
-
8/6/2019 1.Basic Metrics and Measurement
2/19
SQE Vol. 1 R.L. Probert, 2001
Management Objectives
Optimize product quality Development cost
Time to market
Personnel development
-
8/6/2019 1.Basic Metrics and Measurement
3/19
SQE Vol. 1 R.L. Probert, 2001
An Example of the Waterfall Process Model
Requirements
Gathering
and Analysis
Architectural
Design
HLD/IO LLD/I1 CODE/12 UT
Integration
Subsystem Test
RAISE System Test
Early Customer
Feedback and Beta
Test Programs
Release
HLD: High-level design
IO: HLD Inspection
LLD: Low-level design
I1: LLD InspectionI2: Code Inspection
UT: Unit Test
Raise: Reliability, Availability, Install,
Serviceability, and Ease of use
-
8/6/2019 1.Basic Metrics and Measurement
4/19
SQE Vol. 1 R.L. Probert, 2001
Development Process
Requirements Gathering &
Validation Design
Design reviews and inspections
Code
Code inspection
Debug and development test
Integration (of components and
modules to form the product) Formal machine testing
Early customer programs
-
8/6/2019 1.Basic Metrics and Measurement
5/19
SQE Vol. 1 R.L. Probert, 2001
Possible Testable
Hypotheses For software projects, the higher
the percentage of the designs andcode that are inspected, the lower
the defect rate that will beencountered at the later phase offormal machine testing.
The more effective the designreviews and the code inspections as
scored by the inspection team, thelower the defect rate that will beencountered at the later phase offormal machine testing.
The more thorough the developmenttest (in terms of test coverage)done before integration, the lowerthe defect rate that will beencountered at the formal machinetesting phase.
-
8/6/2019 1.Basic Metrics and Measurement
6/19
SQE Vol. 1 R.L. Probert, 2001
Four Levels of Measurement
Nominal scale Ordinal scale
Interval scale
Ratio scale
Note that the measurementscales are hierarchical
-
8/6/2019 1.Basic Metrics and Measurement
7/19
SQE Vol. 1 R.L. Probert, 2001
Basic Measures
Ratio Proportion
PercentageRequirements bugs were 15% of the total,
design bugs were 25% of the total, codingbugs were 50% of the total, and other bugs
made up 10% of the total.
Vs.
The project consists of 8 thousand lines ofcode (KLOC). During its development a total
of 200 defects were detected and removed,giving a defect removal rate of 25 defectsper KLOC. Of the 200 defects,
requirements bugs constituted 15%, designbugs 25%, coding bugs 50%, and other bugs
made up 10%.
-
8/6/2019 1.Basic Metrics and Measurement
8/19
SQE Vol. 1 R.L. Probert, 2001
Percentage Distributions ofDefect Type by Project
Type of Defect Project A Project B Project C(%) (%) (%)
Requirements 15.0 41.0 20.3
Design 25.0 21.8 22.7
Code 50.0 28.6 36.7
Others 10.0 8.6 20.3
Total 100.0 100.0 100.0(N) (200) (105) (128)
Of the total 20 defects for the entire project of 2 KLOC,
there were 3 requirements bugs, 5 design bugs, 10coding bugs, and 2 others.
-
8/6/2019 1.Basic Metrics and Measurement
9/19
SQE Vol. 1 R.L. Probert, 2001
Percentage Distributions ofdefects Across project by Defect
Type
Project
Type of Defect A B C Total (N)
Requirements (%) 30.3 43.4 26.3 100.0 (99)
Design (%) 49.0 22.5 28.5 100.0 (102)
Code (%) 56.5 16.9 26.6 100.0 (177)
Others (%) 36.4 16.4 47.2 100.0 (55)
-
8/6/2019 1.Basic Metrics and Measurement
10/19
SQE Vol. 1 R.L. Probert, 2001
-
8/6/2019 1.Basic Metrics and Measurement
11/19
SQE Vol. 1 R.L. Probert, 2001
Reliability and Validity
Reliability refers to the consistency of anumber of measurements taken using thesame measurement method on the samesubject Reliability can be expressed in terms of the
size of the standard deviations of the
repeated measurements. Index of variation (IV) is used
IV = Standard deviationMean
Validity refers to whether themeasurement or metric really measureswhat we intend it to measure.
Construct validity:
Validity of the operational measurement ormetric
Criterion-related validity: Is predictive validity Content validity refers to the degree to which
a measure covers the range of meanings
-
8/6/2019 1.Basic Metrics and Measurement
12/19
SQE Vol. 1 R.L. Probert, 2001
-
8/6/2019 1.Basic Metrics and Measurement
13/19
SQE Vol. 1 R.L. Probert, 2001
Measurement Errors
Systematic Random
M = T + s + eM is the observed/measured score, T is the true
score, s is systematic error, e is random error
The correlation between the true score and theerror term is zero.
Three is no serial correlation between the truescore and the error term.
The correlation between errors on distinctmeasurements is zero.
Expected value of observed scores is equal to thetrue score:
E(M) = E(T) + E(e)= E(T) + 0= E(T)= T
-
8/6/2019 1.Basic Metrics and Measurement
14/19
SQE Vol. 1 R.L. Probert, 2001
Assess the Impact of e onthe Reliability of the
Measurements
M = T + e
Var(M) = var(T) + var(e) (var represents variance. Thisrelationship is due to the
assumption on error terms)
Reliability = Vm = var(T)/var(M)
= [var(M)-var(e)]/var(M)= 1[var(e)/var(M)].
Reliability of a metric varies between 0 and 1.
If all variance of the observed scores is due torandomerrors, then the reliability is zero
-
8/6/2019 1.Basic Metrics and Measurement
15/19
SQE Vol. 1 R.L. Probert, 2001
Test/Retest Method for
Estimating Reliability
M1M2
e1 e2Test
Retest
T
-
8/6/2019 1.Basic Metrics and Measurement
16/19
SQE Vol. 1 R.L. Probert, 2001
-
8/6/2019 1.Basic Metrics and Measurement
17/19
SQE Vol. 1 R.L. Probert, 2001
y = mx + b (straight line)
-
8/6/2019 1.Basic Metrics and Measurement
18/19
SQE Vol. 1 R.L. Probert, 2001
Criteria for Causality
1. The first requirement in a causalrelationship between two variablesis that the cause precedes theeffect in time or as shown clearlyin logic.
2. The second requirement in a causal
relationship is that the twovariables be empirically correlatedwith one another.
3. The third requirement for a causal
relationship is that the observedempirical correlation between twovariables is not because of aspurious relationship.
-
8/6/2019 1.Basic Metrics and Measurement
19/19
SQE Vol. 1 R.L. Probert, 2001
Spurious Relationships
Z
X
Y
X Z Y
C
X Y
ex ey