seng 530: software verification and validation v&v processes and techniques prof. bojan cukic...
TRANSCRIPT
![Page 1: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/1.jpg)
SENG 530: SoftwareVerification and Validation
V&V Processes and Techniques
Prof. Bojan Cukic
Lane Department of Computer Science and Electrical Engineering
West Virginia University
![Page 2: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/2.jpg)
2
Overview Software Inspections.
02/14/2002 Software Metrics.
Today Software Reliability Engineering.
02/28/2002
![Page 3: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/3.jpg)
3
Agenda Software Engineering
Measurements. Measurement Theory. A Goal-Based Framework for
Software Measurement. Verification and Validation Metrics.
![Page 4: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/4.jpg)
4
Measure? Why? Developers angle.
Completeness of requirements, quality of design, testing readiness.
Managers angle. Delivery readiness, budget and scheduling
issues. Customers angle.
Compliance with requirements, quality. Maintainers angle.
Planning for upgrades and improvements.
![Page 5: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/5.jpg)
5
Measurement A process by which numbers (symbols)
are assigned to attributes of entities in the real world in such a way to describe them according to clearly defined rules.
Measurement process is difficult to define. Measuring colors, intelligence is difficult. Measurement accuracy, margin of errors. Measurement units, scales. Drawing conclusions from measurements is
difficult.
![Page 6: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/6.jpg)
6
Measurement (2) “What is not measurable make
measurable” [Galileo, 1564-1642]. Increased visibility, understanding, control. Measurement:
Direct quantification of an attribute. Calculation:
Indirect, a combination of measurements used to understand some attribute.
(Ex. Overall scores in decathlon).
![Page 7: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/7.jpg)
7
Measurement in Software Engineering Applicable to managing, costing,
planning, modeling, analyzing, specifying, designing, implementing, verifying, validating, maintaining. Engineering implies understanding and
control. Computer science provides theoretical
foundations for building software, software engineering focuses on controlled and scientifically sound implementation process.
![Page 8: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/8.jpg)
8
Measurement in Software Engineering
Considered somewhat a luxury?!? Weakly defined targets: “Product will be
user-friendly, reliable, maintainable”. Gilb’s Principle of Fuzzy Targets: “Projects
without clear goals will not achieve their goals clearly.”
Estimation of costs. Cost of design, cost of testing, cost of coding…
Predicting product quality. Considering technology impacts.
![Page 9: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/9.jpg)
9
Software Measurement Objectives
“You cannot control what you cannot measure.” [DeMarco, 1982]
Managers angle. Cost: Measure time and effort of various processes
(elicitation, design, coding, test). Staff productivity: Measure staff time, size of
artifacts. Use these for predicting the impact of a change.
Product quality: Record faults, failures, changes as they occur. Cross compare different projects.
User satisfaction: Response time, functionality. Potential for improvement.
![Page 10: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/10.jpg)
10
Software Measurement Objectives (2)
Engineers angle. Are requirements testable?
Instead of requiring “reliable operation”, state the expected mean time to failure.
Have all faults been found? Use models of expected detection rates.
Meeting product or process goals. <20 failures per beta-test site in a month. No module contains more than x lines (standards).
What the future holds? Predict product size from specification size.
![Page 11: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/11.jpg)
11
The scope of software metrics
Cost and effort estimation. COCOMO 1, Function points model, etc.
Effort is a function of size (LOC, function points), developer’s capability, level of reuse, etc.
Productivity models and measures. Simplistic approach: Size/Effort.
Can be quite misleading, even dangerous.
![Page 12: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/12.jpg)
12
A productivity model
![Page 13: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/13.jpg)
13
The Scope of Metrics (2) Data collection
Easier said than done. Must be planed and executed carefully. Use simple graphs and charts to
present collected data (see next slide). Good experiments, surveys, case
studies are essential.
![Page 14: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/14.jpg)
14
Presenting collected data
![Page 15: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/15.jpg)
15
The Scope of Metrics (3) Quality models and measurements.
Quality and productivity models are usually combined.
Advanced COCOMO (COCOMO II), McCalls model.
Usually constructed in a tree-like fashion.
At a high level are indirect factors, at a low level are directly measurable factors.
![Page 16: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/16.jpg)
16
![Page 17: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/17.jpg)
17
The Scope of Metrics (4) Reliability models. Performance evaluation and modeling. Structural and complexity metrics.
Readily available structural properties of code (design) serve as surrogate for quality assessment, control, prediction.
Management by metrics. Many companies define standard tracking
and project monitoring/reporting systems. Capability maturity assessment.
![Page 18: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/18.jpg)
18
Agenda Software Engineering
Measurements. Measurement Theory. A Goal-Based Framework for
Software Measurement. Verification and Validation Metrics.
![Page 19: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/19.jpg)
19
The basics of measurement Some pervasive measurement
techniques are taken for granted A rising column of mercury for temperature
measurement was not so obvious 150 years ago.
But, we developed a measurement framework for temperature.
How well do we understand software attributes we want to measure? What is program “complexity”, for example?
![Page 20: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/20.jpg)
20
The basics of measurement (2) Are we really measuring the attribute we
want to measure? Is the number of “bugs” in system testing a
measure of quality? What statements can be made about an
attribute” Can we “double design quality”?
What operations can be applied to measurements? What is “average productivity” of the group? What is “average quality” of software modules?
![Page 21: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/21.jpg)
21
Empirical relations
![Page 22: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/22.jpg)
22
Empirical relations (2) “Taller than” is an empirical relation.
Binary relation (x is taller than y). Unary relation (x is tall).
Empirical relations need to be mapped from real world into mathematics. In this mapping, real world is the domain,
mathematical world is the range. Range can be the set of integers, real
numbers, or even non-numeric symbols.
![Page 23: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/23.jpg)
23
Representation condition The mapping should preserve real
world relations. A is taller than B iff M(A) > M(B). Binary empirical relation “taller than” is
replaced by the numerical relation >. So, “x is much taller than y” may mean
M(x)>M(y)+15.
![Page 24: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/24.jpg)
24
Stages of formal measurement
![Page 25: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/25.jpg)
25
Agenda Software Engineering
Measurements. Measurement Theory. A Goal-Based Framework for
Software Measurement. Verification and Validation Metrics.
![Page 26: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/26.jpg)
26
The framework Classifying the entities to be
examined Determining relevant
measurement goals Identifying the level of maturity
reached by the organization
![Page 27: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/27.jpg)
27
Classifying software measures Processes are collections of
software related activities. Associated with time, schedule.
Products are artifacts, deliverables or documents that result from a process activity.
Resources are entities required by a process activity.
![Page 28: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/28.jpg)
28
Classifying software measures (2) For each entity, we distinguish:
Internal attributes Those that can be measured purely in terms of
the product, process or the resource itself. Size, complexity measures, dependencies.
External attributes Those that can be measured in terms of how
the product, process or the resource relate to their environment.
Experienced failures, timing and performance.
![Page 29: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/29.jpg)
29
Process Measures include:
The duration of the process or one of its activities.
The effort associated with the process or activities
The number of incidents of the specific type arising during the process or one of its activities.
Ave. cost of error=cost/#errors_found.
![Page 30: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/30.jpg)
30
![Page 31: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/31.jpg)
31
Products External attributes:
Reliability, maintainability, understandability (of documentation), usability, integrity, efficiency, reusability, portability, interoperability…
Internal attributes Size, effort, cost, functionality,
modularity, syntactic correctness.
![Page 32: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/32.jpg)
32
Product measurements Direct measure example:
Entity: Module design document (D1) Attribute: Size Measure: No. of bubbles (in flow diagram).
Indirect measure example: Entity: Module design document (D1, D2,…) Attribute: Average module size Measure: Average no. of bubbles (in flow
diagram).
![Page 33: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/33.jpg)
33
Resources Personnel (individual or team), materials
(including office supplies), tools, methods. Resource measurement may show what
resource to blame for poor quality. Cost measured across all types or
resources. Productivity:
amount_of_output/effort_input. Combines resource measure (input) with the
product measure (output).
![Page 34: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/34.jpg)
34
GQM Paradigm Goal-Question-Metric Steps:
List the major goals of the development effort.
Derive from each goal questions that must be answered to determine if goals are being met.
Decide what must be measured to answer the questions adequately.
![Page 35: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/35.jpg)
35
Generic GQM Example
![Page 36: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/36.jpg)
36
AT&T GQM Example
![Page 37: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/37.jpg)
37
Goal definition templates Purpose:
To (characterize, evaluate, predict…) the (process, model, metric…) in order to (understand, assess, manage, learn, improve…)
Perspective Examine the (cost, correctness, defects,
changes…) from the viewpoint of (developer, manager, customer…)
Environment The environment consists of process factors,
people factors, methods, tools, etc.
![Page 38: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/38.jpg)
38
Process improvement Measurement is useful for
understanding, establishing the baseline, assessing and predicting.
But the larger context is improvement.
SEI proposed five maturity levels, ranging from the least to the most predictable and controllable.
![Page 39: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/39.jpg)
39
Process maturity levels
![Page 40: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/40.jpg)
40
Maturity and measurement overview Level Characteristic Metrics Initial Ad hoc Baseline Repeatable Process depends Project
on individuals management Defined Process defined & Product
institutionalized Managed Measured process Process+
feedback Repeatable Improvement fed back
Process+feedbackto the process for changing theprocess.
![Page 41: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/41.jpg)
41
Repeatable process
![Page 42: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/42.jpg)
42
A defined process
![Page 43: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/43.jpg)
43
A managed process
![Page 44: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/44.jpg)
44
Applying the framework Cost and effort estimation
E=a*Sb
E is effort (person months), S is size (thousands of delivered source statements)
A, b are environment specific constants. Data collection
Orthogonal defect classification
![Page 45: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/45.jpg)
45
Applying the framework (2) Reliability models
JM model: MTTFi=a/(N-I+1) N: total no. faults, 1/a is the “fault
size”. Capability maturity assessment
The maturity attribute is also viewed as an attribute of contractor’s process.
![Page 46: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/46.jpg)
46
Applying the framework (3) Evaluation of methods and tools
![Page 47: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/47.jpg)
47
Agenda Software Engineering
Measurements. Measurement Theory. A Goal-Based Framework for
Software Measurement. Verification and Validation Metrics.
![Page 48: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/48.jpg)
48
V&V application of metrics Applicable throughout the lifecycle. Should be condensed for small projects. Used to assess product, process,
resources. V&V metric characteristics:
Simplicity Objectivity Ease of collection Robustness (insensitive to changes) Validity
![Page 49: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/49.jpg)
49
![Page 50: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/50.jpg)
50
![Page 51: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/51.jpg)
51
![Page 52: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/52.jpg)
52
![Page 53: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/53.jpg)
53
V&V specific metrics (requirements and design)
![Page 54: SENG 530: Software Verification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649cf05503460f949c030f/html5/thumbnails/54.jpg)
54