se300 software engineering practicesmercury.pr.erau.edu/~siewerts/se300/documents/lectures/... ·...
Post on 18-Jul-2021
5 Views
Preview:
TRANSCRIPT
April 7, 2015 Sam Siewert
SE300
Software Engineering Practices
Lecture 13 – Traditional Testing,
Planning and Re-planning, Formal
Methods of V&V and Standards
Minute Paper #7
What is your Status (Individual View) on Assignment #5,
Completion of Analysis and Design?
What is your Status (Individual View) on Readiness to
Test your Design and Prototype, Mock-up, or Build a
PoC for key Algorithms?
Sam Siewert 2
Testing Strategies
Sam Siewert 3
Domain Models – Use Case Details Complete a Design
Provide Code Re-Use Candidates and PoC
Sam Siewert 4
Start Here!
https://www.modelio.org/
Helpful Validation and Verification Features for
Design
• Integrated Models
• Checklists – Completeness
• CPP and Java Code Generation
USE Modelio as your DESIGN TOOL
Testing Levels Acceptance Tests – Requirements, Deployment Diagram – Test for each Requirement
– Gating Tests for Shipment, Release Notes
System Test – Analysis Use Cases, Domain Model, OIM Sequence Message Passing, CFD/DFD, Block Diagrams – End-to-End (Fully Integrated)
– Performance
– Major Features
– Fault-Injection
Integration and Test - OM, OIM Methods and Collaboration, Activity Diagrams, Package Diagrams, Subsystems – Components, Subsystems
– Interfaces, Protocols
– Verification of Public Interfaces (Methods and Attributes)
Unit Tests - State Machines, Method Details – Functional, Basic Block, Test Cases (Vectors)
– Test Drivers to Instantiate and Exercise Methods (Functions)
Sam Siewert 5
Copyright {c} 2014 by the McGraw-Hill Companies, Inc. All rights Reserved.
20-6
What Is Software Testing?
• Software testing is a dynamic validation
activity to detect defects and undesirable
behavior in the software, and demonstrate that
the software satisfies its requirements and
constraints.
Copyright {c} 2014 by the McGraw-Hill Companies, Inc. All rights Reserved.
20-7
Objectives of Software Testing
• Testing is a process of executing a program
with the intent of finding an error.
• A good test case is one that has a high
probability of finding an undetected error.
• A successful test is one that uncovers an
undiscovered error.
Copyright {c} 2014 by the McGraw-Hill Companies, Inc. All rights Reserved.
20-8
What is Testing and What is Not?
• Testing is a process for finding semantic or
logical errors.
• Testing is not aimed to find syntax errors.
• Testing can reveal the presence of errors, NOT
their absence.
• Class exercise: what is the difference between
testing and debugging?
Copyright {c} 2014 by the McGraw-Hill Companies, Inc. All rights Reserved.
20-9
Why Software Testing?
• Review and inspection are not adequate because they
do not execute the program. Can you test a car
without driving it?
• Why need testing:
– Instrumentation systems & appliances
• coded in assembly/machine language ===> costly to repair
– Process control systems
• failure is politically unacceptable & economically undesirable.
– Military systems
• involve risk of human life
– Safety - Critical Health Care Systems.
Copyright {c} 2014 by the McGraw-Hill Companies, Inc. All rights Reserved.
20-10
Software Testing in Practice
• Testing amounts to 40% -- 80% of total development
costs
– ~ 40% for information systems
– more than 80% for real-time embedded systems
• On one occasion the average costs of a field detected
bug was US$15,000 (1991).
• Testing receives the least attention, and often not
enough time and resources.
• Testers are often forced to abandon test efforts
because changes have been made.
Copyright {c} 2014 by the McGraw-Hill Companies, Inc. All rights Reserved.
20-11
Software Testing in Practice
• Most companies’ new hires are testers.
• Most testing work is manually testing the
software system; help from tools is limited.
• In many cases, testing is not performed by
using systematic testing methods or
techniques. It is not effective.
• There are “conflict of interest” between testers
and developers, and testers and management.
Copyright {c} 2014 by the McGraw-Hill Companies, Inc. All rights Reserved.
20-12
Benefit of Software Testing
• Benefits: – Conscious of need for error-free software.
– Ensure the more flagrant kinds of errors will be detected.
– Improving software quality.
– Adding a layer of quality assurance to inspection, walkthrough, and peer review.
– Helping in data collection for root cause analysis and process improvement.
– Providing a framework for applying SQA techniques.
• Systematic software testing is more effective than
programmers’ casual testing.
Copyright {c} 2014 by the McGraw-Hill Companies, Inc. All rights Reserved.
20-13
Test Categories
• Unit Testing: testing at the individual module level, functions, modules, classes, etc.
• Integration Testing: testing the interfaces among the modules or components.
• Validation Testing: test according to customer’s functional requirements.
• System Testing
– Recovery Testing
– Security Testing
– Stress Testing
– Performance Testing
Copyright {c} 2014 by the McGraw-Hill Companies, Inc. All rights Reserved.
20-14
Test Driven Development
Y
Write/edit tests for
features to be
implemented next
Run new tests
All new
tests fail?
Write/edit
production
code
Run all tests All tests
pass?
Refactor code
More features
to implement?
Halt N
N
N
Y
Y
Test coverage
accomplished? N
Add tests to
increase coverage
Y
Prepare for
TDD
Copyright {c} 2014 by the McGraw-Hill Companies, Inc. All rights Reserved.
20-15
Software Testing Budget
• Size of testing budget:
• expressed in terms of x% of total SW
development costs.
• in terms of system size: $x per statement.
• compute in terms of productivity of the test
team in terms of time: planned & spent.
SQA – SE420
Test Methods at Each Level for Concurrent Process
– Acceptance Tests – Requirements, Deployment Diagram
– System Test – Analysis Use Cases, Domain Model, OIM
Sequence Message Passing, CFD/DFD, Block Diagrams
– Integration and Test - OM, OIM Methods and Collaboration,
Activity Diagrams, Package Diagrams, Subsystems
– Unit Tests - State Machines, Method Details
Details of Composing Test Drivers, Cases, and
Automation Verification and Validation
Regression Testing
Methods of Black-box and White-box Test Development
Sam Siewert 16
Testing Levels
Acceptance Tests – Requirements, Deployment Diagram
System Test – Analysis Use Cases, Domain Model, OIM Sequence Message Passing, CFD/DFD, Block Diagrams – End-to-End (Fully Integrated)
– Performance
– Major Features
– Fault-Injection
Integration and Test - OM, OIM Methods and Collaboration, Activity Diagrams, Package Diagrams, Subsystems
Unit Tests - State Machines, Method Details
Sam Siewert 17
Take Away Test Definition and Development Should Start With Requirements
Testing Concurrent with Analysis, Design, Implementation
Define Test Cases, Methods, and Use Tools at All Levels of Analysis, Design, Implementation
Define Exit Criteria, So Test Completion is not Arbitrary – E.g. Code Path Coverage
Gating Tests Prevent Shipment of Defective Software
Some Defects and Deficiencies May Be Documented as Limitations in Release Notes
Can’t Prove Code is Bug Free - SWE Formal Methods – Advanced Concepts to Prove Correctness [Research Topic]
– Pre-condition, Post-condition (Axiomatic Proofs)
– Prove Ability to Meet Deadlines (Rate Monotonic)
– Termination and Halting Problem and Undecidable Problems
Sam Siewert 18
Project Planning and Cost Estimation
Assume Cost of Testing and Support
Ramp-Up Time [E.g. Hiring, Training, Tool Acquisition
and Configuration]
Re-Use is not Zero Cost, Even for Open Source
Need to At Least Have Scope Defined through
Requirements and Analysis
Refine and Update over Time
Sam Siewert 19
20
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Project Planning Task Set-I
Establish project scope
Determine feasibility
Analyze risks
Risk analysis is considered in detail in Chapter 25.
Define required resources Determine require human resources
Define reusable software resources
Identify environmental resources
21
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Project Planning Task Set-II
Estimate cost and effort Decompose the problem
Develop two or more estimates using size, function
points, process tasks or use-cases
Reconcile the estimates
Develop a project schedule
Scheduling is considered in detail in Chapter 34.
• Establish a meaningful task set
• Define a task network
• Use scheduling tools to develop a timeline chart
• Define schedule tracking mechanisms
22
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Estimation
Estimation of resources, cost, and schedule for
a software engineering effort requires experience
access to good historical information (metrics)
the courage to commit to quantitative predictions
when qualitative information is all that exists
Estimation carries inherent risk and this risk
leads to uncertainty
23
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Write it Down!
Software
Project
Plan
Project Scope
Estimates
Risks
Schedule
Control strategy
24
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
To Understand Scope ...
Understand the customers needs
understand the business context
understand the project boundaries
understand the customer’s motivation
understand the likely paths for change
understand that ...
Even when you understand,
nothing is guaranteed!
25
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
What is Scope?
Software scope describes the functions and features that are to be delivered to
end-users
the data that are input and output
the “content” that is presented to users as a
consequence of using the software
the performance, constraints, interfaces, and
reliability that bound the system.
Scope is defined using one of two techniques: • A narrative description of software scope is developed
after communication with all stakeholders.
• A set of use-cases is developed by end-users.
26
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Resources
project
people
skills
number
location
reusable
softwareOTS
components
full-experience
components
new
components
part.-experience
components
environment
hardware
software
tools
network
resources
27
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Project Estimation
Project scope must be
understood
Elaboration (decomposition) is
necessary
Historical metrics are very helpful
At least two different techniques
should be used
Uncertainty is inherent in the
process
28
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Estimation Techniques
Past (similar) project experience
Conventional estimation techniques
task breakdown and effort estimates
size (e.g., FP) estimates
Empirical models
Automated tools
29
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Estimation Accuracy
Predicated on …
the degree to which the planner has properly
estimated the size of the product to be built
the ability to translate the size estimate into human
effort, calendar time, and dollars (a function of the
availability of reliable software metrics from past
projects)
the degree to which the project plan reflects the
abilities of the software team
the stability of product requirements and the
environment that supports the software engineering
effort.
30
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Functional Decomposition
functional
decomposition
Statement of
Scope Perform a
Grammatical “parse”
31
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Example: LOC Approach
Average productivity for systems of this type = 620 LOC/pm.
Burdened labor rate =$8000 per month, the cost per line of code is approximately $13.
Based on the LOC estimate and the historical productivity data, the total estimated project cost is $431,000 and the estimated effort is 54 person-months.
Works Best for Product A’ Based
On A – Next Generation
32
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Example: FP Approach
The estimated number of FP is derived:
FPestimated = count-total 3 [0.65 + 0.01 3 S (Fi)]
FPestimated = 375
organizational average productivity = 6.5 FP/pm.
burdened labor rate = $8000 per month, approximately $1230/FP.
Based on the FP estimate and the historical productivity data, total estimated
project cost is $461,000 and estimated effort is 58 person-months.
Feature Point Good Approach for
Green Field or Upgrade Projects
33
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Process-Based Estimation Example
Activity
Task
Function
UICF
2DGA
3DGA
DSM
PCF
CGDF
DAM
Totals
% effort
CC PlanningRisk
Analysis EngineeringConstruction
Release TotalsCE
analysis design code test
0.25 0.25 0.25 3.50 20.50 4.50 16.50 46.00
1% 1% 1% 8% 45% 10% 36%
CC = customer communication CE = customer evaluation
0.50
0.750.50
0.50
0.500.25
2.50
4.004.00
3.00
3.002.00
0.40
0.601.00
1.00
0.750.50
5.002.00
3.00
1.50
1.501.50
8.40
7.358.50
6.00
5.754.25
0.50 2.00 0.50 2.00 5.00
n/an/a
n/a
n/a
n/an/a
n/a
Based on an average burdened labor rate of $8,000 per month, the total estimated project cost is $368,000 and the estimated effort is 46 person-months.
Estimate Time in Each Cycle for
Evolutionary Process for Each Major
Function
Mission Critical Systems Systems where FAILURE and MAJOR Defects Cause Loss of Life, Private Data, Property or Significant Financial Impact – Avionics
– Enterprise Systems for Business, Banking, Finance
– Medical
– Industrial Automation and Infrastructure
– Personal Records, Business IP, National Security
Design and Implementation Assurance
Ideally Provably Correct, but Low to Zero Defect Assurance (Standards) Acceptable Due to Cost of Formal Proofs
Sam Siewert 34
Standards and Formal Methods Standards and Guidelines for Systems and Software Testing – Mission Critical
Enterprise Data Systems – Commercial Standards and IV&V Labs (E.g. Medusa Labs Test Tools), Standards for Protocol (SCSI T10 and SAS, Fiber Channel T11, FCoE, iSCSI) and Performance Test Standards (TPC, SPC), SNIA RAID
Aerospace – NASA Standards (https://standards.nasa.gov/documents/nasa ), Military Standards (MIL-STD-498), FAA Launch and Reentry, FAA
Medical Systems – IEC 62304, E.g. Commercial Support
Energy – NIST Testing, DoE STD-1172
Security – TCG, NIST ITL, NIST FISMA, NIST FIPS, NIST TIRs – Overview
Transportation – FAA DO-178B, Software Assurance, V&V, NHTSA Policies on Automated Vehicles
Sam Siewert 35
36
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Chapter 28
Formal Modeling and Verification
Slide Set to accompany
Software Engineering: A Practitioner’s Approach, 8/e by Roger S. Pressman and Bruce R. Maxim
Slides copyright © 1996, 2001, 2005, 2009, 2014 by Roger S. Pressman
For non-profit educational use only May be reproduced ONLY for student use at the university level when used in conjunction
with Software Engineering: A Practitioner's Approach, 8/e. Any other reproduction or use is
prohibited without the express written permission of the author.
All copyright information MUST appear if these slides are posted on a website for student
use.
37
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Formal Modeling and Verification
Cleanroom software engineering and formal
methods
Both demand a specialized specification approach and
each applies a unique verification method.
Both are quite rigorous and neither is used widely by the
software engineering community.
If you must build “bullet-proof” software, these
methods can help immeasurably.
38
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
The Cleanroom Process Model
39
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
The Cleanroom Strategy-I Increment Planning—adopts the incremental strategy
Requirements Gathering—defines a description of customer level requirements (for each increment)
Box Structure Specification—describes the functional specification
Formal Design—specifications (called “black boxes”) are iteratively refined (with an increment) to become analogous to architectural and procedural designs (called “state boxes” and “clear boxes,” respectively).
Correctness Verification—verification begins with the highest level box structure (specification) and moves toward design detail and code using a set of “correctness questions.” If these do not demonstrate that the specification is correct, more formal (mathematical) methods for verification are used.
Code Generation, Inspection and Verification—the box structure specifications, represented in a specialized language, are transmitted into the appropriate programming language.
40
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
The Cleanroom Strategy-II
Statistical Test Planning—a suite of test cases that exercise of “probability distribution” of usage are planned and designed
Statistical Usage Testing—execute a series of tests derived from a statistical sample (the probability distribution noted above) of all possible program executions by all users from a targeted population
Certification—once verification, inspection and usage testing have been completed (and all errors are corrected) the increment is certified as ready for integration.
41
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Box Structure Specification
black box
state box
clear box
42
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Box Structures
black box state box
clear box
43
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Design Refinement & Verification
If a function f is expanded into a sequence g and h, the correctness condition for all input to f is:
• Does g followed by h do f?
When a function f is refined into a conditional (if-then-else), the correctness condition for all input to f is:
• Whenever condition <c> is true does g do f and whenever <c> is false, does h do f?
When function f is refined as a loop, the correctness conditions for all input to f is:
• Is termination guaranteed?
• Whenever <c> is true does g followed by f do f, and whenever <c> is false, does skipping the loop still do f?
44
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Advantages of Design Verification
It reduces verification to a finite process.
It lets cleanroom teams verify every line of
design and code.
It results in a near zero defect level.
It scales up.
It produces better code than unit testing.
45
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Cleanroom Testing
statistical use testing
tests the actual usage of the program
determine a “usage probability distribution”
analyze the specification to identify a set of stimuli
stimuli cause software to change behavior
create usage scenarios
assign probability of use to each stimuli
test cases are generated for each stimuli according
to the usage probability distribution
46
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Certification
1. Usage scenarios must be created.
2. A usage profile is specified.
3. Test cases are generated from the profile.
4. Tests are executed and failure data are recorded and analyzed.
5. Reliability is computed and certified.
47
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Certification Models
Sampling model. Software testing executes m random test cases and is certified if no failures or a specified numbers of failures occur. The value of m is derived mathematically to ensure that required reliability is achieved.
Component model. A system composed of n components is to be certified. The component model enables the analyst to determine the probability that component i will fail prior to completion.
Certification model. The overall reliability of the system is projected and certified.
48
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Formal Methods
“Formal methods used in developing computer systems are mathematically based techniques for describing system properties. Such formal methods provide frameworks within which people can specify, develop, and verify systems in a systematic, rather than ad hoc manner.”
The Encyclopedia of Software Engineering [Mar01]
The Problem with conventional specs:
contradictions
ambiguities
vagueness
incompleteness
mixed levels of abstraction
49
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Formal Specification
Desired properties—consistency, completeness, and lack of
ambiguity—are the objectives of all specification methods
The formal syntax of a specification language enables
requirements or design to be interpreted in only one way,
eliminating ambiguity that often occurs when a natural
language (e.g., English) or a graphical notation must be
interpreted
The descriptive facilities of set theory and logic notation enable
clear statement of facts (requirements).
Consistency is ensured by mathematically proving that initial
facts can be formally mapped (using inference rules) into later
statements within the specification.
50
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Formal Methods Concepts data invariant—a condition that is true throughout the
execution of the system that contains a collection of data
state
Many formal languages, such as OCL (Section 28.5) , use the notion of states as they were discussed in Chapters 7 and 8, that is, a system can be in one of several states, each representing an externally observable mode of behavior.
The Z language (Section 28.6)defines a state as the stored data which a system accesses and alters
operation—an action that takes place in a system and reads or writes data to a state
precondition defines the circumstances in which a particular operation is valid
postcondition defines what happens when an operation has completed its action
51
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
States and Data Invariant The state of the spooler is represented by the four components Queues, OutputDevices, Limits, and Sizes.
The data invariant has five components:
• Each output device is associated with an upper limit of print lines
• Each output device is associated with a possibly nonempty queue of files awaiting printing
• Each file is associated with a size
• Each queue associated with an output device contains files that have a size less than the upper limit of the output device
• There will be no more than MaxDevs output devices administered by the spooler
52
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Operations
An operation which adds a new output device to the
spooler together with its associated print limit
An operation which removes a file from the queue
associated with a particular output device
An operation which adds a file to the queue associated
with a particular output device
An operation which alters the upper limit of print lines for
a particular output device
An operation which moves a file from a queue associated
with an output device to another queue associated with a
second output device
53
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.
Pre- & Postconditions
For the first operation (adds a new output device to the spooler together with its associated print limit):
Precondition: the output device name does not already exist and that there are currently less than MaxDevs output devices known to the spooler
Postcondition: the name of the new device is added to the collection of existing device names, a new entry is formed for the device with no files being associated with its queue, and the device is associated with its print limit.
Minute Paper #8
Can you Test a Design? Beyond Inspections (Walk-
throughs and Formal Inspection)
If not, Must Test Wait for Code Complete to Start?
Sam Siewert 54
top related