cste skill category 1 2014v03 - · pdf fileessay (10 minutes): cste -skill category 1 -12...
TRANSCRIPT
CSTE - Skill Category 1 - 1 © Copyright 2014 / All rights reserved
Skill Category 1Software Testing Principles and Concepts
The following topics will be discussed in this Skill Category:
• Vocabulary & Quality Basics
• Understanding Defects
• Process and Testing Published Standards
• Software Testing
• Software Development Life Cycle (SDLC) Models
• Agile Development Methodologies
• Testing Throughout the SDLC
• Testing Schools of Thought and Approaches
• Test Categories and Testing Techniques
CSTE - Skill Category 1 - 2 © Copyright 2014 / All rights reserved
Testing VocabularyUnderstand the technical terms used to describe various testing techniques, tools, principles, concepts and activities
contingency
CSTE - Skill Category 1 - 3 © Copyright 2014 / All rights reserved
Quality 101
CSTE - Skill Category 1 - 4 © Copyright 2014 / All rights reserved
Quality Assurance• Activities that modify the development process to prevent the
introduction of flaws
– Staff function
– Implements management’s quality policies
– Responsible for continuous improvement of the software development process
• Proactive approach focused on defect prevention
• Examples:
– Defining change control procedures
– Analyzing metrics to identify opportunities for process improvement
– Facilitating quality improvement activities
CSTE - Skill Category 1 - 5 © Copyright 2014 / All rights reserved
Quality Control• Activities within the development process to detect the introduction of
flaws
– Test planning and execution
– Quality control measures a product against the existence of an attribute
– Determines whether the product conforms to a standard or procedure (also known as compliance checking).
• Proactive approach focused on defect detection
• Examples:
– Writing and executing test cases and scripts
– Participating in verification and validation activities
– Reporting defects to identify opportunities for process improvement
– Conducting post-project reviews
CSTE - Skill Category 1 - 6 © Copyright 2014 / All rights reserved
Exercise
Inspection of source code.Unit testing to validate that the program works.Analysis of defects to determine the stage of origin.Analyze metrics collected that measure the effectiveness of system and unit Testing
QA
QC
Quality Assurance or Quality Control?
CSTE - Skill Category 1 - 7 © Copyright 2014 / All rights reserved
ExerciseThe responsibilities of this job include facilitation, process
configuration, measurement, and risk analysis.a) Quality Controlb) Quality Assurance
The responsibilities of this job include conducting inspections, reviews, testing and focusing on the product.
a) Quality Controlb) Quality Assurance
CSTE - Skill Category 1 - 10 © Copyright 2014 / All rights reserved
Your department has always been titled the SQA (Software Quality Assurance) Group. The primary activity of your group is software testing with some minor QA responsibility. Your company is currently reorganizing and you have decided it is a good time to clearly delineate the QC and QA functions. Describe the activities of these two unique groups.
Essay (10 minutes):
CSTE - Skill Category 1 - 12 © Copyright 2014 / All rights reserved
Questions?
CSTE - Skill Category 1 - 13 © Copyright 2014 / All rights reserved
The Perspectives of QualityThere are five perspectives of Quality.
• Transcendent – I know it when I see it
• Product Based – Possesses desired features
• User Based – Fitness for use
• Development and Manufacturing Based – Conforms to requirements
• Value Based – At an acceptable cost
CSTE - Skill Category 1 - 14 © Copyright 2014 / All rights reserved
How Quality is DefinedProducer’s View of Quality Customer’s View of Quality
QUALITY IN FACT QUALITY IN PERCEPTION
Doing the right thing. Delivering the right product.
Doing it the right way. Satisfying our customer’s needs.
Doing it right the first time. Meeting the customer’s expectations.
Doing it on time. Treating every customer with integrity, courtesy, and respect.
Quality in Perception is more important to the final customer.
CSTE - Skill Category 1 - 15 © Copyright 2014 / All rights reserved
The Two Quality Gaps
Two Working Definitions:
• Producer’s ViewMeets requirements
• Consumer’s View Fit for use Start
ProducerGap
CustomerGap
Quality as delivered
An organization’s quality policy must define and view quality from their customer's perspectives. If there are conflicts, they must be resolved.
CSTE - Skill Category 1 - 16 © Copyright 2014 / All rights reserved
Quality Message• Focus on the Customer
• Customer Satisfaction is the essence of a Quality Product
• Customer may be internal or external
• Internal customer receives anything passed between groups
• External customer uses the products or services provided by the organization
CSTE - Skill Category 1 - 17 © Copyright 2014 / All rights reserved
Improving Software QualityThere are many ways to improve Software Quality
• Software Quality Objectives
• Software Quality Assurance Activities
• Testing Strategy
• Software Engineering Guidelines
• Formal and Informal Technical Reviews
• Audits
• Process Improvement
• Change Control Processes
• Measurement, Prototyping, and Proof
CSTE - Skill Category 1 - 18 © Copyright 2014 / All rights reserved
The Cost of Quality• Money spent beyond the cost to build the product
right the first time
• Frequently referred to today as Cost of Non Quality or Cost of Poor Performance
CSTE - Skill Category 1 - 19 © Copyright 2014 / All rights reserved
Cost of Quality• Preventive Costs - Costs associated with preventing errors
– Training– Establishing methods and procedures– Tool acquisition
• Appraisal Costs - Costs associated with the detection of errors– Inspections– Testing
• Failure Costs - Costs associated with defective products delivered to customers
– Analyze, correct and retest defects– Staffing Help Desk– Damage caused by defect– Damage caused by defect– Idle users
CSTE - Skill Category 1 - 20 © Copyright 2014 / All rights reserved
Mark “P” for Prevention; “A” for Appraisal; “F” for Failure:
1) Conducting Reviews and Inspections2) Implementing Standards3) Conducting Rework4) Attending Training/Education5) Planning for Quality6) Code Inspection7) Cancelled Projects8) Failure & Recovery of Data9) Defect Reporting Standards10) Quality Improvement11) Testing12) Conducting a Post Project Review13) Lost Benefits
Cost of Quality
AP
FP
PA
FFP
PA
AF
CSTE - Skill Category 1 - 21 © Copyright 2014 / All rights reserved
ExerciseWhich of the following is NOT a category of the Cost of Quality?
a) Failure Costb) Appraisal Costc) Build Costd) Preventive Cost
In defining the cost of quality, appraisal costs are BEST described as:a) Costs incurred to review completed products against
requirementsb) Costs which can not be recoupedc) All costs associated with defective productsd) None of the above
CSTE - Skill Category 1 - 24 © Copyright 2014 / All rights reserved
Cost of Quality – Iceberg Diagram
• Cost of Quality will vary from Organisation to Organisation
CSTE - Skill Category 1 - 25 © Copyright 2014 / All rights reserved
Question: Is Quality Free?
CSTE - Skill Category 1 - 26 © Copyright 2014 / All rights reserved
Cost of Quality - ROI
CSTE - Skill Category 1 - 27 © Copyright 2014 / All rights reserved
Essay (5 minutes):
The “cost of quality” (COQ) isn’t the price of creating a quality product or service. It’s the cost of NOT creating a quality product or service. Explain what methods you would use to reduce the Cost of Quality as a function of optimizing the development and testing process.
CSTE - Skill Category 1 - 29 © Copyright 2014 / All rights reserved
Questions?
CSTE - Skill Category 1 - 30 © Copyright 2014 / All rights reserved
Software Quality Factors
Correctness ReliabilityEfficiency Integrity
Usability
MaintainabilityFlexibilityTestability
PortabilityReusability
Interoperability
CSTE - Skill Category 1 - 31 © Copyright 2014 / All rights reserved
Software Quality Factors
Correctness ReliabilityEfficiency Integrity
Usability
MaintainabilityFlexibilityTestability
PortabilityReusability
InteroperabilityA quality factor represents the behavioral characteristic of a system.
Examples: correctness, reliability, efficiency, testability, portability, …
CSTE - Skill Category 1 - 32 © Copyright 2014 / All rights reserved
Software Quality Factors
Correctness ReliabilityEfficiency Integrity
Usability
MaintainabilityFlexibilityTestability
PortabilityReusability
Interoperability
Quality Categories Quality Factors Broad Objectives
Product Operation
CorrectnessReliabilityEfficiencyIntegrityUsability
Does it do what the customer wants?Does it do it accurately all of the time?Does it quickly solve the intended problem?Is it secure?Can I run it?
Product Revision
MaintainabilityTestabilityFlexibility
Can it be fixed?Can it be tested?Can it be changed?
ProductTransition
PortabilityReusabilityInteroperability
Can it be used on another machine?Can parts of it be reused?Can it interface with another system?
CSTE - Skill Category 1 - 33 © Copyright 2014 / All rights reserved
Software Quality Factors
• Attributes of software
• Needed for trade off decisions
• Should be included in software requirements
• Should be included in the test plan
• Lack of the needed factors cause customer dissatisfaction
CSTE - Skill Category 1 - 34 © Copyright 2014 / All rights reserved
Software Quality FactorsFactors DefinitionCorrectness Extent to which a program satisfies its specifications and fulfills the user’s mission
objective.Reliability Extent to which a program can be expected to performs its intended function with
required precision.Efficiency The amount of computing resources and code required by a program to perform a
function.Integrity Extent to which access to software or data by unauthorized persons can be
controlled.Usability Effort required learning, operating, preparing input, and interpreting output of a
program.Maintainability Effort required locating and fixing an error in an operational program.
Testability Effort required testing a program to ensure that it performs its intended function.
Flexibility Effort required to modify an operational program.
Portability Effort required to transfer software from one configuration to another.
Reusability Extent to which a program can be used in other applications - related to the packaging and scope of the functions that programs perform.
Interoperability Effort required to couple on system with another.
CSTE - Skill Category 1 - 35 © Copyright 2014 / All rights reserved
Software Quality Criteria1. Access audit: Ease with which software and data can be checked for
compliance with standards or other requirements.2. Access control: Provisions for control and protection of the software and data.3. Accuracy: Precision of computations and output.4. Communication commonality: Degree to which standard protocols and
interfaces are used.5. Completeness: Degree to which a full implementation of the required
functionalities has been achieved.6. Communicativeness: Ease with which inputs and outputs can be assimilated7. Conciseness: Compactness of the source code, in terms of lines of code. 8. Consistency: Use of uniform design and implementation techniques and
notation throughout a project.9. Data commonality: Use of standard data representations.10.Error tolerance: Degree to which continuity of operation is ensured under
adverse conditions. 11.Execution efficiency: Run time efficiency of the software.12.Expandability: Degree to which storage requirements or software functions
can be expanded.
CSTE - Skill Category 1 - 36 © Copyright 2014 / All rights reserved
Software Quality Criteria13.Generality: Breadth of the potential application of software components.14.Hardware independence: Degree to which the software is dependent on the
underlying hardware.15. Instrumentation: Degree to which the software provides for measurement of
its use or identification of errors.16.Modularity: Provision of highly independent modules.17.Operability: Ease of operation of the software.18.Self-documentation: Provision of in-line documentation that explains
implementation of components.19.Simplicity: Ease with which the software can be understood.20.Software system independence: Degree to which the software is independent
of its software environment—nonstandard language constructs, operating system, libraries, database management system, etc.
21.Software efficiency: Run time storage requirements of the software.22.Traceability: Ability to link software components to requirements.23.Training: Ease with which new users can use the system.
CSTE - Skill Category 1 - 37 © Copyright 2014 / All rights reserved
Software Quality Criteria
A quality criterion is an attribute of a quality factor that is related to software development.
Example: •Modularity is an attribute of the architecture of a software system.•A highly modular software allows designers to put cohesive components in one module, thereby increasing the maintainability of the system.…
CSTE - Skill Category 1 - 38 © Copyright 2014 / All rights reserved
Map Software Quality Factors & Quality Criteria
CSTE - Skill Category 1 - 39 © Copyright 2014 / All rights reserved
Map Software Quality Factors & Quality Criteria
Each quality factor is positively influenced by a set of quality criteria, and the same quality criterion impacts a number of quality factors.
Example: Simplicity impacts reliability, usability, and testability.
If an effort is made to improve one quality factor, another quality factor may be degraded.
Portable code may be less efficient.
Some quality factors positively impact others. An effort to improve the correctness of a system will increase its reliability.
CSTE - Skill Category 1 - 40 © Copyright 2014 / All rights reserved
ISO 25010:2011ISO 25010:2011 Standard defines two models relating to software Product Quality
• Product Model applicable to both Computer Systems and Software Products with 8 characteristics and multiple sub-characteristics
• Quality in Use model applicable to the complete human-computer system with 5 characteristics and multiple sub-characteristics
CSTE - Skill Category 1 - 41 © Copyright 2014 / All rights reserved
You are on the product team which is initiating the development of an inventory control system. The system will maintain inventory status and facilitate requisitioning, reordering, and issuing of supplies. The planned life of the system is ten years.
Identify 3 functions or system characteristics of the software and provide at least one Software Quality Factor related to that function or system characteristic.
For this exercise, work in small groups and be prepared to present your essay answer.
Essay (5 minutes):
CSTE - Skill Category 1 - 43 © Copyright 2014 / All rights reserved
What is a Defect
• Process Defects
• Product Defects
Two Types of Defects
A defect is defined as the lack of a desirable state.
CSTE - Skill Category 1 - 44 © Copyright 2014 / All rights reserved
What is a Process?
A process is a set of activities that represent the way work is performed. This work effort includes efforts of people and equipment guided by policies, standards, and procedures. The outcome from a process is usually a product or service.
CSTE - Skill Category 1 - 45 © Copyright 2014 / All rights reserved
What is a Process Defect?
A process is considered defective when the output of that process is outside the accepted control limits.
CSTE - Skill Category 1 - 46 © Copyright 2014 / All rights reserved
Process Control
Out of control processes
CSTE - Skill Category 1 - 47 © Copyright 2014 / All rights reserved
Common Causes of Variation
• All processes contain some inherent variation, or common causes of variation. The amount of variation in a process is quantified with summary statistics.
• In a computer operation, abnormal terminations cause variation. Typical common causes of abnormal terminations included invalid data, no disk space, errors in operating or job control instructions, etc.
CSTE - Skill Category 1 - 48 © Copyright 2014 / All rights reserved
Common Causes of Variation
CSTE - Skill Category 1 - 49 © Copyright 2014 / All rights reserved
Special Causes of Variation
• Special causes of variation are not present in a process
• They occur because of special or unique circumstances
• In the IT example of abnormal terminations in a computer operation, special causes might include:
– citywide power outages
– earthquakes or hurricanes
CSTE - Skill Category 1 - 50 © Copyright 2014 / All rights reserved
Special Causes of Variation
CSTE - Skill Category 1 - 51 © Copyright 2014 / All rights reserved
How Are Processes Brought Under Control?
• Understand process maturity
• Continued process improvement
• Testers should be a contributor
Do Testers Need to KnowStatistical Process Control (SPC)?
• The concept of measuring and reducing variability is commonly called statistical process control (SPC).
• Testers need to understand process variability, because the more variance in the process the greater the need for software testing.
CSTE - Skill Category 1 - 52 © Copyright 2014 / All rights reserved
List three things that could cause common cause of variation within your test processes and three things that could be special causes of variation impacting your test processes.
For this exercise, work in your small groups and be prepared to present your essay answer.
Essay (5 minutes):
CSTE - Skill Category 1 - 54 © Copyright 2014 / All rights reserved
• Producer view
–A deviation from specification: missing, wrong, or extra
• Customer view
–Anything that causes customer dissatisfaction, whether in the specifications or not
What is a Product Defect?
CSTE - Skill Category 1 - 55 © Copyright 2014 / All rights reserved
Why Are Defects Hard to Find?
• Not looking Tests often are not performed becausea particular test condition was unknown
• Looking but not seeingLike losing your keys onlyto discover they were in plain sight the entire time
• The size and complexity of applications often makes it impossible to test everything
CSTE - Skill Category 1 - 56 © Copyright 2014 / All rights reserved
Test Your SensitivityFirst read the sentence enclosed in the box that next appears
Then, count the F’s in the sentence.Count them only once.
Do not go back and count them again!
FINISHED FILES ARE THE RESULT OF YEARS OF SCIENTIFIC STUDY
COMBINED WITH THE EXPERIENCE OF MANY YEARS.
CSTE - Skill Category 1 - 57 © Copyright 2014 / All rights reserved
What Were Your Results?• Number of F’s __________
• Number of F’s __________
• Number of F’s __________
• Number of F’s __________
• Number of F’s __________
• Number of F’s __________
• Number of F’s __________
• Number of F’s __________
CSTE - Skill Category 1 - 59 © Copyright 2014 / All rights reserved
Defects Typically Found in Software Systems AreThe Results of:
• IT improperly interprets requirements• Users specify the wrong requirements• Requirements are incorrectly recorded• Design specifications are incorrect• Program specifications are incorrect• Errors in program coding• Data entry errors• Testing errors• Mistakes in error correction• The corrected condition causes another defect
CSTE - Skill Category 1 - 60 © Copyright 2014 / All rights reserved
Questions?
CSTE - Skill Category 1 - 61 © Copyright 2014 / All rights reserved
Process and Testing StandardsSome development processes are more effective than others. The need for standards within the Software Engineering discipline were noted. Three are listed.
• CMMI-Dev – A process improvement model for software development
• TMMI – A process improvement model for software testing
• ISO29119 – A set of standards for software testing
CSTE - Skill Category 1 - 62 © Copyright 2014 / All rights reserved
SEI’s Five Levels of Process Maturity
CSTE - Skill Category 1 - 63 © Copyright 2014 / All rights reserved
SEI’s Five Levels of Process Maturity
Culture 1: Manage People• Ad hock and chaotic
processes• Success is not repeatable• Political environment• If you are breathing, you
are at level 1!
CSTE - Skill Category 1 - 64 © Copyright 2014 / All rights reserved
SEI’s Five Levels of Process Maturity
Culture 2: Manage Processes• Managed processes• Results predefined• Skills taught
CSTE - Skill Category 1 - 65 © Copyright 2014 / All rights reserved
SEI’s Five Levels of Process Maturity
Culture 3: Defined• Processes rigorously defined• Tailored processes• Proactive process management
CSTE - Skill Category 1 - 66 © Copyright 2014 / All rights reserved
SEI’s Five Levels of Process Maturity
Culture 4: Quantitatively Managed• Customer based objectives• Performance is predictable• Statistical analysis used
CSTE - Skill Category 1 - 67 © Copyright 2014 / All rights reserved
SEI’s Five Levels of Process Maturity
Culture 5: Optimization• Managed innovation• Continuous improvement• Process variations addressed
CSTE - Skill Category 1 - 68 © Copyright 2014 / All rights reserved
TMMI’s Five Levels of Test Maturity
http://www.tmmi.org/?q=node/64
CSTE - Skill Category 1 - 69 © Copyright 2014 / All rights reserved
You are currently operating at CMMI level 2 in your software testing department. Describe what it means to be operating at a level 2 maturity. Be specific and provide examples.
Essay (5 minutes):
CSTE - Skill Category 1 - 71 © Copyright 2014 / All rights reserved
The maturity of the productequals
the maturity of the process
CSTE - Skill Category 1 - 72 © Copyright 2014 / All rights reserved
ISO29119 – A set of process standards for software testing
There are 5 standards:
• ISO 29119-1: Concepts and Definitions
• ISO 29119-2: Test Processes
• ISO 29119-3: Test Documentation
• ISO 29119-4: Test Techniques
• ISO 29119-5: Keyword Driven Testing
CSTE - Skill Category 1 - 73 © Copyright 2014 / All rights reserved
Software TestingThe process of evaluating a deliverable with the intent of finding errors.
• Testing is not a stage/phase of the project • Testing is not just finding broken code• Testing is not a final exam • Testing is not “debugging”
What it’s Not!
CSTE - Skill Category 1 - 74 © Copyright 2014 / All rights reserved
General Testing Guidelines• Testing shows presence of defects
• Exhaustive testing is impossible
• Early testing
• Defect clustering
• Pesticide paradox
• Testing is context dependent
• Absence-of-errors fallacy
• Testing must be traceable to the requirements
• Defect data should be used to focus future efforts
• Testing should be done incrementally
• Focus on exceptions
CSTE - Skill Category 1 - 75 © Copyright 2014 / All rights reserved
Why Do We Test Software• To find defects• To reduce risk inherent in computer systems• To satisfy the customer• To prove that a program is good or no good• Developers are unable to build defect-free software• The user / customer does not know what they want• The development process is defective• To detect variations from specification/expectation• Establishing confidence that a program does what it is
supposed to do
CSTE - Skill Category 1 - 76 © Copyright 2014 / All rights reserved
Factors Affecting Software Testing
• People relationships
• Scope of testing ( producer vs. customer)
• Understanding life cycle testing
• Plan your work and work your plan
• Testing constraints
CSTE - Skill Category 1 - 77 © Copyright 2014 / All rights reserved
Factors Affecting Software Testing: People Relationships
Some attitudes causing a negative view of testing
• Testers hold up implementation, FALSE
• Giving testers less time to test will reduce the chance that they will find defects, FALSE
• Letting the testers find problems is an appropriate way to debug, FALSE
• Defects found in production are the fault of the testers, FALSE; and
• Testers do not need training; only programmers need training, FALSE!
CSTE - Skill Category 1 - 78 © Copyright 2014 / All rights reserved
Factors Affecting Software Testing: The Scope of Software Testing
• Software testing can help ferret out the true needs of the customer.
• Finding defects early in the software development process.
• Removing defects of all types prior to software going into production.
• Identifying weaknesses in the software development process.
CSTE - Skill Category 1 - 79 © Copyright 2014 / All rights reserved
Value of Life Cycle Testing
The Traditional view was that testing was a phase at the end of the project. This increased the cost because:
• errors found late in the product may cause severe consequences
• errors usually occur early in the development cycle.
CSTE - Skill Category 1 - 80 © Copyright 2014 / All rights reserved
Factors Affecting Software Testing: Testing Constraints
Movement of one of the testing constraints will
cause one or more of the other constraints to move
Scope Schedule
QualityResources Technology
CSTE - Skill Category 1 - 81 © Copyright 2014 / All rights reserved
Testing Cost Curve
After the optimum test point the cost of testing to uncover defects exceeds the losses from those defects.
CSTE - Skill Category 1 - 82 © Copyright 2014 / All rights reserved
Factors Affecting Software Testing: Misunderstanding Life Cycle Testing
Life Cycle Phase• Requirements• Design• Program (build/construction)• Test• Installation• Maintenance
CSTE - Skill Category 1 - 83 © Copyright 2014 / All rights reserved
Independent Testing• Ideally independent testers will report to the project manager
and have a reporting structure independent from the group designing or developing the application in order to assure that the quality of the application is given as much consideration as the project budget and timeline.
• The test manager’s responsibilities include:– Test planning and estimation– Designing the test strategy– Reviewing analysis and design artifacts– Chairing the Test Readiness Review– Managing the test effort– Overseeing acceptance tests
CSTE - Skill Category 1 - 84 © Copyright 2014 / All rights reserved
Independent Testing
• Testers are usually responsible for:– Developing test cases and procedures– Test data planning, capture, and conditioning– Reviewing analysis and design artifacts– Testing execution– Utilizing automated test tools for regression testing – Preparing test documentation– Defect tracking and reporting
CSTE - Skill Category 1 - 85 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
There are many Software Development Life Cycle Models• Ad-Hoc• Waterfall• V-Model• Incremental Mode• Iterative Development Model• Prototype/RAD Model• Spiral Model• Reuse Model
CSTE - Skill Category 1 - 86 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
All the models embrace the following in some format:• System Conceptualization• System Design• Unit Development• Software Integration and Testing• Site installation• Training• Implementation• Maintenance
CSTE - Skill Category 1 - 87 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
Ad-hoc• System Capability is unpredictable• Performance depends on individuals• Repeatability depends on individuals
CSTE - Skill Category 1 - 88 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
WaterfallOne of the original models• System Conceptualization is completed• System Design is completed• Coding is completed• Testing is completedProblems• Real projects are uncertain and rarely follow the
sequential model• Process can be long
CSTE - Skill Category 1 - 89 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
Waterfall
CSTE - Skill Category 1 - 90 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
V-Model• Extension of Waterfall• Shows the relationship between specification
development and the associated dynamic testing phase• Shows inverse relationship high level specifications
moving to details and testing more details to high level
CSTE - Skill Category 1 - 91 © Copyright 2014 / All rights reserved
A Life Cycle Quality Approach
Testing throughout development life-cycle
Early development of test requirements
Early detection of errors
At pre-determined points
Involves static and/or dynamic test techniques
The V-Model focuses on:
CSTE - Skill Category 1 - 92 © Copyright 2014 / All rights reserved
The “V” Testing ConceptOperational orBusinessNeed
VerifyOperational
or Business Need
DefineRequirements
VerifyRequirements
DesignSystem
VerifyDesign
BuildSystem
VerifyConstruction
AcceptanceTest
ValidateOperational
or Business Need
SystemTest
ValidateRequirements
IntegrationTest
ValidateDesign
UnitTest
ValidateConstruction
Static Dynamic
Validates
Validates
Validates
Validates
CSTE - Skill Category 1 - 93 © Copyright 2014 / All rights reserved
• User Acceptance Test
• System Test
• Integration Test
• Unit Test
Basic Test Stages
Construction &Unit Test
IntegrationTest
SystemTest
UserAcceptance Test
CSTE - Skill Category 1 - 94 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
IncrementalSuperset of Waterfall• Break Requirements into smaller buildable projects• Use full Lifecycle model for each smaller subset• Each successive increment adds functionality until the
final product is produced• Variants include Agile, Spiral, and RAD
CSTE - Skill Category 1 - 95 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
Incremental
CSTE - Skill Category 1 - 96 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
Iterative• Break Project into small parts• Get user feedback• Some variations allow immediate implementation into
productionProblems• Extensive user involvement• Communications take center stage• Requests for changes may cause confusion
CSTE - Skill Category 1 - 97 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
Prototyping
• Creation of the major user interfaces without any substantive coding in the background in order to give the users a “feel” for what the system will look like
• Development of an abbreviated version of the system that performs a limited subset of functions; development of a paper system (depicting proposed screens, reports, relationships etc.)
• Use of an existing system or system components to demonstrate some functions that will be included in the developed system
CSTE - Skill Category 1 - 98 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
Rapid Application Development (RAD)
Variation of Prototyping
• Less emphasis on detailed requirements
• User interacts with development team
• Four phases:Requirements Planning PhaseUser Design PhaseConstruction PhaseCutover Phase
CSTE - Skill Category 1 - 99 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
Rapid Application Development (RAD)
CSTE - Skill Category 1 - 100 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
Spiral Model
Waterfall plus Prototyping
• Initial version developed and then repeatedly modified
• Each version designed based on Waterfall method
• Spiral out from the centre
CSTE - Skill Category 1 - 101 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
Spiral
CSTE - Skill Category 1 - 102 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
Reuse Model
Build using Existing Components
• Library of Procedural Modules
• Library of Database Modules
• Borrow and use in the new function
CSTE - Skill Category 1 - 103 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
Agile
Approaches which include the following are collectively known as Agile
• Scrum
• Crystal Clear
• Adaptive Software Development
• Feature Driven Development
• Dynamic Systems Development Methodology (DSDM)
• Extreme Programming
CSTE - Skill Category 1 - 104 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
Agile Practices
• Detailed Planning for each iteration
• Test Driven Development
• Refactor Relentlessly
• Continuous Integration
• Paired Programming
• Onsite Customer
CSTE - Skill Category 1 - 105 © Copyright 2014 / All rights reserved
Software Development Life Cycle Models
Creating and Combining Models
Models should be combined or modified based the relationship between the Information System and its organizational environment. Environments can be classified
• Unchanging
• Turbulent
• Uncertain
• Adaptive
CSTE - Skill Category 1 - 106 © Copyright 2014 / All rights reserved
Questions?
CSTE - Skill Category 1 - 107 © Copyright 2014 / All rights reserved
You feel that a Software Development Life Cycle testing model should be implemented in your testing approach. What justification would you provide to your manager to sell a defined life cycle approach?
Essay (5 minutes)
CSTE - Skill Category 1 - 109 © Copyright 2014 / All rights reserved
Testing Throughout the Lifecycle
• Continuous testing throughout the development process
• Must have formalized lifecycle
• Pre-determined deliverables
CSTE - Skill Category 1 - 110 © Copyright 2014 / All rights reserved
There are a number of constraints that must be recognized and planned for accordingly. An example of a constraint might be budgetary constraints. List 3 additional constraints of which a test manager might need to be aware and describe how trade-off decisions might be necessary regarding these three constraints.
Essay (10 minutes)
CSTE - Skill Category 1 - 112 © Copyright 2014 / All rights reserved
Static vs Dynamic TestingVerification – Static Test
• Walkthroughs
• Reviews
• Inspections
Validation – Dynamic Test• Test cases
• Scripting
• Feasibility reviews• Requirements reviews• Design reviews• Code walkthroughs• Code Inspections• Requirements tracing
• White Box
• Black Box
CSTE - Skill Category 1 - 113 © Copyright 2014 / All rights reserved
Verification and Validation Techniques
• Techniques for quantifiably assuring that a work product meets its stated objectives:
– Verification: Performed during development on key artifacts
• Walkthroughs, reviews and inspections
• Mentor feedback, training, checklists and standards
– Validation: Performed after a work product is produced
• Validating against established criteria
• Ensuring product integrates correctly into the environment
• Reactive approach focused on defect detection and removal
CSTE - Skill Category 1 - 114 © Copyright 2014 / All rights reserved
Verification and Validation Techniques
• Techniques for quantifiably assuring that a work product meets its stated objectives:
– Verification: Performed during development on key artifacts
• Walkthroughs, reviews and inspections
• Mentor feedback, training, checklists and standards
– Validation: Performed after a work product is produced
• Validating against established criteria
• Ensuring product integrates correctly into the environment
• Reactive approach focused on defect detection and removal
EXAM HINTThere is some variation in the industry regarding these definitions. Know them as stated for exam!
CSTE - Skill Category 1 - 115 © Copyright 2014 / All rights reserved
Walkthroughs, Reviews, Inspections
• Feasibility reviews• Requirements reviews• Design reviews• Code walkthroughs• Code Inspections• Requirements tracing
• Walkthroughs and inspections are very disciplined procedures aimed at removing the major responsibility for verification from the developer.
• The main goal of a review is to identify defects within the stage or phase of the project where they originate, rather than in later test stages; this is referred to as “stage containment.”
CSTE - Skill Category 1 - 116 © Copyright 2014 / All rights reserved
Walkthroughs, Reviews, Inspections
• Feasibility reviews• Requirements reviews• Design reviews• Code walkthroughs• Code Inspections• Requirements tracing
• Walkthroughs and inspections are very disciplined procedures aimed at removing the major responsibility for verification from the developer.
• The main goal of a review is to identify defects within the stage or phase of the project where they originate, rather than in later test stages; this is referred to as “stage containment.”
Described in detail in STBOK Section 6
CSTE - Skill Category 1 - 117 © Copyright 2014 / All rights reserved
Unit Testing Testing individual programs or components to
validate that the logic works according to specification
Validates technical quality of the code
Conducted by the developer who created the component
Begins once component development is complete
CSTE - Skill Category 1 - 118 © Copyright 2014 / All rights reserved
Integration Testing Tests the integration of components that have been
successfully unit-tested
Validates the technical quality of the design
Validates the integration of the application and the environment
Conducted by development with support from the test team
Begins once the first components have passed unit test
CSTE - Skill Category 1 - 119 © Copyright 2014 / All rights reserved
System Testing Tests the entire assembled system
Validates delivery of functional and non-functional requirements
Validates interface to upstream and downstream applications
Conducted by independent test team
Begins when integration testing has been successfully completed
CSTE - Skill Category 1 - 120 © Copyright 2014 / All rights reserved
User Acceptance Test
Validates system is “fit for use”
Evaluates how system integrates with manual business processes
Conducted by end users, customers or designated representatives
Begins when system test has been successfully completed
CSTE - Skill Category 1 - 121 © Copyright 2014 / All rights reserved
Essay (6 minutes)In software testing there are at least four stages of testing. Listthe sequence in which those four stages of testing should occurfrom one, (which is the first), to four, (which is the last). Brieflyexplain the objective of each of these four stages of testing.
CSTE - Skill Category 1 - 123 © Copyright 2014 / All rights reserved
ExerciseUnit and Integration testing primarily focuses on which of the following?
a) Verification
b) Validation
c) Test planning
d) Both a & c
Tests that validate system requirements are often called.
a) Functional Tests
b) Structural Tests
CSTE - Skill Category 1 - 126 © Copyright 2014 / All rights reserved
ExerciseUnit and Integration testing primarily focuses on which of the following?
a) Verification
b) Validation
c) Test planning
d) Both a & c
Tests that validate system requirements are often called.
a) Functional Tests
b) Structural Tests
EXAM HINTFor multiple choice parts, read each question’s stem twice, then read ALL the responses
CSTE - Skill Category 1 - 127 © Copyright 2014 / All rights reserved
Traceability Matrix
• Translation from Stage to Stage
• Requirements to be traced throughout the Lifecycle
Requirement ID Func Rqmt 1.1
FuncRqmt1.2
Func Rqmt 1.3
Func Rqmt 1.x
Func Rqmt
x.x
Technical Rqmt 1.1
Technical Rqmt 1.2
Technical Rqmt 1.x
TestCases 3 2 3 1 2 1 1 1
1.1.1 x
1.1.2 x x
CSTE - Skill Category 1 - 128 © Copyright 2014 / All rights reserved
Testing Schools of Thought
A ‘School of Thought’ is a system of beliefs shared by a group. Some of the Schools of thought are:
• Analytical School – testing is rigorous and technical
• Factory School – reduction of testing tasks to basic routines or repetitive tasks
• Quality (Control) School – process and standards
• Context-driven School –focus on product and people
• Agile School – continuous delivery of incremental products
CSTE - Skill Category 1 - 129 © Copyright 2014 / All rights reserved
Testing Approaches
The approaches do not relate to a model or school:
• Requirements-based Testing – quality of Requirements
• Risk-based Testing – list, prioritize, and test based on the assessed risk
• Model-based Testing – create a (simplified) model and build testcases based on the model
• Exploratory Testing - used by professional testers based on their knowledge of the system
• Keyword-driven Testing - define keywords or actions for each function to drive the testing
CSTE - Skill Category 1 - 130 © Copyright 2014 / All rights reserved
Test Categories and Testing Techniques
• Structural
• Functional
• Non-Functional
Will be discussed on the following slides.
CSTE - Skill Category 1 - 131 © Copyright 2014 / All rights reserved
Test Categories• Structural Tests
Tests that validate system architecture is structural testing. Also considered white box testing because knowledge of the internal logic of the system is used to develop test cases.
• Functional Tests
Tests that validate system are called functional testing. This testing addresses the overall behavior of the program by testing transacting flows, input validation, and functional completeness but no knowledge of the internal logic system is used (black box).
• Non-Functional Tests (Software Quality Factors)
Tests that validate system characteristics, such as performance, stability, maintainability, usability, and security. Can be considered a user’s point of view.
CSTE - Skill Category 1 - 132 © Copyright 2014 / All rights reserved
Structural Test TechniquesStructural system testing is designed to verify that the developed system and programs work.
CSTE - Skill Category 1 - 133 © Copyright 2014 / All rights reserved
• Structural test technique• Testing based on knowledge of internal code structure and
logic - usually logic driven
White-Box Testing
Internal View
What is the system doing?How is it doing it?
Input
Output
CSTE - Skill Category 1 - 134 © Copyright 2014 / All rights reserved
White-Box Techniques• Statement coverage
• Decision Coverage
• Condition Coverage
• Decision/Condition Coverage
• Multiple Condition Coverage
CSTE - Skill Category 1 - 135 © Copyright 2014 / All rights reserved
Functional Test TechniquesFunctional system testing ensures that the system requirements and specifications are achieved.
Technique Description ExampleRequirements System performs as specified. * Prove system requirements
* Compliance to policies, regulations
Error Handling Errors can be prevented or detected and then corrected.
* Error introduced into test* Errors re-entered
Intersystem Data is correctly passed from system to system.
* Intersystem parameters changed* Intersystem documentation updated
Control Controls reduce system risk to an acceptable level.
* File reconciliation procedures work* Manual controls in place
Parallel Old system and new system are run and the results compared to detect unplanned differences.
* Old and new systems
CSTE - Skill Category 1 - 136 © Copyright 2014 / All rights reserved
Black-Box Testing• Functional test technique
• Testing based on external specifications without knowledge of how the system is constructed - usually data or business process driven
• Validates that each input produces the appropriate output
External View
What did the application do?What should it have done?
Input
Output
CSTE - Skill Category 1 - 137 © Copyright 2014 / All rights reserved
Black-Box TechniquesEquivalence partitioning
Test cases generated using a subset of data to represent a larger class of data
Example:
A business rule edits credit limits within a given range ($10,000 - $15,000) could have three equivalence classes:
Less than $10,000 (Invalid)Between $10,000 and $15,000 (Valid)Greater than $15,000 (Invalid)
CSTE - Skill Category 1 - 138 © Copyright 2014 / All rights reserved
Black-Box Techniques (continued)
Boundary analysis
Test cases designed using data from the limits of the input and output domain classes
Example (continued)
Testing the boundaries of a business rule that edits credit limits within a given range ($10,000 - $15,000) would test:
Low boundary +/- one ($9,999 and $10,001)On the boundary +/- one ($10,000 and $15,000)Upper boundary +/- one ($14,999 and $15,001)
CSTE - Skill Category 1 - 139 © Copyright 2014 / All rights reserved
Black-Box Techniques (continued)
Other types include:
• Decision Table Testing
• State Transition Testing
• All-pairs (pairwise) Testing
• Cause-Effect Graphing
CSTE - Skill Category 1 - 140 © Copyright 2014 / All rights reserved
Black-Box Techniques (continued)
Ad Hoc or Error Guessing
Test cases and data designed based upon experience and intuition of the tester
Example
Testing the date-driven activities performed on credit reports, the tester might test the dates below based upon experience that year-end, month-end, and other dates usually cause problems:
2/29/2006
CSTE - Skill Category 1 - 141 © Copyright 2014 / All rights reserved
Non-Functional Test Types
• Accessibility Testing
• Conversion Testing
• Maintainability Testing
• Reliability Testing
• Stability Testing
• Usability Testing
CSTE - Skill Category 1 - 142 © Copyright 2014 / All rights reserved
Incremental Testing• Top-Down - testing begins with module
highest in the hierarchy using stubs to simulate lower interfacing modules; modules are added in descending hierarchical order
• Bottom-Up - testing begins with module at bottom of hierarchy using drivers or test harnesses to simulate higher interfacing modules; modules are added in ascending hierarchical order
CSTE - Skill Category 1 - 143 © Copyright 2014 / All rights reserved
Thread Testing• Often used during the integration of units or components
• Demonstrates key functional capabilities by testing a string of units or components that accomplish a specific function in the application
• Can be performed simultaneously with incremental testing
CSTE - Skill Category 1 - 144 © Copyright 2014 / All rights reserved
Regression Testing
Regression testing is a decision to re-test something that has already been tested looking for inadvertently introduced defects. Regression Testing should be performed:
• New releases of packaged software
• Application is enhanced or changes are made
• Support software changes
• Either side of a system interface is changed
• Changes to configuration
• Whenever changes are made after a testing stage is completed
CSTE - Skill Category 1 - 145 © Copyright 2014 / All rights reserved
Exercise
A. InspectionsB. Black BoxC. Capture/PlaybackD. Stress TestingE. Regression Testing
1. Testing previously verified logic
2. Testing Branches/paths
3. Re-use of test data
4. Test correct implementation of specifications
5. Testing processing limits
Match the items to the numbered list
1
2
34
5
CSTE - Skill Category 1 - 146 © Copyright 2014 / All rights reserved
Questions?
CSTE - Skill Category 1 - 147 © Copyright 2014 / All rights reserved
Skill Category 1 - SummarySoftware Testing Principles and Concepts
The following topics were discussed in this Skill Category:
• Vocabulary & Quality Basics
• Understanding Defects
• Process and Testing Published Standards
• Software Testing
• Software Development Life Cycle (SDLC) Models
• Agile Development Methodologies
• Testing Throughout the SDLC
• Testing Schools of Thought and Approaches
• Test Categories and Testing Techniques