what is a process? · pdf fileessay (5 minutes): cste -skill category 1 -54 © copyright...

CSTE - Skill Category 1 - 44 © Copyright 2014 / All rights reserved

What is a Process?

A process is a set of activities that represent the way work is performed. This work effort includes efforts of people and equipment guided by policies, standards, and procedures. The outcome from a process is usually a product or service.


What is a Process Defect?

A process is considered defective when the output of that process is outside the accepted control limits.


Process Control

Out of control processes


Common Causes of Variation

• All processes contain some inherent variation, or common causes of variation. The amount of variation in a process is quantified with summary statistics.

• In a computer operation, abnormal terminations cause variation. Typical common causes of abnormal terminations included invalid data, no disk space, errors in operating or job control instructions, etc.


Common Causes of Variation


Special Causes of Variation

• Special causes of variation are not present in a process

• They occur because of special or unique circumstances

• In the IT example of abnormal terminations in a computer operation, special causes might include:

– citywide power outages

– earthquakes or hurricanes


Special Causes of Variation


How Are Processes Brought Under Control?

• Understand process maturity

• Continued process improvement

• Testers should be a contributor

Do Testers Need to KnowStatistical Process Control (SPC)?

• The concept of measuring and reducing variability is commonly called statistical process control (SPC).

• Testers need to understand process variability, because the more variance in the process the greater the need for software testing.


List three things that could cause common cause of variation within your test processes and three things that could be special causes of variation impacting your test processes.

For this exercise, work in your small groups and be prepared to present your essay answer.

Essay (5 minutes):


• Producer view

–A deviation from specification: missing, wrong, or extra

• Customer view

–Anything that causes customer dissatisfaction, whether in the specifications or not

What is a Product Defect?


Why Are Defects Hard to Find?

• Not looking Tests often are not performed becausea particular test condition was unknown

• Looking but not seeingLike losing your keys onlyto discover they were in plain sight the entire time

• The size and complexity of applications often makes it impossible to test everything


Test Your SensitivityFirst read the sentence enclosed in the box that next appears

Then, count the F’s in the sentence.Count them only once.

Do not go back and count them again!

FINISHED FILES ARE THE RESULT OF YEARS OF SCIENTIFIC STUDY

COMBINED WITH THE EXPERIENCE OF MANY YEARS.


What Were Your Results?• Number of F’s __________

• Number of F’s __________

• Number of F’s __________

• Number of F’s __________

• Number of F’s __________

• Number of F’s __________

• Number of F’s __________

• Number of F’s __________


Defects Typically Found in Software Systems AreThe Results of:

• IT improperly interprets requirements• Users specify the wrong requirements• Requirements are incorrectly recorded• Design specifications are incorrect• Program specifications are incorrect• Errors in program coding• Data entry errors• Testing errors• Mistakes in error correction• The corrected condition causes another defect


Questions?


Process and Testing StandardsSome development processes are more effective than others. The need for standards within the Software Engineering discipline were noted. Three are listed.

• CMMI-Dev – A process improvement model for software development

• TMMI – A process improvement model for software testing

• ISO29119 – A set of standards for software testing


SEI’s Five Levels of Process Maturity



Culture 1: Manage People• Ad hock and chaotic

processes• Success is not repeatable• Political environment• If you are breathing, you

are at level 1!



Culture 2: Manage Processes• Managed processes• Results predefined• Skills taught



Culture 3: Defined• Processes rigorously defined• Tailored processes• Proactive process management



Culture 4: Quantitatively Managed• Customer based objectives• Performance is predictable• Statistical analysis used



Culture 5: Optimization• Managed innovation• Continuous improvement• Process variations addressed


TMMI’s Five Levels of Test Maturity

http://www.tmmi.org/?q=node/64


You are currently operating at CMMI level 2 in your software testing department. Describe what it means to be operating at a level 2 maturity. Be specific and provide examples.

Essay (5 minutes):


The maturity of the productequals

the maturity of the process


ISO29119 – A set of process standards for software testing

There are 5 standards:

• ISO 29119-1: Concepts and Definitions

• ISO 29119-2: Test Processes

• ISO 29119-3: Test Documentation

• ISO 29119-4: Test Techniques

• ISO 29119-5: Keyword Driven Testing


Software TestingThe process of evaluating a deliverable with the intent of finding errors.

• Testing is not a stage/phase of the project • Testing is not just finding broken code• Testing is not a final exam • Testing is not “debugging”

What it’s Not!


General Testing Guidelines• Testing shows presence of defects

• Exhaustive testing is impossible

• Early testing

• Defect clustering

• Pesticide paradox

• Testing is context dependent

• Absence-of-errors fallacy

• Testing must be traceable to the requirements

• Defect data should be used to focus future efforts

• Testing should be done incrementally

• Focus on exceptions


Why Do We Test Software• To find defects• To reduce risk inherent in computer systems• To satisfy the customer• To prove that a program is good or no good• Developers are unable to build defect-free software• The user / customer does not know what they want• The development process is defective• To detect variations from specification/expectation• Establishing confidence that a program does what it is

supposed to do


Factors Affecting Software Testing

• People relationships

• Scope of testing ( producer vs. customer)

• Understanding life cycle testing

• Plan your work and work your plan

• Testing constraints


Factors Affecting Software Testing: People Relationships

Some attitudes causing a negative view of testing

• Testers hold up implementation, FALSE

• Giving testers less time to test will reduce the chance that they will find defects, FALSE

• Letting the testers find problems is an appropriate way to debug, FALSE

• Defects found in production are the fault of the testers, FALSE; and

• Testers do not need training; only programmers need training, FALSE!


Factors Affecting Software Testing: The Scope of Software Testing

• Software testing can help ferret out the true needs of the customer.

• Finding defects early in the software development process.

• Removing defects of all types prior to software going into production.

• Identifying weaknesses in the software development process.


Value of Life Cycle Testing

The Traditional view was that testing was a phase at the end of the project. This increased the cost because:

• errors found late in the product may cause severe consequences

• errors usually occur early in the development cycle.


Factors Affecting Software Testing: Testing Constraints

Movement of one of the testing constraints will

cause one or more of the other constraints to move

Scope Schedule

QualityResources Technology


Testing Cost Curve

After the optimum test point the cost of testing to uncover defects exceeds the losses from those defects.


Factors Affecting Software Testing: Misunderstanding Life Cycle Testing

Life Cycle Phase• Requirements• Design• Program (build/construction)• Test• Installation• Maintenance


Independent Testing• Ideally independent testers will report to the project manager

and have a reporting structure independent from the group designing or developing the application in order to assure that the quality of the application is given as much consideration as the project budget and timeline.

• The test manager’s responsibilities include:– Test planning and estimation– Designing the test strategy– Reviewing analysis and design artifacts– Chairing the Test Readiness Review– Managing the test effort– Overseeing acceptance tests


Independent Testing

• Testers are usually responsible for:– Developing test cases and procedures– Test data planning, capture, and conditioning– Reviewing analysis and design artifacts– Testing execution– Utilizing automated test tools for regression testing – Preparing test documentation– Defect tracking and reporting


Software Development Life Cycle Models

There are many Software Development Life Cycle Models• Ad-Hoc• Waterfall• V-Model• Incremental Mode• Iterative Development Model• Prototype/RAD Model• Spiral Model• Reuse Model



All the models embrace the following in some format:• System Conceptualization• System Design• Unit Development• Software Integration and Testing• Site installation• Training• Implementation• Maintenance



Ad-hoc• System Capability is unpredictable• Performance depends on individuals• Repeatability depends on individuals



WaterfallOne of the original models• System Conceptualization is completed• System Design is completed• Coding is completed• Testing is completedProblems• Real projects are uncertain and rarely follow the

sequential model• Process can be long



Waterfall



V-Model• Extension of Waterfall• Shows the relationship between specification

development and the associated dynamic testing phase• Shows inverse relationship high level specifications

moving to details and testing more details to high level


A Life Cycle Quality Approach

Testing throughout development life-cycle

Early development of test requirements

Early detection of errors

At pre-determined points

Involves static and/or dynamic test techniques

The V-Model focuses on:


The “V” Testing ConceptOperational orBusinessNeed

VerifyOperational

or Business Need

DefineRequirements

VerifyRequirements

DesignSystem

VerifyDesign

BuildSystem

VerifyConstruction

AcceptanceTest

ValidateOperational

or Business Need

SystemTest

ValidateRequirements

IntegrationTest

ValidateDesign

UnitTest

ValidateConstruction

Static Dynamic

Validates

Validates

Validates

Validates


• User Acceptance Test

• System Test

• Integration Test

• Unit Test

Basic Test Stages

Construction &Unit Test

IntegrationTest

SystemTest

UserAcceptance Test



IncrementalSuperset of Waterfall• Break Requirements into smaller buildable projects• Use full Lifecycle model for each smaller subset• Each successive increment adds functionality until the

final product is produced• Variants include Agile, Spiral, and RAD



Incremental



Iterative• Break Project into small parts• Get user feedback• Some variations allow immediate implementation into

productionProblems• Extensive user involvement• Communications take center stage• Requests for changes may cause confusion



Prototyping

• Creation of the major user interfaces without any substantive coding in the background in order to give the users a “feel” for what the system will look like

• Development of an abbreviated version of the system that performs a limited subset of functions; development of a paper system (depicting proposed screens, reports, relationships etc.)

• Use of an existing system or system components to demonstrate some functions that will be included in the developed system



Rapid Application Development (RAD)

Variation of Prototyping

• Less emphasis on detailed requirements

• User interacts with development team

• Four phases:Requirements Planning PhaseUser Design PhaseConstruction PhaseCutover Phase



Rapid Application Development (RAD)



Spiral Model

Waterfall plus Prototyping

• Initial version developed and then repeatedly modified

• Each version designed based on Waterfall method

• Spiral out from the centre



Spiral



Reuse Model

Build using Existing Components

• Library of Procedural Modules

• Library of Database Modules

• Borrow and use in the new function



Agile

Approaches which include the following are collectively known as Agile

• Scrum

• Crystal Clear

• Adaptive Software Development

• Feature Driven Development

• Dynamic Systems Development Methodology (DSDM)

• Extreme Programming



Agile Practices

• Detailed Planning for each iteration

• Test Driven Development

• Refactor Relentlessly

• Continuous Integration

• Paired Programming

• Onsite Customer



Creating and Combining Models

Models should be combined or modified based the relationship between the Information System and its organizational environment. Environments can be classified

• Unchanging

• Turbulent

• Uncertain

• Adaptive


Questions?


You feel that a Software Development Life Cycle testing model should be implemented in your testing approach. What justification would you provide to your manager to sell a defined life cycle approach?

Essay (5 minutes)


Testing Throughout the Lifecycle

• Continuous testing throughout the development process

• Must have formalized lifecycle

• Pre-determined deliverables


There are a number of constraints that must be recognized and planned for accordingly. An example of a constraint might be budgetary constraints. List 3 additional constraints of which a test manager might need to be aware and describe how trade-off decisions might be necessary regarding these three constraints.

Essay (10 minutes)


Static vs Dynamic TestingVerification – Static Test

• Walkthroughs

• Reviews

• Inspections

Validation – Dynamic Test• Test cases

• Scripting

• Feasibility reviews• Requirements reviews• Design reviews• Code walkthroughs• Code Inspections• Requirements tracing

• White Box

• Black Box


Verification and Validation Techniques

• Techniques for quantifiably assuring that a work product meets its stated objectives:

– Verification: Performed during development on key artifacts

• Walkthroughs, reviews and inspections

• Mentor feedback, training, checklists and standards

– Validation: Performed after a work product is produced

• Validating against established criteria

• Ensuring product integrates correctly into the environment

• Reactive approach focused on defect detection and removal


Verification and Validation Techniques

• Techniques for quantifiably assuring that a work product meets its stated objectives:

– Verification: Performed during development on key artifacts

• Walkthroughs, reviews and inspections

• Mentor feedback, training, checklists and standards

– Validation: Performed after a work product is produced

• Validating against established criteria

• Ensuring product integrates correctly into the environment

• Reactive approach focused on defect detection and removal

EXAM HINTThere is some variation in the industry regarding these definitions. Know them as stated for exam!


Walkthroughs, Reviews, Inspections


• Walkthroughs and inspections are very disciplined procedures aimed at removing the major responsibility for verification from the developer.

• The main goal of a review is to identify defects within the stage or phase of the project where they originate, rather than in later test stages; this is referred to as “stage containment.”


Walkthroughs, Reviews, Inspections


• Walkthroughs and inspections are very disciplined procedures aimed at removing the major responsibility for verification from the developer.

• The main goal of a review is to identify defects within the stage or phase of the project where they originate, rather than in later test stages; this is referred to as “stage containment.”

Described in detail in STBOK Section 6


Unit Testing Testing individual programs or components to

validate that the logic works according to specification

Validates technical quality of the code

Conducted by the developer who created the component

Begins once component development is complete


Integration Testing Tests the integration of components that have been

successfully unit-tested

Validates the technical quality of the design

Validates the integration of the application and the environment

Conducted by development with support from the test team

Begins once the first components have passed unit test


System Testing Tests the entire assembled system

Validates delivery of functional and non-functional requirements

Validates interface to upstream and downstream applications

Conducted by independent test team

Begins when integration testing has been successfully completed


User Acceptance Test

Validates system is “fit for use”

Evaluates how system integrates with manual business processes

Conducted by end users, customers or designated representatives

Begins when system test has been successfully completed


Essay (6 minutes)In software testing there are at least four stages of testing. Listthe sequence in which those four stages of testing should occurfrom one, (which is the first), to four, (which is the last). Brieflyexplain the objective of each of these four stages of testing.


ExerciseUnit and Integration testing primarily focuses on which of the following?

a) Verification

b) Validation

c) Test planning

d) Both a & c

Tests that validate system requirements are often called.

a) Functional Tests

b) Structural Tests


ExerciseUnit and Integration testing primarily focuses on which of the following?

a) Verification

b) Validation

c) Test planning

d) Both a & c

Tests that validate system requirements are often called.

a) Functional Tests

b) Structural Tests

EXAM HINTFor multiple choice parts, read each question’s stem twice, then read ALL the responses


Traceability Matrix

• Translation from Stage to Stage

• Requirements to be traced throughout the Lifecycle

Requirement ID Func Rqmt 1.1

FuncRqmt1.2

Func Rqmt 1.3

Func Rqmt 1.x

Func Rqmt

x.x

Technical Rqmt 1.1

Technical Rqmt 1.2

Technical Rqmt 1.x

TestCases 3 2 3 1 2 1 1 1

1.1.1 x

1.1.2 x x


Testing Schools of Thought

A ‘School of Thought’ is a system of beliefs shared by a group. Some of the Schools of thought are:

• Analytical School – testing is rigorous and technical

• Factory School – reduction of testing tasks to basic routines or repetitive tasks

• Quality (Control) School – process and standards

• Context-driven School –focus on product and people

• Agile School – continuous delivery of incremental products


Testing Approaches

The approaches do not relate to a model or school:

• Requirements-based Testing – quality of Requirements

• Risk-based Testing – list, prioritize, and test based on the assessed risk

• Model-based Testing – create a (simplified) model and build testcases based on the model

• Exploratory Testing - used by professional testers based on their knowledge of the system

• Keyword-driven Testing - define keywords or actions for each function to drive the testing


Test Categories and Testing Techniques

• Structural

• Functional

• Non-Functional

Will be discussed on the following slides.


Test Categories• Structural Tests

Tests that validate system architecture is structural testing. Also considered white box testing because knowledge of the internal logic of the system is used to develop test cases.

• Functional Tests

Tests that validate system are called functional testing. This testing addresses the overall behavior of the program by testing transacting flows, input validation, and functional completeness but no knowledge of the internal logic system is used (black box).

• Non-Functional Tests (Software Quality Factors)

Tests that validate system characteristics, such as performance, stability, maintainability, usability, and security. Can be considered a user’s point of view.


Structural Test TechniquesStructural system testing is designed to verify that the developed system and programs work.


• Structural test technique• Testing based on knowledge of internal code structure and

logic - usually logic driven

White-Box Testing

Internal View

What is the system doing?How is it doing it?

Input

Output


White-Box Techniques• Statement coverage

• Decision Coverage

• Condition Coverage

• Decision/Condition Coverage

• Multiple Condition Coverage


Functional Test TechniquesFunctional system testing ensures that the system requirements and specifications are achieved.

Technique Description ExampleRequirements System performs as specified. * Prove system requirements

* Compliance to policies, regulations

Error Handling Errors can be prevented or detected and then corrected.

* Error introduced into test* Errors re-entered

Intersystem Data is correctly passed from system to system.

* Intersystem parameters changed* Intersystem documentation updated

Control Controls reduce system risk to an acceptable level.

* File reconciliation procedures work* Manual controls in place

Parallel Old system and new system are run and the results compared to detect unplanned differences.

* Old and new systems


Black-Box Testing• Functional test technique

• Testing based on external specifications without knowledge of how the system is constructed - usually data or business process driven

• Validates that each input produces the appropriate output

External View

What did the application do?What should it have done?

Input

Output


Black-Box TechniquesEquivalence partitioning

Test cases generated using a subset of data to represent a larger class of data

Example:

A business rule edits credit limits within a given range ($10,000 - $15,000) could have three equivalence classes:

Less than $10,000 (Invalid)Between $10,000 and $15,000 (Valid)Greater than $15,000 (Invalid)


Black-Box Techniques (continued)

Boundary analysis

Test cases designed using data from the limits of the input and output domain classes

Example (continued)

Testing the boundaries of a business rule that edits credit limits within a given range ($10,000 - $15,000) would test:

Low boundary +/- one ($9,999 and $10,001)On the boundary +/- one ($10,000 and $15,000)Upper boundary +/- one ($14,999 and $15,001)



Other types include:

• Decision Table Testing

• State Transition Testing

• All-pairs (pairwise) Testing

• Cause-Effect Graphing



Ad Hoc or Error Guessing

Test cases and data designed based upon experience and intuition of the tester

Example

Testing the date-driven activities performed on credit reports, the tester might test the dates below based upon experience that year-end, month-end, and other dates usually cause problems:

2/29/2006


Non-Functional Test Types

• Accessibility Testing

• Conversion Testing

• Maintainability Testing

• Reliability Testing

• Stability Testing

• Usability Testing


Incremental Testing• Top-Down - testing begins with module

highest in the hierarchy using stubs to simulate lower interfacing modules; modules are added in descending hierarchical order

• Bottom-Up - testing begins with module at bottom of hierarchy using drivers or test harnesses to simulate higher interfacing modules; modules are added in ascending hierarchical order


Thread Testing• Often used during the integration of units or components

• Demonstrates key functional capabilities by testing a string of units or components that accomplish a specific function in the application

• Can be performed simultaneously with incremental testing


Regression Testing

Regression testing is a decision to re-test something that has already been tested looking for inadvertently introduced defects. Regression Testing should be performed:

• New releases of packaged software

• Application is enhanced or changes are made

• Support software changes

• Either side of a system interface is changed

• Changes to configuration

• Whenever changes are made after a testing stage is completed


Exercise

A. InspectionsB. Black BoxC. Capture/PlaybackD. Stress TestingE. Regression Testing

1. Testing previously verified logic

2. Testing Branches/paths

3. Re-use of test data

4. Test correct implementation of specifications

5. Testing processing limits

Match the items to the numbered list

1

2

34

5


Questions?


Skill Category 1 - SummarySoftware Testing Principles and Concepts

The following topics were discussed in this Skill Category:

• Vocabulary & Quality Basics

• Understanding Defects

• Process and Testing Published Standards

• Software Testing

• Software Development Life Cycle (SDLC) Models

• Agile Development Methodologies

• Testing Throughout the SDLC

• Testing Schools of Thought and Approaches

• Test Categories and Testing Techniques

what is a process? · pdf fileessay (5 minutes): cste -skill category 1 -54 © copyright...

Documents