07 regression-testing-and-scm

Product Release ,Product Release ,Configuration Management,Configuration Management,

andand Regression Testing Regression Testing

G [email protected]

References for this References for this classclass

• Software Maintenance, Gopalaswamy Ramesh and Ramesh Bhattiprolu

• Managing Global Software Projects, Gopalaswamy Ramesh, Chapters 6, 17, 18)

• Software Testing, Principles and Practices, Gopalaswamy Ramesh and Srinivasan Desikan, Chapter 15

• Slides in class

2

Today’s AgendaToday’s Agenda

Product Release Process, Product Release Process,

Configuration Management and Configuration Management and

Regression Testing Regression Testing • Understand typical product release processes• Know the role of configuration management and

other environmental tools in the release process• Appreciate the close link between product release,

maintenance and (regression) testing• Get into details of how to effectively plan

regression testing for a product

3

Genesis and effect of change…Genesis and effect of change…

4

Change in UserRequirements

Change in TechnicalRequirements

Change in BusinessRequirements

Approved Change Requests

ProjectPlan,

Other Documents

Code,Test Data etc

ProductDocumentation

ProductTraining Material

5

When any of the affected components are not changed consistently,

It results in defects to be reported from the point of view of customers…

Impacts of (Uncontrolled) ChangeImpacts of (Uncontrolled) Change• Product Behavior Expectation Changes • Re-work• Slipped schedule• Incompatibility across components• Question of “who pays for it”?• Possible mistrust and “padded” schedules and

costs

6

External Manifestation of Change: External Manifestation of Change:

Understanding Product Release Understanding Product Release

ProcessProcess

V1

V2

V3

1.1 1.2 …

2.1 2.2

Time for which BOTH V1 and V2Are to be maintained

V1 Obsolete

V2 Obsolete

TIME

7

What is a “V2” made What is a “V2” made up of?up of?

8

Relevant fixes from V1.1, V1.2… that are applicable to V2Plus

Support for the new technologies introduced in V2Minus

Support for older technologies in V1 that are no longer applicable in V2

Implications of Product Release Implications of Product Release

Process on Testing and MaintenanceProcess on Testing and Maintenance• A defect may surface in any one release / version

and may have to be tested in other releases• A fix may be made on one release / version and

may then have to be moved to other releases / versions

• Maintenance / Testing effect can be compounded

9




Regression Testing Regression Testing • Understand typical product release processes• Know the role of configuration management and




10

Topics in Topics in Configuration Configuration ManagementManagement

• Change Management Configuration Managemento The transition and the differenceo Definitions and Terminologyo The vendor’s viewpointo Processes and Activities in Software Configuration Management

• SCM Tools and Automation o Need for toolso Typical functions served by toolso Integrating with vendor’s tools

• SCM Issues in Global Teams

11

12

Accounts Receivable V1.2

General Ledger V2.1

GL Users Guide V2AR Users Guide V1

SourceFiles(e.g., .c / .h)

SystemTools(e.g.,

compilers)+

Other Software

likeTest tools

Test Scripts,Makefiles,

Documents (e.g., Design Doc)

Customer’s ConfigurationProducer’s Configuration

Accounts Receivable V1.2

General Ledger V2.1

GL Users Guide V2AR Users Guide V1

Mapping From Customer Configuration to Producer Configuration

Change Management Change Management Configuration Configuration

ManagementManagement

• Componentization and Dependencies• Co-existence of multiple vendors• Distributed teams working simultaneously • Co-existence of multiple versions• Co-existence of multiple “states” (Alpha, Beta…)• Support for multiple platforms

15

SCM = Change Control + Version Control + Movement Control

What is a What is a “Configuration”?“Configuration”?

• A Configuration is a set of (related) items that is:o Identifiableo Consistento Re-creatable

• Configuration can be combinations of:o Hardware and Software Itemso Manual and Computer Based Itemso (We will restrict ourselves to Software Configuration

Management)

16

What is an “Item”?What is an “Item”?• An elementary part of a configuration that must

be:o Identified (“Versioned”)o Trackedo Controlled

17

What is “Configuration What is “Configuration Management”?Management”?

• The art of coordinating software development and changes to the items to minimize confusion• Consists of:

o Institutionalized Procedureso Tools

18

What Items are What Items are Configurable? Configurable?

• See MGSP Book, Chapter 6

19

Configuration Management ProcessConfiguration Management Process

20

Requestor SCCB

Development

Change Request

Change RequestResource Estimates

Evaluation OK to change?

Change Authorization

CompletedChange

Workspace

1

1 2

3

4

NO

YES

4 Communication to Requestor

Check-out

5Check-in

6

7

8

Change ControlRegisterBaseline

Register

Baselining CriteriaBaselining Criteria• Which items need baselining?• Which items do NOT need baselining?• When will you baseline an item?• Who should be the reviewers and approving

authority for each item?• What would be the appropriate review criteria for

evaluating change of the item?

21

Change Review DimensionsChange Review Dimensions• Necessity• Appropriateness• Impact

22

Change Control Change Control MechanismsMechanisms

• Checkin / Checkouto A checkout locks the items, other people wanting the item should wait

till freedo Ensures ACID properties

23

DifficultiesDifficulties inin ManualManual

System System • Difficult to obtain compliance Inexperienced persons lack perspective/motivation Procedures may be cumbersome/irritating

Human error • Require more “training” on procedures• Suffer in case of personnel turn over

27

Automating SCMAutomating SCM• Can be done at various levels• Allows geographical distribution• Often a mix of automated and manual procedures• Compliance is usually better • Saves time• Less chances of error

28

SCM Tools SCM Tools Functionality Functionality CategoriesCategories

• Need for consistency between the vendor’s configuration and user configurationo Making sure the right directory structures existo Ensuring compatibility of file versions

• Some evaluation criteriao Platformso Operationso Extensibilityo Documentation and Reports

29

Requirements From Configuration Requirements From Configuration

Management For Global TeamsManagement For Global Teams• Knowing the recent versions of any file• Facilitating quick creation and testing of a build / patch• Associating a label with any build• Knowing the composition and build of any version• Providing concurrent access to multiple users• Mapping defects in a build to the appropriate source

files• Ability to locate the tests to be run for a given patch /

set of changes• Providing an automated workflow for change

management

30

Configuration Management = Change Control + Version Management

SCM in Global Teams:SCM in Global Teams:Issues and ChallengesIssues and Challenges

• Technologyo Need for a common SCM Repository and toolo Compatibility among the systems across the globeo Speed and dependence on bandwidth

• Processo Consistency of procedures across locationso Ignoring consistent updates to training, documentation

• Peopleo Communication on top of the tools and processo Operational issues of backup time, 24x7

31

SCM In Global Teams:SCM In Global Teams:People IssuesPeople Issues

• Need for consistent policieso Sense of parity required to maintain the spirit of the process

• Need for increased communicationo Automation cannot solve all the problems!

• Strong management support for policieso Should not be viewed as a bureaucratic overhead

32

SCM In Global Teams:SCM In Global Teams:Technology IssuesTechnology Issues

• Choice of tool architectureo Centralized Vs Distributed Vs Replicated

• Dependence of communication linko Everyone needs to have a consistent configuration

• Performance Criticalityo “Intelligent sync up”

• Compatibility across various toolso Data compatibility and meta-data compatibility

33

SCM In Global Teams:SCM In Global Teams:Process IssuesProcess Issues

• Consistent Processes across the teams• Documentation standards• Distributed SCCB and co-ordination

34

Other Environmental Tools Needed in Other Environmental Tools Needed in

Product DevelopmentProduct Development

35

Infrastructure Infrastructure Required ForRequired For

Distributed TeamsDistributed Teams

36

Development andDebugging

Infrastructure

Repos

itorie

s

Repos

itorie

s

Repositories

Repositories

NonComputingInfrastructure

NonComputingInfrastructure

© “Software Maintenance” Gopalaswamy Ramesh and Ramesh Bhattiprolu, Tata McGraw Hill

Components of Components of Development and Development and

Debugging Debugging Infrastructure Infrastructure

37

DebugTools

ConfigMgmtt

RemoteDiagnostic

TestingTools

© “Software Maintenance” Gopalaswamy Ramesh and Ramesh Bhattiprolu, Tata McGraw Hill

Configuration Configuration ManagementManagement

• Getting a single view of o Product sourceso Binaries, Build Tools o Test Scriptso Intermediate Builds / Labelso What was shipped to customers

• Choiceso Centralized Vs Distributed Vs Replicatedo Pros and Cons for each

38

Builder and Test ToolsBuilder and Test Tools• Centralized end of day build• Automatic selection of appropriate regression

tests• Automatic running of regression tests• Automatic analysis of results of regression tests• Associating test results to component owners

39

Debuggers: The Debuggers: The “Stethoscope” For “Stethoscope” For

Engineers!Engineers!Symbolic Debugger

Interrupt-based

System call based

Interpreter based

Kernel-modeUser-mode

Infr

astr

uc t

ure

-bas

ed

Machine-based Debugger

Interrupt-based

System call based

Interpreter based

Kernel-modeUser-mode

Infr

astr

uc t

ure

-bas

ed

Infr

astr

uc t

ure

-bas

ed

40© “Software Maintenance” Gopalaswamy Ramesh and Ramesh Bhattiprolu,

Tata McGraw Hill, 2005

Remote Diagnostics Remote Diagnostics ToolsTools

• Ability to dial into customer machineso Communication infrastructureo Security issueso Firewall issues

• Ability to instrument code and collect diagnostic informationo Trace files, audit files, etc

• Physical Access (if required) issueso Government clearances

41

RepositoriesRepositoriesRefer Maintenance Refer Maintenance

BookBook

42

Development And

DebuggingInfrastructure

DebugTools

ConfigMgmtt

RemoteDiagnostic

TestingTools

Def

ect R

epos

itory

Custo

mer

Rep

osito

ry

Customer Support

Repository

Knowledge Repository

© “Software Maintenance” Gopalaswamy Ramesh and Ramesh Bhattiprolu, Tata McGraw Hill, 2005

43

KnowledgeRepository

CustomerRepository

CustomerSupport

Repository

DefectRepository

Support

Development Customer

Contact Details,Support Eligibility,Supported Configurations

Details of communication with Customer such as:

Problem details, workarounds, solution provided etc.

Reporting the problem, providing required updates, getting current status etc

Known problems,Workarounds,FAQs etc

Problem Details,Updates etc

Known problems,Workarounds,FAQs etc

Communication betweenSupport & Development

Making use of Self Service facility

© “Software Maintenance” Gopalaswamy Ramesh and Ramesh Bhattiprolu, Tata McGraw Hill, 2005




Regression Testing Regression Testing

• Understand typical product release processes• Know the role of configuration management and




44

What is Regression What is Regression Testing?Testing?

• What is meant by “Regression”?• What is meant by “Regression Testing”?• Why do Regression Testing?

45

““Regression”Regression”• “Regress” – to go back, to worsen…• Regression in Software context refers to software

“going back”, i.e., old defects resurfacing in the wake of some changes somewhere

• Regression Testing is a type of testing to catch such “regressed” defects

46

What is Regression What is Regression Testing?Testing?

Regression testing is done to ensure enhancements or defect fixes made to the software work properly and do not affect the existing functionality

47

• Selective Retesting to ensure defect fixes work• Ensure no side effects

How Bugs Can How Bugs Can RegressRegress

48

V1 contains defects D1, D2 and D3; Developers fix & test these fixes

In the process, in V2, they introduce defects D4, D5;D4, D5 fixed and verified; D1, D2, D3 NOT verified

In V3, D1 comes back!




Regression Testing Regression Testing





49

Why do regression Why do regression testing?testing?

• Re-appearing bugs entail unnecessary rework• Intermittent bugs irritate customers• These are avoidable costs

50

Types of Regression Types of Regression TestingTesting

• Based on time when doneo Regular or Build-level regression testingo Final Regression Testing, before the product is shipped out

• Based on extent of testingo Smoke Test / Short Regressiono Full Test / Complete Regression

51

Steps in Regression Steps in Regression TestingTesting

• Performing an initial “Smoke” or “Sanity” test• Understanding the criteria for selecting the test

cases• Classifying the test cases into different priorities• Selecting and running the test cases, based on

the criteria and priorities• Concluding the results of a regression cycle

52

Smoke TestingSmoke Testing• Identifying the basic functionality that a product must

satisfy• Designing test cases to ensure that these basic

functionality work and packaging them into a Smoke Test Suite

• Ensuring that every time a product is built, this suite is run successfully first before anything else is run.

• If this suite fails, escalating to the developers to identify the changes and perhaps change or roll back the changes to a state where the Smoke Test Suite succeeds.

53

Criteria for selecting Criteria for selecting test casestest cases

• Include test cases that have produced maximum defects• Include test cases for a functionality in which a change is

made• Include test cases in which production problems are

reported• Include test cases that test the basic functionalities or the

core features of the product which are mandatory requirements of the customer

• Include test cases that test the end-to-end behavior of the application or the product

• Include test cases to test the positive test conditions• Includes the area which is highly visible to the users

54

Prioritizing Test CasesPrioritizing Test Cases• Based on when the test was last run: These tests

can be run in a round robin manner through the different versions and builds of a product.

• Based on the resources required for running the test: Depending on the availability of resources, a given test may be run or not. The resources include time and machine resources

• Based on functionality that is closer in the next level to the code that is changed

55

Concluding the results Concluding the results of a regression runof a regression run

• How many defects that were reported in the previous cycle got fixed in this cycle?o Good!

• How many defects that were reported in the previous cycle did not get fixed in this cycle?o May imply that development did not fix the defects correctly

• How many test cases failed anew in the current cycle?o Indicates side effectso good candidates to continue to have in the future regressions

• How many test cases have worked with a “workaround”, but no real code fix?o The workaround needs to be institutionalized into a code fix

56

57

Integrating Regression Testing with the environment:TCDB

58

TCDB

Test CaseProduct XREF

Test CaseInfo

Test CaseDefect XREF

Test CaseRun Info

SCM

EnvironmentFiles

ProductTest Cases

ProductSource Code

ProductDocumen-

tation

DR

Defect Fix Details

DefectDetails

Defect Communi-

cationDefect Test

Details

Typical Contents of Typical Contents of TCDB: Table 3 of TCDB: Table 3 of

HandoutHandout

59

Challenges in Challenges in Regression TestingRegression Testing

• What do I test?o Effective prioritization and selection criteriao Automatic integration to SCM repository

• Possible Continuous Growth of Tests to be performedo A policy for “retiring” regression tests (e.g., defect yield of a test)

60

Today’s Agenda:Today’s Agenda:

RecapRecap





61

07 regression-testing-and-scm

Software

release version

customer configuration

producer configuration

terminology o

o hardware

versioned o

software testing

change control