SSoftware oftware QQuality uality AAssurancessurance

Part OnePart One

Reviews and InspectionsReviews and Inspections

Development PlanDevelopment Planthat includes Quality Assurance

RequirementsSpecification

Design

Coding

UnitTesting

Installation &Training

IntegrationTesting

Maintenance

Review the SRS

Design Reviews

CodingStandards

Validation

Co

nfig

uratio

n C

on

trol

Test Proceduresand Tolerances

Do

cum

entatio

n

Defect T

racking

Code Reviews

Types of EvaluationsTypes of Evaluations

VerificationVerification Unit Test, Integration Test, Usability Test, etc

Formal ReviewsFormal Reviews aka "formal design review", "formal technical reviews", etc.

conducted by senior personnel or outside experts uncover potential problems

Peer ReviewsPeer Reviews aka "inspections", "walkthroughs", etc.

done by peers detect errors, adherence to standards, etc.

Contract Review ProcessContract Review Process

RFP

1st Draft

Revisions

Final ContractFinal Contract

SOW

Contract

Reality Check...

Q: Why should the software geeks worry about the contract?

A: Because the software team must do the work and assure the product's quality. loosely defined requirements unrealistic budgets unrealistic schedules

A: Contract review is required by ISO 9001

Formal Reviews

Reviewers should be senior personnel and/or outside experts

Review Leader should not be Project Leader

Usually done at the end of a phase. very appropriate for SRS and Design rarely appropriate for code

Outcome: approve approve pending changes reject

Software Quality Assurance by Galinsection 8.2

Sample Checklist

Formal Review of a Design Adequate Well-structured

Simple

Efficient

Flexible

Practical

Implementable

General:1. Does the architecture convey a clear vision of the system that can be used for

further development?2. Is the architecture structured to support likely changes?3. Does the architecture describe the system at a high level of detail? (No

interface or implementation details.)4. Does the architecture cleanly decompose the system?5. Is the architecture independent of the infrastructure used to develop the

system?6. Has maintainability been considered?7. No duplicate functionality in the architecture? Complete:1. Are software requirements reflected in the software architecture? 2. Is effective modularity achieved? Are modules functionally independent? 3. Does each module/class have an understandable name?4. Is each association well named?5. Is each association’s and aggregation’s cardinality correct? Correct:1. Does each association reflect a relationship that exists over the lives of the

related modules/classes?2. Does the architecture have loose coupling and good cohesion?

www.cs.trincoll.edu/~hellis2/CPSC240/Project/Design Review Checklist.doc

Peer Reviews / Inspections / Walkthroughs

guided by: checklists, standards, past problems

attendees: review leader the author scribe 1 or 2 people with domain knowledge possibly an SQA team member (for standards)

Galin section 8.3

Why schedule a meeting with so many people? Why not just have two people review the item without a meeting?

Inspection Process

1. pre-meeting review the item ahead of time

2. meeting author presents overview review team asks questions and express opinions

3. after meeting scribe prepares summary team approves summary follow up

Inspection Guidelines

Review the Product, not the person! Find errors, don't try to solve them! Keep Records

Take written notes. Review your earlier reviews.

Allocate resources and schedule time for FTRs. 3 to 5 people Conduct training for reviewers

Keep it short limit debate and rebuttal

Set an agenda and keep it. no more than two hours preparation

small portions only narrow focus increases likelihood of finding an error

meeting duration less than one hour

Sample Design Inspection

1. Does the algorithm accomplishes desired function?

2. Is the algorithm logically correct?

3. Is the interface consistent with architectural design?

4. Is the logical complexity reasonable?

5. Have error handling and "anti-bugging" been specified?

6. Are local data structures properly defined?

7. Are structured programming constructs used throughout?

8. Is design detail amenable to implementation language?

9. Which are used: operating system or language dependent features?

10. Is compound or inverse logic used?

11. Has maintainability been considered?

stolen from Pressman

IBM (relative costs) during design $1.5 prior to coding $1 during coding $1.5 during test $60 in field use $100

Reviews are 2 or 3 times as efficient as testing at finding defects.

Combined design and code reviews have a yield of 60% to 80%.

Effectiveness of ReviewsEffectiveness of Reviews

Without QA ReviewsWithout QA Reviews 50 KLOC with 2500 defects to be found average 4 hours of work per defect 10,000 hours 10,000 hours to remove defects

With QA ReviewsWith QA Reviews 50 KLOC with 2500 defects 70% (1750) of defects removed via Reviews

average 0.5 hours per defect 875 hours of work

remaining 750 defects average 8 hours each 6000 hours

Total = 6,875 hours6,875 hours

ExampleExample

Real NumbersCost of Software Quality for 15 Projects at Raytheon’s Equipment Division

0

10

20

30

40

50

60

70

Pe

rce

nta

ge

of to

tal p

roje

ct co

st

Year

CMM level 3Start of intiative CMM level 1

TCoSQ

PreventionRework

Appraisal

Cost ofConformance

Rework

87 88 89 90 91 92 93 94 95 96

htt

p:/

/ww

w2

.um

ass

d.e

du

/sw

pi/c

ost

mo

de

ling

/pa

pe

rs/s

coq

pa

p1

.do

c

How much SQA is cost effective?

Software Quality

Costs

InitialCost of SQA

Cost of Failure

SQA+

Failure

Optimal Quality Level

EventualEventualCost of SQACost of SQA

ExamplesExamples

NASA's Software Review Guidelines

Case : Software Review What went wrong? How would you fix the problem(s)?