Dynamic Software Tracking

J. Erik HemdalRobert L. Galen

BOSCON 2004 – Track C4

Agenda

● Introduction

● Project Goals

● Measures from Defect Data

● Triage and Workflow

● Conclusions

● Questions and Answers

Why are we here?

● Discuss the Big Four elements you need to control for a successful project

● Examine a variety of defect-data measures– For what they tell us– For how they can be defeated

● Show how you can manage defects so that you can stay in control of your project and workload – especially in the endgame close to release points.

The Big Four Elements

● Have stable, agreed-on goals

● Know the quality level and quality trends of your software

● Keep your team healthy

● Always know how much work remains and how much time is left

Understanding & Influencing the Goals

● The formal requirements

● Unstated requirements

● Expectations from customer, sponsor, team

● Negotiation and balance are the key– Features– Specific delivery timing– Level of quality (Hot-button issues)– Essential vs. “nice-to-have” elements

Understanding & Influencing the Goals (cont.)

Understanding the overall project plan– Development phasing (building to features, stability and

completeness)– Understanding the methodology (waterfall, incremental,

agile/extreme, RUP)– Test phasing (how many passes, reduction of change,

entrance & exit criteria)

● Key Milestones– Delivered phases– Code freeze / complete– Beta tests– Final release

Understanding & Influencing the Goals (cont.)

● Release Criteria is the critical guide for releasing software. Should cover –– Scope– Quality– Time– Team

● Good release criteria should1. Define success2. Learn what’s important for the project3. Be Drafted and thoroughly reviewed4. Be “SMART” (specific, measurable, attainable, realistic

and trackable)5. Assist in gaining consensus6. Serve as a goal / guide throughout triage and release

Software Measures from Defect Data

● Good defect tracking provides insight– Into the quality of the product– Into the performance and health of the team– Into the amount of remaining work and rework

● Can be a hot-button issue because of misuse

● New way of thinking: A “Defect” means “something to change”. More things to change means more work.

● Changes stand between you and Done!

Common Defect Measures

Cost per defect, predict development repair efficiency

Fix hours / defect

Defect workflow in time, potential bottlenecks

Defects by state counts

Root cause, true cost of qualityPhase –containment of defects

Determine overall maturity - release / ship readiness

# high – medium – low severity defects

Determine overall maturity - release / ship readiness

# found, # fixed, # remaining over time

Time distribution of defects, looking for downward trends

Defect count over time

Testing workflow and productivityDefects / Unit of testing time

Defect density, predictor for future defect trends and overall product quality

Defect counts or defects / KLOC

IllustratesMeasure

Defects/KLOC

● Uses: ● Indicate overall quality of

code● Guide to trouble spots

● Depends on:● Definition of a KLOC● Modular architecture with

visible granularity

● Defeat by:● Attacking the KLOC● Blame requirements● Writing longer code

E A C D B AVG0

20

40

60

80

100

120

140

Defects per KLOC by Module

Module ID

Def

ect

Den

sity

Defects/Unit of Test Time

● Uses: ● Overall level of test “productivity”● Check test strategies or test “wearout”

● Depends on:● Factors testers usually don't control

● Defeat by:● Untestable/unreachable code● Cutting test time

● Graphical display – similar to Defects/KLOC

Defect Counts over Time

● Uses:● Adjust test sequencing and

scheduling● Can indicate significant

problems in test

● Depends on:● Overall coordination

● Defeat by:● Interrupting testing● Finding significant defects● Missing functionality

1 2 3 4 5 6 7 80

5

10

15

20

25

30

35

40

45

New Defects by Week

Week Number

De

fect

Co

un

t

# Found, # Fixed and # Open

● Uses: • Suggests when to ship

● Depends on:● Reliable, repeatable test

capability● Change control

● Defeat by:● Curtailing test● Many small changes● Late changes

1 2 3 4 5 6 7 8 9 10 11 120

20

40

60

80

100

120

Found vs. Fixed Defects

New

Closed

Cum-New

Cum-Closed

Week Number

De

fect

Co

un

t

# High/Med/Low Severity Defects

● Use:● Indicates overall readiness of a

product

● Depends on:● Proper triage● Effective criteria

● Defeat by:● Management fiat● Subtle negotiation● Peer pressure

1 2 3 4 50

10

20

30

40

50

60

70

Defect Counts by Class

Class CClass BClass A

Week Number

De

fect

Co

un

t

Phase-Containment of Defects

● Uses:● Identify process problems● Justify process improvement

● Depends on:● A defined phase-gate

process

● Defeat by:● Lack of commitment to the

process● Lack of time to analyze raw

defect data

Reqts Design Code Test Int'gn0

5

10

15

20

25

30

35

40

45

When Found vs. When Caused

Reqts.DesignCodeTestIntegration

Phase When Found

Def

ect C

ount

per

Pha

se W

hen

Cau

sed

Average Fix Hours per Defect

● Uses:● Illustrate the cost of poor quality● Provide data for repair estimates

● Depends on:● Ability to capture data● High trust culture and within the development team

● Defeat by:● Misusing the data

● Similar measures possible for build, test, review time

Defects By Status Histogram

● Uses:• Show team bottlenecks• Gauge project status and

product maturity

● Depends on:• Consistent and reliable state

update activity

● Defeat by:• Gamesmanship about defect

status

1 2 3 40

2

4

6

8

10

12

14

16

18

20

Defects by State

NewAssignedOpenClosed

Week Number

Co

un

t of D

efe

cts

Dynamic Tracking

● Defect Triage

● Workflow Management

● Defect Packaging

Defect Triage

● Manage three elements– Severity How serious is this defect?– Priority How soon should this be fixed?– Impact What is the effect on team and customer?

● Define a triage team and procedure early– Include QA, developers, project office, support– Set up the “drill”– Update defect reports with triage decisions – who, what,

why

● Map triage policy to the overall release plan

Workflow Management

● Don’t just let work happen based on priority or chance - rather, proactively manage repairs!

● Pre analyze scope, level of difficulty and potential impact – prior to assigning major repairs

● Each engineer has an assigned “work queue” or a to-do list that is managed from the defect tracking system

● Leverage the DTS for reporting and work flow management

Workflow Management (cont.)

● Create guidelines per engineer 5-10 work items– 1 or 2 high priority defects– 3-4 moderate priority defects– 1-2 defects to investigate/analyze

● Good idea: Focus on your “Top 10” issues, then stop and analyze

● Reallocate judiciously; avoid churning

● Create engineer profiles to help understand strengths and skills

Defect Packaging

● Schedule a series of code drops or “packages” for testing– First release– Updates; new functions– “Critical fix” package– Release candidate– Final release

● Each should have a primary goal

● Testing involves high fixed costs; packages give you the most value (fixed defects) in each cycle

● Don't waste the “power of the package”

Tying Things Together

● Supporting the Goals

● Maintaining the Level of Quality

● Watching the Health of the Team

● Knowing What's Left to be Done

Supporting the Goals

● Effective defect triage helps you to maintain your focus on the goals of the project.

● Triage directs your team's effort to the most important work, balancing the demands of all stakeholders

● Defect data can flag roadblocks and slowdowns before they derail the project

Maintaining the Level of Quality

● Defect measures and reports can give you a direct reading on the state of your project

– How many defects there are

– How defects affect the product and project

– Where extra design, test, or review effort is warranted

– When it's appropriate to release (or Not to release)

Watching the Health of the Team

● Defect data helps here by -

– Signaling overloaded and frustrated team members

– Capturing and documenting key decisions and key changes

– Communicating changes that WILL NOT be made

– Preventing unnecessary work

Knowing What's Left to be Done

● Defect tracking data helps here by indicating -

– How many items must be changed

– How long will these changes take

– How much will they cost

– What will be unfinished when we stop

Questions?

References

● A. Allison, "Meaningful Metrics" The Software Testing & Quality Engineering Magazine (May/Jun 2001)

● R. Black, Managing the Testing Process, 2’nd Edition, New York, NY: John Wiley & Sons Inc., 2002

● R. Galen, “Mastering the Software Project Endgame”, New York, NY: Dorset House, 2004 - forthcoming

● C. Necaise, "Managing the Endgame." The Software Testing & Quality Engineering Magazine (Jan/Feb 2000)

● J. Rothman, "Release Criteria: Is This Software Done?" The Software Testing & Quality Engineering Magazine (Mar/Apr 2002)

● J. Rothman, "Managing Projects: Release Criteria, or Is it Ready to Ship." Newsletter Vol. 1, No. 2 (1999)

● B. Schoor, “Managing Quality During the Endgame” Presentation from schoorconsulting.com website and PSQT/PSTT 2002 North conference – www.softdim.com (2002)

Contact Information

Erik HemdalIndependent Consultantand Instructor

ehemdal@brandeis.edu

Bob GalenEMC2 Corp. & RGalen Consulting

Group, LLC

galen_bob@emc.com

bob@rgalen.comwww.rgalen.com