dynamic software tracking

Download Dynamic Software Tracking

Post on 11-Jan-2016




0 download

Embed Size (px)


J. Erik Hemdal Robert L. Galen BOSCON 2004 Track C4. Dynamic Software Tracking. Agenda. Introduction Project Goals Measures from Defect Data Triage and Workflow Conclusions Questions and Answers. Why are we here?. - PowerPoint PPT Presentation


  • Dynamic Software TrackingJ. Erik HemdalRobert L. GalenBOSCON 2004 Track C4

  • AgendaIntroduction

    Project Goals

    Measures from Defect Data

    Triage and Workflow


    Questions and Answers

  • Why are we here?Discuss the Big Four elements you need to control for a successful project

    Examine a variety of defect-data measuresFor what they tell usFor how they can be defeated

    Show how you can manage defects so that you can stay in control of your project and workload especially in the endgame close to release points.

  • The Big Four ElementsHave stable, agreed-on goals

    Know the quality level and quality trends of your software

    Keep your team healthy

    Always know how much work remains and how much time is left

  • Understanding & Influencing the GoalsThe formal requirements

    Unstated requirements

    Expectations from customer, sponsor, team

    Negotiation and balance are the keyFeaturesSpecific delivery timingLevel of quality (Hot-button issues)Essential vs. nice-to-have elements

  • Understanding & Influencing the Goals (cont.)Understanding the overall project planDevelopment phasing (building to features, stability and completeness)Understanding the methodology (waterfall, incremental, agile/extreme, RUP)Test phasing (how many passes, reduction of change, entrance & exit criteria)

    Key MilestonesDelivered phasesCode freeze / completeBeta testsFinal release

  • Understanding & Influencing the Goals (cont.)Release Criteria is the critical guide for releasing software. Should cover ScopeQualityTimeTeam

    Good release criteria shouldDefine successLearn whats important for the projectBe Drafted and thoroughly reviewedBe SMART (specific, measurable, attainable, realistic and trackable)Assist in gaining consensusServe as a goal / guide throughout triage and release

  • Software Measures from Defect DataGood defect tracking provides insightInto the quality of the productInto the performance and health of the teamInto the amount of remaining work and rework

    Can be a hot-button issue because of misuse

    New way of thinking: A Defect means something to change. More things to change means more work.

    Changes stand between you and Done!

  • Common Defect Measures

  • Defects/KLOCUses: Indicate overall quality of codeGuide to trouble spots

    Depends on:Definition of a KLOCModular architecture with visible granularity

    Defeat by:Attacking the KLOCBlame requirementsWriting longer code

  • Defects/Unit of Test TimeUses: Overall level of test productivityCheck test strategies or test wearout

    Depends on:Factors testers usually don't control

    Defeat by:Untestable/unreachable codeCutting test time

    Graphical display similar to Defects/KLOC

  • Defect Counts over TimeUses:Adjust test sequencing and schedulingCan indicate significant problems in test

    Depends on:Overall coordination

    Defeat by:Interrupting testingFinding significant defectsMissing functionality

  • # Found, # Fixed and # OpenUses: Suggests when to ship

    Depends on:Reliable, repeatable test capabilityChange control

    Defeat by:Curtailing testMany small changesLate changes

  • # High/Med/Low Severity DefectsUse:Indicates overall readiness of a product

    Depends on:Proper triageEffective criteria

    Defeat by:Management fiatSubtle negotiationPeer pressure

  • Phase-Containment of DefectsUses:Identify process problemsJustify process improvement

    Depends on: A defined phase-gate process

    Defeat by:Lack of commitment to the processLack of time to analyze raw defect data

  • Average Fix Hours per DefectUses:Illustrate the cost of poor qualityProvide data for repair estimates

    Depends on:Ability to capture dataHigh trust culture and within the development team

    Defeat by:Misusing the data

    Similar measures possible for build, test, review time

  • Defects By Status HistogramUses:Show team bottlenecksGauge project status and product maturity

    Depends on:Consistent and reliable state update activity

    Defeat by:Gamesmanship about defect status

  • Dynamic TrackingDefect Triage

    Workflow Management

    Defect Packaging

  • Defect TriageManage three elementsSeverity How serious is this defect?Priority How soon should this be fixed?Impact What is the effect on team and customer?

    Define a triage team and procedure earlyInclude QA, developers, project office, supportSet up the drillUpdate defect reports with triage decisions who, what, why

    Map triage policy to the overall release plan

  • Workflow ManagementDont just let work happen based on priority or chance - rather, proactively manage repairs!

    Pre analyze scope, level of difficulty and potential impact prior to assigning major repairs

    Each engineer has an assigned work queue or a to-do list that is managed from the defect tracking system

    Leverage the DTS for reporting and work flow management

  • Workflow Management (cont.)Create guidelines per engineer 5-10 work items1 or 2 high priority defects3-4 moderate priority defects1-2 defects to investigate/analyze

    Good idea: Focus on your Top 10 issues, then stop and analyze

    Reallocate judiciously; avoid churning

    Create engineer profiles to help understand strengths and skills

  • Defect PackagingSchedule a series of code drops or packages for testingFirst releaseUpdates; new functionsCritical fix packageRelease candidateFinal release

    Each should have a primary goal

    Testing involves high fixed costs; packages give you the most value (fixed defects) in each cycle

    Don't waste the power of the package

  • Tying Things TogetherSupporting the Goals

    Maintaining the Level of Quality

    Watching the Health of the Team

    Knowing What's Left to be Done

  • Supporting the GoalsEffective defect triage helps you to maintain your focus on the goals of the project.

    Triage directs your team's effort to the most important work, balancing the demands of all stakeholders

    Defect data can flag roadblocks and slowdowns before they derail the project

  • Maintaining the Level of QualityDefect measures and reports can give you a direct reading on the state of your project

    How many defects there are

    How defects affect the product and project

    Where extra design, test, or review effort is warranted

    When it's appropriate to release (or Not to release)

  • Watching the Health of the TeamDefect data helps here by -

    Signaling overloaded and frustrated team members

    Capturing and documenting key decisions and key changes

    Communicating changes that WILL NOT be made

    Preventing unnecessary work

  • Knowing What's Left to be DoneDefect tracking data helps here by indicating -

    How many items must be changed

    How long will these changes take

    How much will they cost

    What will be unfinished when we stop

  • Questions?

  • ReferencesA. Allison, "Meaningful Metrics" The Software Testing & Quality Engineering Magazine (May/Jun 2001)R. Black, Managing the Testing Process, 2nd Edition, New York, NY: John Wiley & Sons Inc., 2002R. Galen, Mastering the Software Project Endgame, New York, NY: Dorset House, 2004 - forthcomingC. Necaise, "Managing the Endgame." The Software Testing & Quality Engineering Magazine (Jan/Feb 2000)J. Rothman, "Release Criteria: Is This Software Done?" The Software Testing & Quality Engineering Magazine (Mar/Apr 2002)J. Rothman, "Managing Projects: Release Criteria, or Is it Ready to Ship." Newsletter Vol. 1, No. 2 (1999)B. Schoor, Managing Quality During the Endgame Presentation from schoorconsulting.com website and PSQT/PSTT 2002 North conference www.softdim.com (2002)

  • Contact InformationErik HemdalIndependent Consultantand Instructor


    Bob GalenEMC2 Corp. & RGalen Consulting Group, LLC



    Outline for our tag-teaming the presentation:Bob does the introduction and goals coverage (slide 1-7)Erik then focuses on the metrics parts (slides 8-17)Bob comes back to discuss the triage parts (slides 18-22)Erik then wraps-up the presentation with (slides 23-28) and we both do Q&ADefect data can be the key to all four of these elements:

    Goals/Requirements: these govern what you will consider to be a defectQuality level and trends: How many issues do you need to fix? How good is the software? Is the software improving in functionality/performance/reliability or are you losing ground?

    Team health: Are engineers overworked? Is the test team burned out? Is the quality of your product at the mercy of project politics?

    And the big one: What work is left to be done? Who is doing it? When will it be done? What risks or surprises might derail your project?

    There are a variety of measures you can derive from a DTS that can help you answer all these questions. But you can become misled by what the DTS tells you.

    A well run project will have documented requirements. But there are often unstated requirements because stakeholders believe they do not need to be stated or are afraid to state them.

    Requirements tend to hide the negotiation necessary for an effective project effort.

    Negotiation involves a LOT: features, quality level, delivery dates, throwaways. Negotiation on quality level is a sensitive topic because few are comfortable making explicit tradeoffs about quality..This is the classic measure, used for decades. It is a simple measure to calculate