John Fodeh - Spend Wisely, Test Well

Download John Fodeh - Spend Wisely, Test Well

Post on 15-Jul-2015




0 download


<p>INSERT PRESENTATION TITLE HERE</p> <p>SPEND WISELY, TEST WELLMaking a Financial Case for TestingHP - Global Testing PracticeJohn 2009 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice 11Introductions</p> <p>22Originally presented as a day tutorial by Susan Herrick, EDS US at StarEAST and StarWEST in 2008 &amp; 2009About You (and Me)Briefly introduce yourself by telling us:Your name</p> <p>Your testing role</p> <p>How long youve been in the testing arena</p> <p>One thing youd like to get out of this Workshop33This is a workshop, so dont get to comfortable, you are expected to participate, lets start with some details????About This SessionAfter youve completed this workshop, you should be able to:Address and even turn around the prevailing view of testing among business stakeholdersThrough the effective use of industry data:Establish and/or reinforce your credibilitySubstantiate your testing recommendations and strategiesDemonstrate to business stakeholders in terms they can understand and accept the relative cost of testing and not testing for a specific project</p> <p>Credits: Ian Howles &amp; Susan Herrick!</p> <p>44AdministriviaThe duration of this workshop is 1.5 hours</p> <p>There will be no breaks</p> <p>Ask questions whenever they occur to you</p> <p>55Session uses a simple waterfall approach but is applicable to all lifecycles models.Its All About Communication</p> <p>66Remember, testing is a service organization.</p> <p>Our mission is to provide information about quality and risk to decision makers within our organization.TestingOur Ideal Organization ChartThe Remainder of the OrganizationForgetting the Organizational ContextFrom The Nine Forgettings by Lee Copeland77why we exist and who we serve </p> <p>Testing does not create quality. All we do is look for it.</p> <p>Forgetting the Organizational ContextTesting should not be the Ship / Dont Ship decision maker. That is not our stewardship.88upside down and/punishment mechanisms Wally Im going to write me a minivan</p> <p>The Business SpectrumMass-market packaged softwareMarket-dominating softwareNiche softwareHigh consumer toleranceCustom softwareMarket-entry softwareCompetitive softwareLow consumer toleranceRegulatory complianceCritical safety requirementsDefect ToleranceHighLow</p> <p>Perfection is the enemy of the good- Voltaire99Cost of QualityQuality PerspectiveC Quality = C Conformance + C Non-Conformance</p> <p>WHEREC Quality = Cost of Quality C Conformance = Cost of ConformanceCost of preventionCost of detectionCost of internal failure</p> <p>C Non-Conformance = Cost of Non-ConformanceCost of external failureOther related costs</p> <p>1010Cost of QualityBusiness PerspectivePV C Quality = NV</p> <p>WHEREPV = Potential Value, e.g., ALL sources of revenue associated with a software product, annualized cost reduction that results from productivity improvementsNV = Net Value, e.g., actual revenue for the software product, actual annualized cost reduction</p> <p>AND</p> <p>PV= PM or ROI (% + or -)NV</p> <p>WHEREPM = Profit Margin ROI = Return on Investment </p> <p>1111Its All Greek to Me!The language of Testing focuses on: Quality Defects Test coverage RiskThe language of Business focuses on: Scope Time Money Risk (and, oh, yes, Quality)</p> <p>1212Bridging the Communication GapThe decision-makers will tell you they care about quality and customer satisfaction but not if they believe that achieving either will adversely affect their bottom lineTalking about defect discovery alone isnt enoughTesting has to be able to:Demonstrate the financial consequences of testing involvement or lack thereof in a projectSubstantiate the financial consequences with reliable data</p> <p>1313For example. Ive selected Profit Margin and/or ROI because they are commonly used terms.However, YOU NEED TO HAVE A SOUND UNDERSTANDING OF WHATEVER LANGUAGE YOUR COMPANY WORKS IN. DO THEY USE INTERNAL RATE OF RETURN, PAYBACK PERIOD, TIME TO VALUE? SOMETIMES BUSINESS LEADERS DON'T ALWAYS SYNC UP TO THE VALUE LANGUAGE OF THE COMPANY. IF CAPITAL IS INVOLVED, YOU NEED TO UNDERSTAND THE PROCESS YOUR FINANCE DEPARTMENT USES TO APPROVE THE BUDGET AND GET IT INTO THE LANGUAGE THEY SPEAK.Business case assumptions must be thoughtful and clearly supported in terms a CFO will understand, Show Me the MoneyTesting requires financial investment in:</p> <p>Resources, tools, equipment</p> <p>Time to develop, test, manage</p> <p>Escaped defects</p> <p>Rework and retesting</p> <p>Business stakeholders legitimately expect a healthy return on that investment</p> <p>1414Defining the Communication ChallengeLets quickly discuss some views of testing that:</p> <p>You hear from decision-makers in your organization and/or on your projects</p> <p>You would like to change</p> <p>1515Prevailing View vs. Desired ViewTesting is a back-end project phase</p> <p>Testing is easy and anyone cando it</p> <p>Significant rework is inevitable</p> <p>Testing provides advice that can be selectively ignored</p> <p>Testing creates bottlenecks and increases the risk of schedule overruns</p> <p>Testing costs too muchTesting is an integral aspect of all phases in the product life cycle</p> <p>Effective and efficient testing requires skilled testing professionals</p> <p>Focusing on early defect discovery and prevention can significantly reduce rework</p> <p>Testing provides information that quantitatively specifies outstanding risk, application readiness and potential cost ignore it at your peril!</p> <p>Testing early and testing often can actually reduce the risk of schedule overruns and accelerate time-to-market</p> <p>Not testing costs way too much in rework (pre- and post-release), failures, and production support</p> <p>1616Exercise:Turning a Problem into a SolutionThe following exercise is derived from a tutorial developed by Steve Bender of The Quality Connection (</p> <p>Effective Methods to Develop Creative Approaches to Todays Challenges</p> <p>The exercise requires you to:Define a problemDefine a solution that is:Stated in positive termsUnder your controlIntegratedTestable</p> <p>1717ExerciseStep 1 - Defining the ProblemTake 5 minutes to answer these questions. Whats the problem?How long has this problem been a problem?Why does this problem exist?How are you and/or your team limited by this problem?What is the worst thing this problem has caused?</p> <p>1818ExerciseStep 2 - Defining the SolutionTake 5 minutes to answer these questions. What do you want? Be as specific as possible.How will you know when you get it?Where, when and with whom do you want it?When you get it, what else will improve?What resources do you already have that will help you get it?What is the first step to take to get it?</p> <p>1919Possibly in groups, depends on the audience size, could be a collaborative exercise?Defining the ProblemAn ExampleWhats the problem?Perception that testing costs more than its worthHow long has this problem been a problem?For as long as I can remember (15+ years)Why does this problem exist?No real proof to the contraryTesting world does not speak the language of businessHow are you and/or your team limited by this problem?Inadequate allocation of time, money, resourcesMarginal influence over allocation decisionsLack of senior management supportWhat is the worst thing this problem has caused?5-10% of project effort allocated to testingUnnecessary schedule and cost over-runsDefective software in production</p> <p>2020Example from Susan HerrickDefining the SolutionAn ExampleWhat do you want? Be as specific as possible. Real, tangible proof that the decision-makers get itHow will you know when you get it?Commitment through actionProjects - Upfront testing involvement in planning, scheduling, impact analysisSenior management Testing Policy and StrategyWhere, when and with whom do you want it?Senior management and project purse string holdersWhen you get it, what else will improve?Testing involvement and influenceProduct quality, cost, speed to marketWhat resources do you already have that will help you get it?Industry dataAccess to audienceWhat is the first step to take to get it?Pull together the dataFilter and home in on compelling proof</p> <p>2121Communication PrincipleThe meaning and merit of the communication is in the response it elicits</p> <p>2222The Proof is in the Data</p> <p>2323First a couple of quotes:</p> <p>Demming</p> <p>CarrolIm from Missouri show meIn God we trust all others bring data</p> <p> - W. Edward Deming</p> <p>2424A Common Modus OperandiIf you dont know where you are going, any road will get you there - Lewis Carroll</p> <p>2525An Alternative PerspectiveIf you dont know where youre going, how willyou know when youve arrived?</p> <p>OR</p> <p>If you dont know what testing or not testing is really going to cost you, how can you know if youve spent wisely?</p> <p>2626Show Me the ValueTwo axioms of management are:What gets measured gets doneWhat gets valued gets funded</p> <p>So, it's important to remember that testing delivers value that we can and should measure- Rex Black</p> <p>2727Note on ROI:In todays economic climate, the word investment is probably anathema to a lot of organizations. An alternative to the term ROI could be Return on Expenses or, even better, Returned Value.</p> <p>From Hung Q, Nguyen Chairman and CEO, LogiGear Corp.Addressing the ChallengeFocus on the financial impact of defect resolution through testing activities throughout the product life cycleUse the following industry data to substantiate claims about the financial impact of:Defect Insertion RateAmplification FactorDefect Removal EffectivenessTime to Resolve a DefectCost of Defect ResolutionRework RateAddress the assumption that automation solves the problemDemonstrate the cost of schedule compression and inadequate testing staff (numbers and experience)Quantify the cost and cost avoidance associated with testing or not testing2828Some DefinitionsDefect insertion rate defines the defects introduced in a project phase as a % of the defects discovered throughout the product lifecycleAmplification factor is based on the premise that every defect introduced in a project phase that remains undiscovered migrates to the next project phase AND introduces additional defectsDefect removal effectiveness is a quantitative measure of the effectiveness with which each discrete testing activity finds defects and supports their resolution</p> <p>2929Some More DefinitionsTime to resolve a defect is based on the time to discover and remove one (1) defect by project phase in the product life cycleCost of defect resolution is based on the cost of discovering and removing one (1) defect by project phase in the product lifecycleRework rate is the effort associated with fixing and retesting defects introduced in a project phase as a % of overall effort associated with fixing the defects discovered throughout the product lifecycle</p> <p>3030Defect Insertion Rate</p> <p>56% of defects are introduced in the requirements phase of a project.From James Martin</p> <p>3131Amplification FactorRequirements1 defect 1.5 defects in Design</p> <p>Design1 defect 1.25 defects in CodeFrom What IT Managers Should Know About Testing ROI by Rex Black</p> <p>3232In addition, Capers Jones contends that bad fixes inserted as a result of defects found during dynamic testing (e.g., System-Level, Acceptance Testing) and even Production Support Testing can result in the introduction of additional defects at a rate of anywhere from 1 20%. Well use an average of 10% for our calculations.These bad fixes also create a nesting effect, because they block the discovery of latent defects by blocking the execution of tests.Defect Removal EffectivenessEach discrete testing activity can detect and support resolution of 30% of latent defects:</p> <p>Defects that escaped from the previous testing activity</p> <p>AND</p> <p>Defects that were introduced during the project phase associated with the current testing activity</p> <p>A latent defect is a defect that is present and capable of becoming visible, obvious, active, or symptomatic From Capers Jones</p> <p>333350100200500RequirementsDesignCodeSystem TestingAcceptance TestingOperation &amp;Maintenance1Cost of Defect ResolutionIncremental CostsProduct Life Cycle StageFrom Boehm &amp; Baziuk, Standish Chaos Report</p> <p>3434On large, complex projects, 50-70% of project effort is devoted to rework an avoidable costRework is the root cause of project bottlenecks, schedule delays and rising costsTime to Resolve a Defect</p> <p>3535Rework Rate</p> <p>82% of effort in fixing defects is caused by inadequate requirements.</p> <p>From James Martin3636Time is MoneyDesign ReviewCode InspectionTesting8.44 hours1.38 hours0.17 hoursCollofello and Woodfield (1989) For every hour spent on37The data in this study stemmed from a large real-time software project, consisting of about 700 000 lines of code developed by over 400 people. </p> <p>The time saved by the process is the time that would have been spent if the process had not been performed and faults had to be corrected later. The time-saving results found in this study are shown here. These results indicate that, for every hour spent in design reviews and correcting design faults, more than eight hours of work are saved. The time benefits from the testing phase itself are remarkably low. This is not really surprising, since much time is wasted during the actual testing phase in performing tests that do not reveal any faults. These findings once more confirm the statement that early testing really pays off. Remember time is money.</p> <p>As an aside The figure of 8 hours saved for every hour invested is also bourne out in Inspection Dot Grahams Inspection book (page 315) gives a study done of 1000 defects by Trevor Reeve, who found it saved 9.3 hours on average, so 8 hours saved (subtracting the hour invested and rounding down).</p> <p>Putting it all Together</p> <p>3838Putting the Data to WorkEach of these data sources provides thought-provoking insight, but not enough to make a compelling, quantitative case for testing or not testingPulling the data together can help your business stakeholders understand:Whether testing will really cost more than not testingHow much and what types of testing will reduce cost and maximize value through profit and/or return on investment (ROI)</p> <p>3939Ask them to open up Time, Cost and Value CalculatorConsiders both pre-release and post-release activitiesClassifies testing activities as:</p> <p>Static TestingTesting of a component or system at the specification or implementation level without execution of that software, e.g. reviews or static code analysis</p> <p>Dynamic TestingTesting that involves the execution of the software of a component or system.From Standard Glossary of terms Used In Software Testing, v 2.0 - ISTQB</p> <p>4040Time, Cost and Value CalculatorStatic TestingTesting CategoryPotential # of Discrete Testing ActivitiesTypically Includes:Requirements Verification2Ambiguity and Risk AnalysisContent ReviewDesign Verification2High-Level Design ReviewDetailed Design / Prototype ReviewCode Verification2Code inspection (manual)Code com...</p>