How to Select a Static Analysis Tool

2011

Agenda for this session

Define static analysisLayout strategy for evaluating and choosing a static analysis tool that will actually work in the fieldList possible evaluation criteria

Parasoft Proprietary and Confidential

Parasoft Proprietary and Confidential

About Parasoft

Founded in 198727 Patents for automated quality processesBuild quality into the processStatic Analysis tools since 1994

Parasoft Capabilities

TechnologiesQuality Policy ManagementTask Management Code Analysis – Pattern BasedCode Analysis – Flow BasedCode Analysis - MetricsCode ReviewUnit Testing FrameworkMemory Error DetectionRuntime AnalysisMessage/Protocol TestingApplication Behavior EmulationFunctional TestingLoad Testing

Parasoft Proprietary and Confidential

What IS Static Analysis?

Variety of methodsPeer Review / Manual Code Review / Code InspectionPattern-based code scannersFlow-based code scannersMetrics-based code scannersCompiler / build output

Parasoft Proprietary and Confidential

First things first

More organizations are adopting formal policies regarding static analysisMany companies use a bake-off to choose toolsBake-offs are not very useful to select the best tool

Parasoft Proprietary and Confidential

Assess Your Needs

What pains do you plain to address?FDA, MISRA, PCI, etc.

Is your current development process stable, repeatable, and streamlined?Have you tried static analysis before?

Why did it fail – how can you prevent a repeatHow is your organization structured?

Corporate wide config or varied by group/projectWill analysis apply to all projects? New Code? Legacy?Where do you want to be in the future?

Parasoft Proprietary and Confidential

Compile Candidate List

Get recommendationsPerform due diligence

Even if the tool comes highly recommendedEven if the tool has been used by someone in the groupYour code, process, culture, and environment are unique

Keep the big picture in sight

Parasoft Proprietary and Confidential

Explore Vendors

You’re committing to a relationship with a vendorWhat is their vision?What best practices do they have?

Do they have a coherent strategy for the enterpriseIf they don’t have best practices you’ll need to develop them

ReputationWho uses tool?Case studies

Parasoft Proprietary and Confidential

Talk with Vendors

Evaluations are disruptive – get data upfrontAre your visions in sync with vendor?Explain what problems you want to solve?

Find out if they think static analysis will resolve itCan they set objective criteria to assess success

Describe your environmentHow have they helped others like you?

Explain your vision for deployment and adoption for the next 2-3 years

Do they believe its feasible?

Parasoft Proprietary and Confidential

Handling vision mismatch

Vendor should accept requests that fit their general businessIf they vendor disagrees with your strategy do they have a convincing explanation and alternative?Does the vendor bend over backward? Even for unreasonable requests?

Parasoft Proprietary and Confidential

Pilot Top Candidates

Setup test bed and run preliminary testsFamiliarize yourself with the toolIdentify obstacles

Be ready to assist those doing the actual pilotSelect one group

Real project – not static legacy codeEngineers who like new things

Parasoft Proprietary and Confidential

Work with pilot users

Don’t just give pilot users the program and expect useful resultsExplain

How to use it in your workflowWhat parts of the application to testWhat code should be testedWhat to look for while using the tool

Parasoft Proprietary and Confidential

Pilot tasks

Pilot users should have a list of tasksHow did the tool make their lives better?What could make it even better?How did the tool make their lives worse?How bad was the learning curve?

Parasoft Proprietary and Confidential

Compare Post-pilot Candidates

Zero in on required functionalityEvaluate vendors response to requests and issuesJudge what the relationship will be like

Parasoft Proprietary and Confidential

Evaluation Criteria: Rules

Number of built-in rules you’re really willing to enforceQuality of built-in rules you’re really willing to enforceDepth and breadth of analysisFeasible means to reduce noiseFew to no false positivesTolerable number of missed negativesEase of adjusting built-in rulesEase of adding custom rulesLevel of complexity possible in new rulesVendors plan for adding new rules

Parasoft Proprietary and Confidential

Evaluation Criteria: Workflow

IDE integrationBatch modeViolation reporting / review mechanismAutomated assignment of errors to responsible developersLegacy code identification and supportRule severity customizationAbility to suppress violation reportingAutomated violation correction

Parasoft Proprietary and Confidential

Evaluation Criteria: Scalability

Scalable usage modelEase of updating the rule set team-wide or organization wideAbility to support tiered rule setsExtensibility – APISupport for additional languages and verification methods (unit test, code review, etc)Speed of analysis (end-to-end)

Parasoft Proprietary and Confidential

Evaluation Criteria: Vendor

Product stability Having some issues is inevitable

Defect reportsFeature requestsOverall support

Parasoft Proprietary and Confidential

The 2 Most Important Questions

Will our engineers really adopt it and use it?Can you make the tool work on real code with zero noise?Will it scale?Is the work-flow practical?

Is this a long-term solution?Evaluations consume a lot of time and effortDon’t settle for “it’s good enough”Will it help reach your corporate goals?Time spent now will reward you laterAvoid continuous product evaluations

Parasoft Proprietary and Confidential

Q&A

Parasoft Proprietary and Confidential