Coverity Development Testing

Accelerating Risk Mitigation through Continuous Integration and Development Testing

“Software is Eating the World”

Software

- Marc Andreessen

Health

Financial

Communications

SCM / Logistics

Enterprise

Mobile

81% of business leaders believe technology is a fundamental element of their business model Over 60 million tablets and 175 million smart phones will be in the workplace by the end of 2012 By 2016, open source software will be included in mission-critical applications within 99% of Global 2000 enterprises

Automotive

Development Testing … … is transforming software development by:

Reducing operational costs

Accelerating development and time to market

Protecting brands from catastrophic failure

Why All the Risk? Software Complexity and Speed have Outpaced Legacy Testing Methods

Development Testing

Software Complexity

Time to Market

Testing Methods Security Testing Functional Testing

Performance Testing Manual Testing

Fewer defects escape development

Design Development Quality & Security

Assurance

Product Release & Management

Development Testing Transform software testing, from reactive to proactive

Transformation Maturity Model

Level 1 Automatic Defect Detection

No new defects introduced.

Level 3 Developer Workflow Optimization

Feeding all components into the developer workflow

Level 4 Code Governance

Establish source code acceptance criteria

Complete Enterprise Code Assurance

All critical code and code impacted by change is tested

Inte

grat

ion

into

SD

LC

Hig

h

Development Testing Adoption High

Level 2 Identification of Residual Risk

Ensure critical code is prioritized & tested

How Coverity Static Analysis Works

Mimicks the behavior of dozens of compilers

Integrates with existing build systems

Statically tests all execution paths

Finds defects and inconsistent coding patterns

Analyze Build

Explains the location and root cause of defects

Manage and share triage of defects across teams

Present & Manage

Meaningful, real results Focus on finding real defects, not style violations or superficial issues.

Over 12 years of experience analyzing open source and commercial code.

Industry-leading low false positive/negative rate False positive rates typically below 15%

False positives waste time, hinder adoption, and reduce trust in the results.

Broadest Checker Library + Deepest Algorithms Optimal balance of breadth, depth, and scalability to large code bases.

High Quality Results

Sample Project: PostgreSQL

Defects Fixed in 2012 per Category Category # Defects Impact Memory  –  corrup,ons   20   High  Memory  -­‐  illegal  accesses   10   High  Resource  leaks   43   High  Unini,alized  variables   10   High  API  usage  errors   1   Medium  Control  flow  issues   4   Medium  Error  handling  issues   14   Medium  Incorrect  expression   3   Medium  Insecure  data  handling   24   Medium  Integer  handling  issues   8   Medium  Null  pointer  dereferences   43   Medium  Code  maintainability  issues   58   Low  Security  best  prac,ces  viola,ons   15   Low  

Grand  Total   253  

•  ~20 Developers •  Weekly Build •  680k LOC •  False Positive Rate: 11.1% •  Defect Density: 0,273

We Find Critical Defects •  Tomcat Webserver 5.5.17 •  Among several hundred defects, we found a “reverse lock

bug” that can lead to deadlock of the entire server

Focus testing time where it matters

… don’t waste time writing tests you don’t need

Test Advisor Improving Unit Testing Effectiveness and Efficiency

High Risk Code

High Risk Code

Risk Mitigation Architecture

Test Advice Actionable work items to address risk

due to inadequate testing

Test Policy Evaluation •  Critical code analysis •  Change impact analysis •  Test execution analysis

Test Monitoring Code Ownership and Change History

Static Code Analysis

Customized Test Policy

Move Quality into the Inner Loop of Development

Code

Build

Test Nightly Build Continuous Integration

Finding and Fixing Quality Defects

QA

Development Testing Workflow

Code Check In

Development Security Audit

Static Analysis Results

Nightly/Continuous Build Regression Test

•  Built into development process •  Retesting minimized •  Immediately actionable by developers •  Reduces burden on auditing team

Developer QA Security

Issue Responsibility Is Critical

Ingredients for Success

Code

Build

Test Nightly Build Continuous Integration

High-Fidelity Code

Compilation

High-Performance

Analysis

Low False Positive Rate

Detecting Critical Defects

Easy Defect Navigation and Comprehension

Comprehensive Triage and

Remediation

Management Visibility and Governance

Team Collaboration

Governance with Metrics

Automated high-fidelity analysis on daily basis

18

Fast and educated triage of results to categorize and prioritize issues Accurate

Data

Precise actions based on comprehensive data analysis

Trusted Data

Policy Definition and Monitoring Definition of organizational-wide policies for code quality Aggregated sanity view of code by component, team, supplier

Supplier SLA Enforcement Supplier self-certification based upon policies

Transformation Maturity Model

Level 1 Automatic Defect Detection

No new defects introduced.

Level 3 Developer Workflow Optimization

Feeding all components into the developer workflow

Level 4 Code Governance

Establish source code acceptance criteria

Complete Enterprise Code Assurance

All critical code and code impacted by change is tested

Inte

grat

ion

into

SD

LC

Hig

h

Development Testing Adoption High

Level 2 Identification of Residual Risk

Ensure critical code is prioritized & tested

Coverity Development Testing Platform

Security Advisor

Test Advisor

Analysis Packs

Coverity SAVE™ Static Analysis Verification Engine

SDLC Integrations Policy Manager

Quality Advisor

Architecture Analysis

Dynamic Analysis

FindBugs™ Analysis

Analysis Integration

Toolkit

Coverity Connect

Test Execution

Third Party Metrics

Build/ Continuous Integration

HP ALM

IDE

Code Coverage

Defect Tracking

SCM

ü  Proven significant operational cost reductions

ü  Metric visibility of code estate onshore and offshore

ü  Proven history of finding crash causing or

unexpected behavior causing defects

ü  Process Improvement of the Application Lifecycle

Management

Coverity Summary

Questions ?!?!!!