would static analysis tools help developers with code reviews?

Would Static Analysis Tools Help Developers with Code Reviews?

Sebastiano Panichella Venera Arnaoudova Massimiliano Di Penta Giuliano Antoniol

OUTLINE

Context: Code Reviews.

Case Study: Code Reviews of 6 Open Source Projects.

Results: Warnings Resolved by Developers During Reviews.

CODE REVIEWSWhy, What, How?

CODE REVIEWSWhy?

CODE REVIEWSWhy: concrete benefits…

Improved Code Quality

Fewer defects in Code

Improved Knowledge Transfer

Education of Junior Programmers

Benefits

“Expectations, Outcomes, and Challenges of Modern Code Review” Alberto Bacchelli and Christian Bird - ICSE 2013

“Common Outcomes of Code Review”

CODE REVIEWSWhat: types of peer code reviews?

Formal Inspection Process

Over The Shoulder Reviews

Email Pass Around Interviews

Tool assisted reviews

Pair Programming

CODE REVIEWSWhat: types of peer code reviews?

Over The Shoulder Reviews

Email Pass Around Interviews

Tool assisted reviews

Pair Programming

“Modern code review is a form of code inspection which has the qualities of being informal, tool-based

and frequent.”


Formal Inspection Process

MODERN CODE REVIEWS

“Modern code review is a form of code inspection which has the qualities of

being informal, tool-based and frequent.”


MODERN CODE REVIEWS: TOOLS(I)

Code Reviews Management

GERRIT: a Tool to Conduct and Manage Code Reviews

MODERN CODE REVIEWS(I)




(II) Bugs/Issues

Detection



(II) Bugs/Issues

Detection

LIMITATION:provide a too extensive list

of recommendations

Past Work

Kim et al. - FSE 2007

Only10%, of suggestedwarnings are removed

by bug fix changes

To What Extend Static Analysis Tools Help Developers During Code Reviews?

To What Extent Static Analysis Tools Help Developers During Code Reviews?

Project History

To What Extent Static Analysis Tools Help Developers During Code Reviews?

Project History

During Code Reviews

We argue that the Use ofStatic Analysis Tools

Would be Highly Beneficial During Code Reviews…

CASE STUDY

Code Reviews of 6 Open Source Projects.

Goal: understanding how static analysis tools could have helped in dealing with warnings developers solved during code reviews.

Quality focus: reducing developers’ effort during the code review task.

Perspective: develop tool to support the configuration of static analysis tools towards warnings that are considered relevant by developers.

CASE STUDY

RESEARCH QUESTIONS

RQ1: To what extent warnings detected by static analysis tools are removed during code reviews?

RQ2: What kinds of warnings detected by static analysis tool are mainly considered during code reviews?

Projects Observe Period KLOC # of Reviews Analysed

Uses Checkstylee

Uses PDM

Eclipse CDT 2013-11-29 - 2014-09-22

1,500–1,550

309

Eclipse Platform UI

2013-06-24 - 2014-09-09

2,092–2,305

16

Eclipse JDT Core

2013-05-23 - 2014-09-24

2,736–2,554

113

OpenDaylight Controller

2013-01-01 - 2014-09-24

149–171 161

Motech 2013-07-24 - 2014-09-24

586–1,909 209

Vaadin 2013-06-01 - 2014-09-24

6,174–6,114

180

CONTEXTObject:

Tools Experimented:

STUDY PROCEDURE

PATCH SETS COMPARISON…

Given a Code Review


Given a Code Review

We use…


Given a Code Review

We use... to compare warnings density

variation between…

First patch

Last patch

RQ1To what extent warnings detected by

static analysis tools are removed during code reviews?

ProjectsDensity of Warnings

[P-value]# of Warning

[P-value]Density of Warnings


[P-value]

Eclipse CDT 0.074 0.025 0.028 <001

Eclipse JDT Core 0.450 0.919 0.351 0.624

Eclipse Platform UI

0.132 0.857 0.011 0.2


0.080 <0.01 0.614 <0.01

Motech >0.01 <0.01 0.205 <0.01

Vaadin NA NA 0.148 0.209

Changes of Warnings Density (and Absolute Number) During Code Reviews.

ProjectsDensity of Wornings


[P-value]Density of Wornings


[P-value]

Eclipse CDT 0.074 0.025 0.028 <001

Eclipse JDT Core 0.450 0.919 0.351 0.624

Eclipse Platform UI

0.132 0.857 0.011 0.2


0.080 <0.01 0.614 <0.01

Motech >0.01 <0.01 0.205 <0.01

Vaadin NA NA 0.148 0.209

Changes of Warnings Density (and Absolute Number) During Code Reviews.

Cumulative Percentage of Removed Warnings

Projects Uses Checkstyle

Uses PDM

% of Resolved Warnings % of Resolved Warnings

Eclipse CDT 11% 11%Eclipse

Platform UI 5% 7%Eclipse JDT

Core 11% 9%OpenDaylight

Controller 15% 15%

Motech 23% 13%

Vaadin - 13%

RQ2What kinds of warnings detected by

static analysis tool are mainly considered during code reviews?

Qualitative Analysis


“We randomly sampled 10% of code reviews that resolved at least one warning”

Qualitative AnalysisQualitative Analysis

“Warning that Developers Fix During Code Reviews:”



Type Resolution



Unused code

Type Resolution



Imports

Regular Expression

Type Resolution

Unused code

Eclipse CDT: Percentage of PDM’ Resolved Warnings

Warning Types % Resolved Warnings

Type Resolution 100% Import 100%Basic 75%

Sunsecure 67%Codesize 59%

Unusedcode 58%Logging-java 51%

j2ee 47%

Design 42%

junit 38%

Empty 33%Javabeans 26%

Naming 14%

Braces 14%

…. …..

Eclipse CDT: Percentage of PDM’ Resolved Warnings


Type Resolution 100% Import 100%Basic 75%

Sunsecure 67%Codesize 59%

Unusedcode 58%Logging-java 51%

j2ee 47%

Design 42%

junit 38%

Empty 33%Javabeans 26%

Naming 14%

Braces 14%

…. …..

“Quantitative Analisys Confirms Findings of the Qualitative analysis..”

OpenDaylight Controller: Percentage of Checkstyle’ Resolved Warnings


Regular Expressions 100% Modifiers 100%

Metrics 100%import 53%

Whitespace 48%Class Design 47%

Annotations 40%Naming 16%Coding 15%

%Javadoc Comments 12%

Size Violations 11%Javabeans 26%

Block Checks 10%

Miscellaneous 8%

…. …..

“Similar Results for Checkstyle Warnings..”

OpenDaylight Controller: Percentage of Checkstyle’ Resolved Warnings


Regular Expressions 100% Modifiers 100%

Metrics 100%import 53%

Whitespace 48%Class Design 47%Annotations 40%

Naming 16%Coding 15%

%Javadoc Comments 12%Size Violations 11%

Javabeans 26%

Block Checks 10%Miscellaneous 8%

…. …..

Developers Fix also Warnings related to:

1) naming convention

2) code formatting

3) code comments

By implication…

“Enforcing the removal of certain warnings before submitting a patch..”

would static analysis tools help developers with code reviews?

Presentations & Public Speaking