verification at scale: fitting static code analysis into continuous integration

1© 2016 Rogue Wave Software, Inc. All Rights Reserved.

1

Verification at scale: Fitting static code analysis into continuous integration

Embedded Conference Scandinavia


2

Evolution of SCA


3

At first there was the desktop analysis …• All started in December 1977, Stephen C

Johnson, Bell Labs• Simple, structural static analysis on a file-by-file

basis

Advantages

Disadvantages

• Developer learning• Immediate, post compile• Fast feedback loop

• Low quality due to file scope

• Unregulated platform / build

• No collaborative working


4

Then server-side analysis …• Deep, inter-procedural, control- and data-

flow analysis by dedicated servers on a periodic basis (nightly, weekly, per-release)

Advantages

Disadvantages

• Centralised – It is the simplest option to implement and maintain

• Reliable – The build process and platform are verified• Accurate – Performs a full analysis and sees the full picture

• Feedback – slow feedback creates rework, lower fix rates and higher new issue rates

• Developer adoption – not seen as developer task


5

Now client-server connected desktop …• Current state of the art solution: Continuous

integration and static code analysis• Deep, inter-procedural control- and data-flow analysis

connected to the developer desktop• Enables pre-flight analysis of new and changed code,

in real time

Advantages

Disadvantages

• Desktop feedback loop and developer focus• Server accuracy, centralised configuration• The best of both worlds!

• Management of desktop tools• Requires desktop build to be possible• Desktop environments don’t always match


6

Continuous integration


7

What is continuous integration (CI)?

• In software engineering, CI is the practice of merging all developer working copies to a shared mainline several times a day. Grady Booch first named and proposed CI in his 1991 method, although he did not advocate integrating several times a day.

• Continuous integration – the practice of frequently integrating one's new or changed code with the existing code repository – should occur frequently enough that no intervening window remains between commit and build, and such that no errors can arise without developers noticing them and correcting them immediately.


8

Example CI process


9

CI best practices

• Automate the build• Everyone commits to the baseline every day• Every commit should be built• Keep the build fast• Fast feedback


10

So why CI?• CI embodies the concept that by

checking the impact of changes more frequently we’re able to quickly identify the cause of any problems and remediate as soon as possible with no nasty surprises downstream

• CI gives us greater understanding, earlier, so we can then act upon that knowledge as we see fit

• It is an early warning system – we are checking the things that could cause delays or risk later earlier in the cycle – we are shifting them to the left


11

Shift left• CI shifts to the left our checking of code integration and

build problems• But it doesn’t stop there, consider:

– Continuous deployment– Continuous analysis– Continuous testing– Continuous reporting– Continuous compliance


12

Continuous integration and static code analysis

– better together


13

Example CI process with SCAKlocwork

KlocworkKlocworkKlocwork


14

Enhanced SCA process with CI• For some developers, compiling their code on the desktop is not

possible, so desktop analysis is not an option• In addition, integration issues may still be detected after check

in, even when using desktop analysis

Edit & Save

Analyze& Fix

Compile & Test

Check In

Developer 1

Edit & Save

Analyze& Fix

Compile & Test

Check In

Developer 2

Time

Integrate

Check In

Compile & Test

Check In

New possible issues found

here!


15

Continuous static code analysis


16

The future: Continuous static code analysis …• Continuous static code analysis (CSCA) brings all the benefits of

centralised server-side, deep, inter-procedural control- and data-flow analysis to a near-desktop feedback timescale!

• Central management of development systems fits well with DevOps movement

• Enables continuous reporting and continuous compliance

Advantages

Disadvantages

• Near desktop speed feedback loop• Server accuracy, centralised configuration• Visibility of the current status

• Not quite as fast as connected desktop analysis• Server resource requirements of CI builds• Tooling must be designed for CI


17

Designed for CITo work in a true CI environment CSCA tools must be designed to be:

Automated

Fast(er)

Scalable

RelevantTo reduce feedback time, only the affected code should be analyzed

By requiring minimal resources & deploying across multiple agents

By reporting only the information that is required for the given context (example: only the diffs since the last build / build X)

Supporting the most important CI build management systems


18

Consider before implementing CSCA• The addition of static code analysis to a CI build system means that

an integration analysis (similar to a server-analysis) is performed on every check in– Developers get integration static code analysis results after

every check in or prior to each promotion?– Less effort spent on changes and less risk to delivery timescales,

as there should be no outstanding issues at release time– Continuous static code analysis performance is fast if only the

new and changed code is analyzed and reported

• If there are additional tests that must be written due to defects detected, they can become part of the test plan immediately


19

Conclusion


20

So what’s right for you?• There is no one size fits all and even within one organization. All of

these factors play a part in the decision: :– The type of projects– The phase of projects (developing, maintenance, re-opened

legacy) – The development tools and reporting requirements

• New engineers benefit hugely from desktop feedback, where they can experiment, learn, and improve without the fear of exposure

– SCA in this scenario provides a huge benefit for developer training

– But the issues have to be accurate so connected (client-server) desktops are better

– Detailed checker documentation, linking back to appropriate industry standards or compliance guidelines will certainly help


21

So what’s right for you?• Rapidly developing, new projects, where large groups of engineers are

committing new code will reap the benefits of regular integration analysis giving them a public view of the merged code and analysis results

– Here true CI, CSCA is hard to beat if done properly, whereas server only analysis would result in a fast growing backlog and a mammoth task of unpicking complex issues

– Connected desktop analysis pre-check in may detect many self contained issues even earlier keeping integration issues very clearly highlighted

• Legacy projects or projects entering a maintenance phase with compliance requirements will work well with results that are public, reliable (based on certified hardware and tool chains) and clear

– Server only analysis is sufficient and the lowest level of effort to deliver this


22

It all boils down to staying Agile…• Ultimately then, we’re going to

need different capabilities from our static code analysis tooling at different times for different projects and with different teams

• Tool agility is as important as team agility for making the most of the productivity opportunities


23

Questions?

23

Steve HowardField technical services, EMEARogue Wave Software

[email protected]


24

roguewave.com

http://www.roguewave.com/

http://www.roguewave.com/