coverity multi-threaded whitepaper

8/2/2019 Coverity Multi-threaded Whitepaper

1/13

Ensuring Code

Quality inMulti-threaded

ApplicationsHow to Eliminate Concurrency Deectswith Static Analysis

Ben Chel, CTO


2/13

Ensuring Code Quality in Multi-threaded Applications

Introduction

Most developers would agree that consumers o sotware today continually demand more rom their applications.

Because o its pace o evolution to date, the world now anticipates a seemingly endless expansion o capabilities

rom their sotware, regardless o where that sotware is applied. For the last 40 years o computing, Moores

Law has held true, with the number o transistors available in an integrated circuit doubling approximately

every two years. This constant pace o innovation provided sotware developers with critical hardware resources,

enabling them to expand the eatures and unctionalities o their applications without concern or perormance.

However, while Moores Law still is holding steady, single-core processing has reached its perormance ceiling.

In response, hardware manuacturers have developed new multi-core (or multi-processor) devices to accomplish

the speed gains to which the world had grown accustomed.

Today, the world o sotware development is presented with a new challenge. To ully leverage this new class o

multi-core hardware, sotware developers must change the way they create applications. By turning their ocus

to multi-threaded applications, developers will be able to take ull advantage o multi-core devices and deliver

sotware that meets the demands o the world. But this paradigm o multi-threaded sotware development adds

a new wrinkle o complexity or those who care the utmost about sotware quality. Concurrency deects such as

race conditions and deadlocks are sotware deect types that are unique to multi-threaded applications. Complex

and hard-to-ind, these deects can quickly derail a sotware project. To avoid catastrophic ailures in multi-

threaded applications, sotware development organizations must understand how to identiy and eliminate these

deadly problems early in the application development liecycle.


3/13 Ensuring Code Quality in Multi-threaded Applications

Most sotware development over time has been predicated on single-core

hardware. Thereore, the collective knowledge o sotware developers across

universities, companies, organizations and countries has been based primarily

on single processor hardware platorms. Despite the act that multi-processor

systems have existed or a long time, they have not been the ocus o general

purpose sotware development. Instead, early multi-processor hardware was

preserved or computationally-intensive applications. As a result, multi-

threaded sotware development has not been much o a ocus in mainstreamsotware development.

Now that the world is moving to multi-core on nearly everything, every

sotware developer must become versed in multi-threaded development. There

are armies o developers that are now challenged with learning how to take

advantage o multi-core on the ly, because they have no choice in delivering

on the demands o aster sotware release ater release.

The bad news or sotware developers is that multi-threaded applications

introduce a host o concurrency-related errors, which historically, have been

impossible to eectively test or. When they occur in the ield, concurrency

deects can be catastrophic or sotware, requiring days, i not weeks or months,

to diagnose and repair.

Developers responsible or creating high reliability applications, such as those

in the embedded space, inherit this dangerous new blind spot regarding code quality when they move rom

single- to multi-threaded applications. This paper will review the most common pitalls that sotware developersace when creating multi-threaded applications, and how innovative new sotware analysis capabilities are

emerging that will help development organizations capitalize on the exciting new world o multi-core hardware.

Moving to Multi-threadedApplication Development

Did you usemulti-core today?

The most advertised use o multi-

core processing has been in con-

sumer markets on devices ranging

rom PCs that leverage processors

rom Intel or AMD to gaming

consoles such as Sonys Play

Station 3. In these markets, the

number o processors is a clear

competitive advantage to which

consumers are becoming attuned.

In enterprise markets, multi-core

technology has been adopted

in a number o processor types

including those used in the

digital signal, network and

embedded markets.



To take advantage o multi-core hardware, sotware developers are required

to create multi-threaded applications that can execute multiple operations

concurrently. Concurrency creates new complexities in the sotware

development process that can introduce hard-to-ind, crash-causing sotware

deects. Because widespread

multi-threaded development is still in its relative inancy, many o todays

existing deect detection tools are

ill prepared to help identiy and eliminate this new class o deects.

Testing sotware code is notoriously diicult, even or single-threaded

applications. One reason this is a challenge, is that properly testing an

application requires the simulation o all possible inputs to the program.

Many organizations are satisied when they achieve 95% line coverage in

their programs (meaning 95% o all lines o code in their program are executed

at least once in their test suite). However, this metric o coverage does not even

come close to the total number o possible execution paths through a code base.

Each decision point in a program means at least a actor o two in the number

o paths though that program.

Multi-threaded applications add a new level o complexity to the program that

results in an exponential increase in the number o possible run time scenarios.

See Figure below or an example as to how turning a simple single-threaded

application into a multi-threaded application increases the complexity

tremendously.

CommonCurrency DeectsRace Condition:

Multiple threads access the

same shared data without the

appropriate locks to protect access

points. When this deect occurs,

one thread may inadvertently

overwrite data used by another

thread, leading to both loss o

inormation and data corruption.

Thread Block:

A thread calls a long-running

operation while holding a lock

thereby preventing the progress

o other threads. When this

deect occurs, application

perormance can drop

dramatically due to a single

bottleneck or all threads.

Deadlock:

Two or more threads wait or

locks in a circular chain such that

the locks can never be acquired.

When this deect occurs, the

entire sotware system may halt,

as none o the threads can either

proceed along their current

execution paths or exit.

Why Concurrency (Multi-core)Deects Are So Dangerous



Figure 1: Simple single-threaded program

To completely test the program in Figure , the sotware developer would simply need to make sure that they

ed the program a value that was less than 00 as well as a value that was greater than or equal to 00. Then thetwo paths through the program would be explored and the program would be tested entirely. (NOTE: This

assumes that nothing interesting happens in the input_from_user functionadmittedly a bad assumption in

sotware development!). Now imagine that the end-user wanted this program to input two values instead o one

with the same logic ater the input. This change would increase the complexity o the single-threaded application

in such a way that the sotware developer would need to explore our dierent paths to ully test the program.

Now imagine that a requirement comes in rom the ield to make this application multi-threaded to take

advantage o a dual-core processor. This change would require that the code in Figure would be executed

simultaneously by two dierent threads to take in the two inputs. Assuming that each statement is made up o

three instructions (likely an underestimate), the number o possible interleavings o the instructions in the two

threads leads to a mind boggling 94,480 dierent execution possibilities.

By moving to multi-threaded code in this simpliied example, complexity explodes by ive orders o magnitude,

creating a tremendous challenge or any tester o sotware. I this type o deect does happen to slip into the

ield, developers are then aced with a dramatic increase in the number o avenues that require exploration

beore they can determine the root cause o the problem. Worse, or the irst time, developers may not be able

to reproduce problems that occur in multi-threaded applications because they are not in control o how theirapplication will be executed when it runs. The operating system and thread scheduler together decide when

each thread will be executed as multiple threads execute.

Fortunately new technology in Coverity Prevent is helping developers avoid costly concurrency deects. Leveraging

breakthroughs in static analysis, Coveritys solution can analyze code priorto run-time enabling developers to

identiy and eliminate onerous multi-threaded deects including race conditions, thread blocks and deadlocks.



Deadlocks (situations where a multi-threaded program cannot make any progress) are a particularly diicult

concurrency issue to debug. They arise when the order o lock acquisition used by the program is inconsistent.

A simple example is lock inversion, which happens when one thread attempts to acquire a lock(a), then holds

on to the lock while attempting to acquire lock(b) . I another thread tries to acquire the locks in the opposite

order (acquiring b beore a), neither thread can make progress, and the program is deadlocked. A sample o

this type o sotware deect can be ound in Figure . (Note: While the remaining code examples in this paper

are in Java, all the deects discussed herein can occur in any programming language. Coverity Prevent discovers

concurrency deects in C, C++, and Java code.)

Figure 2: Deadlock Defect in Java Code

I two distinct threads call methods lock1 and lock2, an unlucky scheduling might have the irst thread acquire

lock(a), while the second thread acquires lock(b). In this state, it is impossible or either thread to acquire the

lock it needs to continue execution or release the lock it holds. More complicated deadlocks exist that involve

the acquisition o multiple locks across multiple threads.

Deadlocks



One way that advanced static analysis technology detects deadlocks is to ensure that all locks in the program

are acquired and released in a consistent order. More concretely, the static analysis can construct an acyclic lock

graph or a program. Nodes in a lock graph represent lock names, and an edge between nodes (a) and (b) denotes

that at some point, lock(a) was held while lock(b) was acquired.

I a lock graph contains a cycle, the program may have a lock ordering or lock inversion deadlock. Innovative

static analysis is now capable o inding these lock ordering deects inter-procedurally (through method calls),

as these are likely to be missed by conventional code review or other deect detection techniques.

In Figure above, Coverity Prevent would create the ollowing lock graph or method lock1:

The edge rom Lock.a to Lock.b indicates that Lock.a should always be acquired beore Lock.b. Analysis o

method lock2, however, causes the analysis to add an edge rom Lock.b to Lock.a:

This violates the requirement that the lock graph be acyclic. Thereore, a deect would be reported in lock2.

One o the biggest diiculties in leveraging static analysis to detect deadlocks is the issue o naming locks. In

Figure , it is clear that synchronized(a) locks the static ield a declared in the Lock class. But naming is

not always so obvious. Consider the ollowing reactoring olock2.



Figure 3: Refactoring lock2

Coverity Prevent is sophisticated enough to determine that the call to method (f) has the same eect as beore.It should attempt to add an edge rom Lock.b to Lock.a, ail, and report a deect. The details behind the lock

naming algorithm are part o the innovation that makes Coverity Prevent so powerul at inding complex,

inter-procedural deadlock deects in C, C++ and Java code.



Race Conditions

Race conditions are another potentially disastrous type o concurrency deect that is endemic to multi-threaded

applications. A race condition occurs when multiple threads access the same piece o data concurrently, at least

one thread accessing the data is writing to the data (as opposed to simply reading the data), and the execution

depends critically on the relative timing o events.

This is as concrete a deinition as can be achieved in languages like C and C++ that do not have a ormally

deined memory model. In Java, a more precise (but not necessarily equivalent) deinition can be ormulated

based on the Java memory model. A data race occurs when a variable is read by one or more threads and written

by at least one thread, where the read and write events are not ordered by the happens-before relationship.

The happens-beore relationship precisely deines when events are made visible to other threads (check the JavaLanguage Speciication or a complete deinition o happens-beore). See Figure 4 below or an example o this.

Figure 4: Race Condition

Suppose that an instance o class Race is used to keep a counter o active events across multiple threads. One

problem with Race is that the ++ operator is not atomic. Iincrement is called 0 times rom two dierent

threads, the inal value ocount could be less than 0, because some o the writes to count can be clobberedby writes rom the second thread. One possible ix is to synchronize the increment method as shown in

Figure 5 below.



Figure 5: Race Condition with Synchronized Increment Method

With count accessed in a synchronized block, interleaved calls to increment on the same instance object are

eliminated. Coverity Prevent helps programmers avoid race conditions by inerring and enorcing guarded-by

relationships on shared ields in the program. A ield (f) is guarded-by lock(l) i accesses to (f) must be protected

by holding lock(l). In Figure 5, the count variable is guarded-by the lockield o the Race class. The set o

guarded-by relationships essentially deine the locking discipline or each class in the program.

Coverity Prevent is the irst static analysis solution to iner guarded-by relationships. When a preponderance

o accesses to a ield occur with the same lock held, a guarded-by relationship is inerred between the accessed

ield and the held lock. This may sound simple, but a number o sophisticated techniques are necessary to make

this approach tenable.

Most importantly, the same lock-naming algorithm used in the context o deadlock detection is applied to

associate ields with the speciic lock used to protect access. This serves two purposes. First, it yields stronger

correlations, since ields accessed incidentally with dierent locks held are not used to iner guarded-by

relationships. Second, it allows Coverity Prevent to suggest a possible way to correct the discovered deect, such

as enclosing the access in a synchronized scope with the suggested lock.

Another important technique in discovering race conditions with static analysis is to iner inormation about theprograms thread execution model. This includes inormation such as where threads are created, which unctions

can be called by multiple threads simultaneously, and which pieces o data are shared among threads. Because

Coverity Prevent understands a programs thread execution model, it is able to eliminate alse positives results

due to idioms such as data hando between threads.

A inal, important technique in discovering race conditions is the use o datalow analysis to track values both



intra- and inter-procedurally. Tracking data low makes the analysis less brittle to simple reactorings such as

common subexpression elimination that may preserve the semantics o a given program, but could still conuse

the statistical inerence. Datalow analysis provides a more semantic representation o the program, making code

analysis less sensitive to syntactic changes.

Once guarded-by relationships have been inerred, a best practice that can be leveraged by Java developers

is to permanently enorce the discovered relationships using annotations. Annotations are a Java language-

supported mechanism or associating metadata with program elements. In this case, Coverity Prevent provides

an annotation @GuardedBy(String lockname) that can be placed on ields to denote the lock that should be held

when accessing the annotated ield. Note the change in Figure below rom our previous example in Figure 5.

Figure 6: Updated Race Condition with Synchronized Increment Method

In the example in Figure , Coverity Prevent SQS will report a deect in increment since the access to count is

not protected by the appropriate lock.



Multi-core hardware is clearly increasing sotware complexity by driving the need or multi-threaded

applications. Based on the rising rate o multi-core hardware adoption in both enterprise and consumer devices,

the challenge o creating multi-threaded applications is here to stay or sotware developers. In the coming years,

multi-threaded application development will most likely become the dominant paradigm in sotware.

As this shit continues, many development organizations will transition to multi-threaded application

development on the ly. This creates new liabilities or development teams in terms o application quality and

security, because their code is now vulnerable to a host o concurrency deectsa class o deect that can easily

slip past manual code review and traditional testing techniques.

Developers moving to multi-threaded applications need advanced new testing capabilities to help them control

this new cause o sotware complexity. Because they are nearly impossible to replicate in dynamic testing

environments, static analysis is uniquely suited to play an important role in eliminating concurrency deects early

in the sotware development liecycle. However, due to their underlying complexity, some concurrency deect

types (such as race conditions) have historically escaped detection by conventional static analysis tools.

Today, sophisticated new technology rom Coverity is advancing the science o static analysis to help developers

meet the challenge o creating multi-threaded applications. By arming developers with static analysis capabilities

that detect complex concurrency deects early in the development liecycle, organizations will accelerate their

ability to produce and release multi-threaded applications with greater conidence.

Conclusion


13/13

About CoverityCoverity (www.coverity.com), the leader in improving sotware quality and security, is a privately held company

headquartered in San Francisco. Coveritys groundbreaking technology enables developers to control complexity

in the development process by automatically inding and helping to repair critical sotware deects and security

vulnerabilities throughout the application liecycle. More than 450 leading companies including ARM, Phillips,

RIM, Rockwell-Collins, Samsung and UBS rely on Coverity to help them ensure the delivery o superior sotware.

Free TrialRequest a ree Coverity trial and see irst hand how to rapidly detect and remediate serious deects and

vulnerabilities. No changes to your code are necessary. There are no limitations on code size, and you will

receive a complimentary report detailing actionable analysis results. Register or the on-site evaluation at www.

coverity.com or call us at (800) 83-893.

Headquarters

85 Berry Street, Suite 400

San Francisco, CA 940

(800) 83-893

http://www.coverity.com

[email protected]

Boston

30 Congress Street

Suite 303

Boston, MA 00

() 933-500

UK

Coverity Limited

Magdalen Centre

Robert Robinson Avenue

The Oxord Science Park

Oxord OX4 4GA

England

Japan

Coverity Asia Pacifc

Level 3, Shinjuku Nomura Bldg.

-- Nishi-Shinjuku, Shinjuku-ku

Tokyo 3-053

Japan

coverity multi-threaded whitepaper

Documents