testing concurrent programs program testing and analysis€¦ · int tmp1 = balance; balance = tmp1...

1

Program Testing and Analysis:

Testing Concurrent Programs

Prof. Dr. Michael Pradel

Software Lab, TU Darmstadt

2

Warm-up Quiz

var a = (0.1 + 0.2) + 0.3;var b = 0.1 + (0.2 + 0.3);console.log(a === b);

What does the following code print?

Something elsefalsetrue

2

Warm-up Quiz




Floating point numbers are representedwith finite precision(not only in JavaScript)

2

Warm-up Quiz




0.30000000000000004(due to rounding)

3

Outline

1. Introduction

2. Dynamic Data Race Detection

3. Testing Thread-Safe Classes

4. Exploring Interleavings

Mostly based on these papers:

� Eraser: A Dynamic Data Race Detector for MultithreadedPrograms, Savage et al., ACM TOCS, 1997

� Fully Automatic and Precise Detection of Thread SafetyViolations, Pradel and Gross, PLDI 2012

� Finding and Reproducing Heisenbugs in ConcurrentPrograms, Musuvathi et al., USENIX 2008

4

Why Bother with Concurrency?

� The free lunch provided by Moore’s law is over� CPU clock speeds stopped to increase around 2005� Instead, multi-core processors became mainstream� Need concurrent programs to make full use of the

hardware

� Many real-world problems are inherentlyconcurrent, e.g.,� Servers must handle multiple concurrent requests� Computations done on huge data often are

”embarrasingly parallel”

5

Concurrency Styles

� Message-passing� Popular for large-scale scientific computing, e.g.,

MPI (message-passing interface)� Used in actor concurrency model, e.g., popular in

Erlang and Scala� No shared memory (ideally), all communication via

messages

� Thread-based, shared memory� Multiple concurrently executing threads� All threads access the same shared memory� Synchronize via locks and barriers

5

Concurrency Styles

� Message-passing� Popular for large-scale scientific computing, e.g.,

MPI (message-passing interface)� Used in actor concurrency model, e.g., popular in

Erlang and Scala� No shared memory (ideally), all communication via

messages

� Thread-based, shared memory� Multiple concurrently executing threads� All threads access the same shared memory� Synchronize via locks and barriers

Focus of this lecture

6

Example

int a = 0, b = 0;

boolean r = false, t = false;

a = 1;

r = true;

t = r;

b = a;

Thread 1 Thread 2

What does this program mean?

→ Behavior depends on threadinterleaving

8

Sequential Consistency

Assumption made here:Programs execute under sequential consistency

� Program order is preserved: Each thread’sinstructions execute in the specified order

� Shared memory behaves like a global array:Reads and writes are done immediately

� We assume sequential consistency for the rest ofthe lecture

� Many real-world platforms have more complexsemantics (”memory models”)

9

What Can Go Wrong?

Common source of errors: Data races

� Two accesses to the same shared memorylocation

� At least one access is a write

� Ordering of accesses is non-deterministic

10

Example

// bank account

int balance = 10;

// deposit money

int tmp1 = balance;

balance = tmp1 + 5;

// withdraw money

int tmp2 = balance;

balance = tmp2 - 7;

Thread 1 Thread 2

10

Example

// bank account

int balance = 10;

// deposit money

int tmp1 = balance;

balance = tmp1 + 5;

// withdraw money

int tmp2 = balance;

balance = tmp2 - 7;

Thread 1 Thread 2

Sharedmemorylocation

Read

Write

10

Example

// bank account

int balance = 10;

// deposit money

int tmp1 = balance;

balance = tmp1 + 5;

// withdraw money

int tmp2 = balance;

balance = tmp2 - 7;

Thread 1 Thread 2

3 races

10

Example

// bank account

int balance = 10;

// deposit money

int tmp1 = balance;

balance = tmp1 + 5;

// withdraw money

int tmp2 = balance;

balance = tmp2 - 7;

Thread 1 Thread 2

Quiz: What values can balance

have after executing this code?

10

Example

// bank account

int balance = 10;

// deposit money

int tmp1 = balance;

balance = tmp1 + 5;

// withdraw money

int tmp2 = balance;

balance = tmp2 - 7;

Thread 1 Thread 2

Possible outcomes:balance may be 3, 8, and 15

But: Only 8 is correct

11

Avoiding Data Races

Use locks to ensure that accesses toshared memory do not interfere

int balance = 10;

acquire(L);

int tmp1 = balance;

balance = tmp1 + 5;

release(L);

acquire(L);

int tmp2 = balance;

balance = tmp2 - 7;

release(L);

Thread 1 Thread 2

11

Avoiding Data Races


int balance = 10;

acquire(L);

int tmp1 = balance;

balance = tmp1 + 5;

release(L);

acquire(L);

int tmp2 = balance;

balance = tmp2 - 7;

release(L);

Thread 1 Thread 2

Same lock⇒ Mutuallyexclusive critical sections

11

Avoiding Data Races


int balance = 10;Thread 1 Thread 2

synchronized (L) {

int tmp1 = balance;

balance = tmp1 + 5;

}

synchronized (L) {

int tmp2 = balance;

balance = tmp2 - 7;

}

(Java syntax)

12

Outline

1. Introduction








13

Eraser: Data Race Detection

� Basic idea: Look for ”unprotected” accesses toshared memory

� Assumption: All accesses to a shared memorylocation v should happen while holding the samelock L

→ Consistent locking discipline

� Dynamic analysis that monitors all lockacquisitions, lock releases, and accesses ofshared memory locations

14

Lockset Algorithm (Simple Form)

� Let locksHeld(t) be the set of locksheld by thread t

� For each shared memory location v,initialize C(v) to the set of all locks

� On each access to v by thread t� Set C(v) := C(v) ∩ locksHeld(t)

� If C(v) = ∅, issue a warning

14

Lockset Algorithm (Simple Form)

� Let locksHeld(t) be the set of locksheld by thread t

� For each shared memory location v,initialize C(v) to the set of all locks

� On each access to v by thread t� Set C(v) := C(v) ∩ locksHeld(t)

� If C(v) = ∅, issue a warning

Lockset

Lockset refinement

16

Simple Lockset is Too Strict

Simple lockset algorithm produces falsepositives for� variables initialized without locks held

� read-shared data read without locks held

� read-write locking mechanisms(producer-consumer style)

17

Refining the Lockset Algorithm

Virgin

ExclusiveShared-modified

Shared

� Keep state of each shared memory location� Issue warnings only in the Shared-modified

state

writeread/write by1st thread

write by2nd thread

read by2nd thread

read

write

18

Summary: Eraser

� Dynamic analysis to detect data races

� Assumes consistent locking discipline

� Limitations� May report false positives when locks are

acquired inconsistently but correctly

� May miss data races because it does notconsider all possible interleavings

19

Outline

1. Introduction








20

Thread Safety

� Popular way to encapsulate the challenges ofconcurrent programming: Thread-safe classes

� Class ensures correct synchronization

� Clients can use instances as if they were alone

� Rest of program can treat implementation ofthread-safe class as a blackbox

21

Thread Safety (2)

“behaves correctly when accessedfrom multiple threads ... with noadditional synchronization ... (inthe) calling code”

“operations ... behave as if they occurin some serial order that is consistentwith the order of the method callsmade by each of the individualthreads”

page 18

StringBuffer API documentation, JDK 6

22

Example from JDK

StringBuffer b = new StringBuffer()

b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

22

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

Quiz: What can be the content of b ifStringBuffer is thread-safe?

22

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7

23

Testing Thread-Safe Classes

� Correctness of program relies on thread safety ofspecific classes

� But: What if the class is actually not thread-safe?

� ConTeGe = Concurrent Test Generator

� Creates multi-threaded unit tests

� Detects thread safety violations by comparingconcurrent behavior against linearizations

24

Example Bug from JDK


b.append("abc")

b.insert(1, b) b.deleteCharAt(1)

Thread 1 Thread 2

IndexOutOfBoundsException

Confirmed as bug: Issue #7100996

!

25

ConTeGe

Bug

Classundertest(CUT)

Execute

Thread safetyoracle

Generate aconcurrent test

26

Generating Concurrent Tests


b.append("abc")


Thread 1 Thread 2

Example:

26



b.append("abc")


Thread 1 Thread 2

Sequential prefix:

Create and set upCUT instance

Example:

26



b.append("abc")


Thread 1 Thread 2

Concurrent suffixes:

Use shared CUTinstance

Example:

27

Test Generation Algorithm

1. Create prefix� Instantiate CUT

� Call methods

2. Create suffixes for prefix� Call methods on shared CUT instance

3. Prefix + two suffixes = test

Selection of methods similar tofeedback-directed test generation

28

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

28

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

Randomlyselect aconstructor

28

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

After adding a call:Execute

3

28

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

Randomlyselect amethod

b.append(/* String */)

28

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

b.append(/* String */)

Arguments:a) Take available objectb) Call method returning

required typec) Random value

28

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2



28

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2


3

28

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

29

Creating Suffixes

2. Create suffixesfor prefix

� Call methods onshared CUT instance

29

Creating Suffixes




b.append("abc")

b.insert(1, b)

Randomlyselect amethod

b.insert(/* int */, /* CharSequence */)

29

Creating Suffixes




b.append("abc")

b.insert(1, b)b.insert(/* int */, /* CharSequence */)



29

Creating Suffixes




b.append("abc")

b.insert(1, b)b.insert(-5, b)



29

Creating Suffixes




b.append("abc")

b.insert(1, b)b.insert(-5, b)


!

29

Creating Suffixes




b.append("abc")

b.insert(1, b)b.insert(/* int */, /* CharSequence */)



29

Creating Suffixes




b.append("abc")

b.insert(1, b)



29

Creating Suffixes




b.append("abc")

b.insert(1, b)


3

29

Creating Suffixes




b.append("abc")


29

Creating Suffixes




b.append("abc")



3

29

Creating Suffixes




b.append("abc")


30

Creating a Test

3. Prefix + two suffixes = test


b.append("abc")


Thread 1 Thread 2

Spawn new threadfor each suffix

31

Approach

Bug

Classundertest(CUT)

Execute

Thread safetyoracle

Generate aconcurrent test

32

Thread Safety Oracle

Does the test executionexpose a thread safetyviolation?

� Focus on exceptionsand deadlocks

� Compare concurrentexecution tolinearizations

33

Linearizations

� Put all calls into one thread� Preserve order of calls within a thread

33

Linearizations


21 3

33

Linearizations

213

2

13

21

3


21 3

34

The Oracle

Exception ordeadlock?

Execute concurrently

No

Yes

Yes

No Thread safetyviolation

Samefailure?

Execute linearization All linearizationschecked

3

3

35

Example


b.append("abc")


StringBuffer b = ..

b.append("abc")

b.insert(1, b)

b.deleteCharAt(1) 3

Thread 1 Thread 2

StringBuffer b = ..

b.append("abc")

b.deleteCharAt(1)

b.insert(1, b) 3

!

35

Example


b.append("abc")


StringBuffer b = ..

b.append("abc")

b.insert(1, b)

b.deleteCharAt(1) 3

Thread 1 Thread 2

StringBuffer b = ..

b.append("abc")

b.deleteCharAt(1)

b.insert(1, b) 3

!Thread safety violation

36

Properties of the Oracle

Sound but incomplete *

� All reported violations are real� Cannot guarantee thread safety

Independent of bug type� Data races� Atomicity violations� Deadlocks

* with respect to incorrectness

37

Implementation & Results

� Implemented for Java classes

� Applied to popular thread-safe classesfrom JDK, Apache libraries, etc.

� Found 15 concurrency bugs, includingpreviously unknown problems in JDK

� Takes between several seconds andseveral hours (worst-case: 19 hours)� Recent master thesis, published at ICSE’17:

Worst-case time reduced to several minutes

38

Outline

1. Introduction








39

Scheduling Non-Determinism

� A single program executed with a single inputmay have many different interleavings

� Scheduler decides interleavingsnon-deterministically

� Some interleavings may expose bugs, othersexecute correctly (”Heisenbugs”)

� Challenge: How to explore different interleavings?How to detect buggy interleavings?

40

CHESS in a Nutshell

� A user mode scheduler that controlsall scheduling non-determinism

� Guarantees:� Every program run takes a new thread

interleaving� Can reproduce the interleaving for every run

� Systematic but non-exhaustiveexploration of the set of possibleinterleavings

41

Tree of Interleavings

� Search space of possibleinterleavings: Represent as a tree

� Node = points of scheduling decision

� Edge = decisions taken

� Each path = one possible schedule

42

Example

// bank account

int balance = 10;

// deposit money

int tmp1 = balance;

balance = tmp1 + 5;

// withdraw money

int tmp2 = balance;

balance = tmp2 - 7;

Thread 1 Thread 2

44

State Space Explosion

Thread 1:

instr. 1instr. 2...instr. k

n threads

k instructions

� Number ofinterleavings: O(nn·k)

� Exponential in both n

and k

� Typically: n < 10,k > 100

� Exploring allinterleavings does notscale to largeprograms (i.e., large k)

Thread 2:

instr. 1instr. 2...instr. k

45

Preemption Bounding

� Limit exploration to schedules with a smallnumber c of preemptions� Preemption = Context switches forced by the

scheduler

� Number of schedules: O((n2 · k)c · n!)� Exponential in c and n, but not in k

� Based on empirical observation: Mostconcurrency bugs can be triggered with few (< 2)interleavings

46

Implementation and Results

� Implemented via binary instrumentation

� Applied to eight mid-size and large systems (upto 175K lines of code),

� Found a total of 27 bugs

� Major benefit over stress testing: Once a failure isdetected, can easily reproduce and debug it

47

Summary

� Concurrent programming is inevitable

� Writing correct concurrent programsis hard

� Techniques to detect concurrencybugs� Dynamic data race detection� Test generation and thread safety checking� Systematic exploration of interleavings

testing concurrent programs program testing and analysis€¦ · int tmp1 = balance; balance = tmp1...

Documents