automated atomicity-violation fixing

AutomatedAtomicity-ViolationFixing

Guoliang Jin, Linhai Song, Wei Zhang, Shan Lu, and Ben LiblitUniversity of Wisconsin–Madison

AFix

1

Multicore era is coming already here Programmers struggle to reason about concurrency More and more concurrency bugs

Many concurrency bugs can be automatically detected

But bugs need to be fixed

Needs to Find Concurrency BugsNeeds to Find and Fix Concurrency Bugs

2

Thread 1

if (ptr != NULL) {

ptr->field = 1;

}

Thread 2

ptr = NULL;

Segmentation Fault

Understand bug1

…

…

Understand bugn

Bug-fixing process is lengthy and resource consuming

Nearly 70% of patches are buggy in their first releases Automated fixing is desired, but difficult in general

Bug-fixing

3

Understand a bug Generate a patchReview& testthe patch

CorrectnessPerformanceReadability

Concurrency bugs are feasible to be fixed automatically Program is correct in most interleavings.

Only need to remove some bad interleavings.

Automated Concurrency-Bug Fixing

4

Thread 1if (ptr != NULL) { ptr->field = 1;}

Thread 2

ptr = NULL;

Thread 1

if (ptr != NULL) { ptr->field = 1;}

Thread 2ptr = NULL;

Thread 1if (ptr != NULL) {

ptr->field = 1;}

Thread 2

ptr = NULL;

Segmentation Fault

Why atomicity-violation bugs? One of the most common types of concurrency bug

Strategy Statically adding locks to remove buggy interleavings.

Goal Automate the whole bug-fixing process Provides best-effort atomicity-violation patches

• Correctness• Performance• Readability

AFix: Automated Atomicity-Violation Fixing

5

Input from

CTrigger

AFix Overview

6

Bug understanding

Manual Bug Fixing Progress

A single-variable atomicity-violation detection & testing tool

It reports a list of buggy instruction triples Abbreviated as {(p1, c1, r1), …, (pn, cn, rn)}

CTrigger Bug-Detector Review

7

Thread 1

if (ptr != NULL) {

ptr->field = 1;}

Thread 2

ptr = NULL;

previous access

current access

remote access

Input from

CTrigger

AFix Overview

patchn

patch1(p1, c1, r1)…...…(pn, cn, rn)

mergedpatch1

adding runtime support

patchtesting

……… merged

patchm

…

8

Patch generationBug understanding

Patch testing

Manual Bug Fixing Progress

Motivation Overview AFix

One bug patching Patch Merging Runtime support Patch testing

Evaluation Conclusion

Outline Motivation Overview AFix






9

Make the p-c code region mutually exclusive with r Put p and c into a critical section Put r into a critical section Select or introduce a lock for the two critical sections

One Bug Patching(p, c, r) Patching

10

p

c

r

A naïve solution Add lock on edges reaching p Add unlock on edges leaving c

Potential new bugs Could lock without unlock Could unlock without lock etc.

Put p and c into a Critical Section: naïve

p

c

p

c

11

p

c

p

c

Assume p and c are in the same function f Step 1: find protected nodes in critical section

In f’s CFG, find nodes on any p c path Step 2: add lock operations

unprotected node protected node

protected node unprotected node Avoid those potential bugs mentioned

Put p and c into a Critical Section: AFix

p

c

12

p and c adjustment when they are in different functions Observation: people put lock and unlock in one function Find the longest common prefix of p’s and c’s stack traces Adjust p and c accordingly

p and c Adjustment

void newlog(){ … close(); open(); …}

void close() { … log = CLOSE;}

void open() { … log = OPEN;}

void newlog(){ …p: close();c: open(); …}

p:

c:

…newlog()close()

…newlog()open()

13

(p, c, r) Patching: put r into a critical section Lock-acquisition before r, lock-release after r

Only if r cannot be reached from the p–c critical section

fpc() {lock(L1)

p...r

…c

unlock(L1)}

case 1

fpc() {lock(L1)

p...

foo() {…r}…c

unlock(L1)}

case 2r’s call stack: … fpc foo …r

14

(p, c, r) Patching: select or introduce a lock Use the same lock for the critical sections Lock type:

Lock with timeout : in case of potential new deadlock Reentrant lock : in case of recursion Otherwise: normal lock

Lock instance: Global lock instances are easy to reuse

15

c1

r1

p1

p2

c2, r2

void buf_write() { int tmp = buf_len + str_len; if (tmp > MAX) return;

memcpy(buf[buf_len], str, str_len); buf_len = tmp;}

Patch Merging

16

p1

c1 p2r1 c2, r2

Too many lock/unlock operations Potential new deadlocks May hurt performance and readability

One programming mistake can lead to multiple bug reports They should be fixed all together

Redundant patch, when p1–c1, p2–c2 critical sections are in the same function: redundant when one protected

region is a subset of the other are in different functions: consulting the stack trace again

Patch Merging: redundant patch

lock(L1)p1

lock(L2)p2c2

unlock(L2)c1

unlock(L1)

lock(L1)r1

unlock(L1)

lock(L2)r2

unlock(L2)

lock(L1)p1

p2c2

c1unlock(L1)

lock(L1)r2

unlock(L1)

17

Related patch Merge if p, c, or r is in some other patch’s critical sections

Patch Merging: related patch

lock(L1)p1

lock(L2)p2c1

unlock(L1)c2

unlock(L2)

lock(L1)r1

unlock(L1)

lock(L2)r2

unlock(L2)

lock(L1)p1

p2c1

c2unlock(L1)

lock(L1)r2

unlock(L1)

18

Runtime support to handle deadlock Lightweight patch-oriented deadlock detection

• Whether timeout is caused by potential deadlock?• Only detect deadlocks caused by the patches• Has low-overhead, and suitable for production runs• Help patch refinement

Traditional deadlock detection

In-house patch testing

Runtime Support and Testing

19







Outline

20

Evaluation: Overall Patch QualityBug Naïve Unmerged Merged Manual

Apache - - MySQL 1 - MySQL 2 -

Mozilla 1 -

Mozilla 2 - Cherokee - FFT - - PBZIP2 - - -

Patched failure rates: 0% (except PBZIP2 and FFT)

Patched overheads: <0.6% (except PBZIP2)

With timeout triggered deadlock detection21

Conclusion Atomicity violations are feasible to be fixed automatically

By removing bad interleavings Must be careful in the details

Use some heuristics, and excellent results in practice Completely eliminates detected bugs in targeted class Overheads too low to reliably measure Produces small, simple, understandable patches

Future research should do detector and fixer co-design

22

Questions about AFix*?*DISCLAIMER FROM AFIX: “ I REPRESENT HUMANS’ EFFORTS TOWARDS FIXING THE WORLD AUTOMATICALLY USING TOOLS. HOWEVER, THE WORLD IS SO IMPERFECT THAT I DO NOT KNOW WHETHER THE WORLD IS FULLY F IXABLE, THUS I MAKE NO 100% GUARANTEE.”

23

automated atomicity-violation fixing

Documents