artemis: practical runtime monitoring of applications for execution anomalies long fei and samuel p....
TRANSCRIPT
![Page 1: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/1.jpg)
Artemis: Practical Runtime Monitoring of Applications for
Execution Anomalies
Long Fei and Samuel P. MidkiffSchool of Electrical and Computer Engineering
Purdue University, West LafayettePLDI 2006
Subproject of PROBE
![Page 2: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/2.jpg)
PLDI‘06
Motivation
• Bugs are expensive!• Cost in 2002: 60 billion dollars, 0.6% GDP
• Debugging Approaches• Manual debugging – inefficient, impossible• Extensive testing – inefficient, path explosion• Static analysis – conservative, false alarms
• Manual annotation – inefficient, not scalable
• Runtime debugging – high overhead
![Page 3: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/3.jpg)
PLDI‘06
What is Artemis?• Is not a bug detection tool• Makes existing tools more efficient in bug detection
program execution
runtime analysis
program execution
Artemis
runtime analysisExisting schemes: a few times slowdown is common, can be up to 2 orders of magnitude
With Artemis, much less data is examined by runtime analysis, reducing overhead to
<10% in long-running programs
![Page 4: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/4.jpg)
PLDI‘06
Outline for the Rest of the Talk
• Bird’s eye view of related work
• Artemis framework
• Experimental results
• Conclusions
![Page 5: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/5.jpg)
PLDI‘06
Bird’s Eye View of Compiler-Aided Debugging Techniques
compilertechniques
softwaredebugging
static
dynamic
runtimeoverhead
faster
parallel
selective
no programinformation
use programinformation
samplingrandom
adaptive
Artemis
More efficient design•Problem specific•Usually involves assumptions about OS, compiler, or hardware
Exploit parallelism•Shadow checking process (Patil SPE’97)•Thread-level speculation (Oplinger ASPLOS’02)
Perform fewer checks
Liblit PLDI’03
Chilimbi ASPLOS’04
![Page 6: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/6.jpg)
PLDI‘06
Artemis Design Goals• General
• Work with multiple pre-existing debugging schemes• Pure software approach that works with general
hardware, OS and compiler
• Effective• Improve overhead in general• Have low asymptotic overhead in long-running
programs
• Adaptive• Adjust coverage of monitoring to system load
![Page 7: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/7.jpg)
PLDI‘06
Key Idea
• Because runtime monitoring is expensive
… want to monitor only when a bug occurs
• Our goal is to approximate this• avoid re-monitoring executions whose outcome
has been previously observed
![Page 8: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/8.jpg)
PLDI‘06
How to Determine Where Bugs are Likely to be Seen
• Code region behavior (and bug behavior) is determined by region’s context
• Monitor the 1st time a region executes under a context• If buggy, the bug is monitored• if not buggy, only monitor this region with this context once• Over time, almost all executions of a region have a previously seen
context – yields low asymptotic monitoring overhead
• Hard part – efficiently representing, storing and comparing contexts for a region• The context could be the whole program state!
RegionContext Outcome
![Page 9: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/9.jpg)
PLDI‘06
Decision to Monitor
first entrance ?context seen
before ?
initialize contextupdate context
record, add current context
use monitored version
use unmonitoredversion
N Y
Y N
code segmententrance
![Page 10: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/10.jpg)
PLDI‘06
Target Programs• Our prototype targets sequential code
regions• Determined by how contexts are defined
• Can be used with race-free programs without loss of precision• Target the sequential regions of these
programs
• Use with programs with races is ongoing research
![Page 11: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/11.jpg)
PLDI‘06
Implementation Issues
• Define code regions
• Represent and compare contexts
• Interface with existing runtime debugging schemes
• Adhere to overhead constraints
• Adapt to system load
![Page 12: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/12.jpg)
PLDI‘06
Defining Code Regions
• Spatial granularity
• Temporal granularity: • Context check frequency
• Context check efficiency• Ideal case: a small context dominates the
behavior of a large piece of code
• Our choice:• Procedure: natural logic boundary
![Page 13: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/13.jpg)
PLDI‘06
Approximating Context for Efficiency• Exact Context
• Too large to store and check (might be entire program state)• Represent approximately – tradeoff between precision and
efficiency• Approximated Context
• In-scope global variables, method parameters, in-scope pointers• Values of non-pointer variables are mapped into a compact form
(value invariant – as in DIDUCE ICSE ’02)• Requires 2 integer fields; 2 bitwise operations for each check; 3 bitwise
operations for each update• Pointers tracked by declared (not actual) types•argv approximated by vector length
• Correlations between context elements are lost• If {a=4,b=3} and {a=5,b=8} are two contexts of a region, we track
{a=(4,5), b=(3,8)}
![Page 14: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/14.jpg)
PLDI‘06
Simulating Monitoring Schemes• We need to measure performance on a wide
range of runtime monitoring schemes• A generic monitoring scheme
• Inserts instrumentation into application at probability p• Calls a dummy monitoring function, which simulates the
overhead of some real monitoring scheme• Can adjust overhead from zero to arbitrarily large
• Disable dummy monitoring to reveal the asymptotic overhead of Artemis• Only performs the context checks associated with the
cost of monitoring, but not the monitoring• Allows measuring context checking overhead only
![Page 15: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/15.jpg)
PLDI‘06
Experiment – Asymptotic Overhead Measured by Simulation
![Page 16: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/16.jpg)
PLDI‘06
Two Findings• Performance floor
• As monitoring scheme overhead approaches zero, Artemis overhead is 5.57% of unmonitored program execution time
• When can we use Artemis to improve overhead ? • Break even baseline monitoring overhead• Monitoring overhead > 5.6%, Artemis helps• By solving x=0.0045x+0.0557, x = 5.60%• This covers most of the monitoring techniques
![Page 17: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/17.jpg)
PLDI‘06
An Optimization: Reuse Context Sets Across Runs
• Eliminates the initial building of sets of observed contexts• Converges faster to the asymptotic overhead
• Invariant profile:• Dump the context invariants into a file at program exit• Load dumped invariants at the next run
• Invariant profile size is 0.4 ~ 4.7% of program binary size (average 1.7%, std 0.95%)
![Page 18: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/18.jpg)
PLDI‘06
Using Artemis (with invariant profile)
source instrumentation build
training Artemis
production run bug report
baseline Artemisinvariant
profile
![Page 19: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/19.jpg)
PLDI‘06
Convergence to Asymptotic Overhead – e.g. bzip2 from SPECint
• Asymptotic overhead reduced to < 7.5% (from ~280%)
~7.3%
![Page 20: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/20.jpg)
PLDI‘06
Experiments with Real Monitoring Schemes• Measuring how well does (monitoring scheme
guided by Artemis) approximates the capabilities of original monitoring scheme
• Artemis with hardware-based monitoring (AccMon) – detected 3/3 bugs, 2.67 times improvement, in very short-running programs
• Artemis with value invariant detection and checking (C-DIDUCE) – Source-level instrumentation – covered 75% of violations, 4.6 times improvement, in short-running programs
• Full results and details are in the paper
![Page 21: Artemis: Practical Runtime Monitoring of Applications for Execution Anomalies Long Fei and Samuel P. Midkiff School of Electrical and Computer Engineering](https://reader034.vdocuments.site/reader034/viewer/2022051515/55147600550346b0158b52e3/html5/thumbnails/21.jpg)
PLDI‘06
Conclusions
• General framework
• Eliminate redundant runtime monitoring with context checking
• Improve overhead, low asymptotic overhead in long-running programs
• Small precision loss
• Have enabled practical runtime monitoring of long-running programs