![Page 1: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/1.jpg)
Reducing Performance Nondeterminism via Cache-aware
Page Allocation Strategies
Distributed Systems Research Grouphttp://dsrg.mff.cuni.cz
Department of Software Engineering
Charles University in PragueFaculty of Mathematics and Physics
Czech Republic
Michal Hocko, Tomas Kalibera
![Page 2: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/2.jpg)
Tomas Kalibera WOSP/SIPEW 2010
Performance Non-determinism
Request index
Re
spo
nse
tim
e
![Page 3: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/3.jpg)
Non-determinism in execution is particuarly bad for benchmarking
![Page 4: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/4.jpg)
Tomas Kalibera WOSP/SIPEW 2010
Sample Benchmark Structure
Main() {
initialize();
warm-up();
for(i=0;i<nmeasurements, i++) {
before = getCurrentTime();
doOperation();
after = getCurrentTime();
results[i] = after – before;
}
print(results);
}
Cmd-line> ./benchmark
Cmd-line> ./benchmark
Cmd-line> ./benchmark
One measurement
One execution
![Page 5: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/5.jpg)
Tomas Kalibera WOSP/SIPEW 2010
Non-Determinism in MeasurementTi
me
[m
s]
1st execution 2nd execution
Non-Determinism in Measurement and Execution
measurements
measurements
average
![Page 6: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/6.jpg)
Tomas Kalibera WOSP/SIPEW 2010
Non-Determinism in MeasurementNon-Determinism in Measurement and Execution
![Page 7: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/7.jpg)
Tomas Kalibera WOSP/SIPEW 2010
Non-Determinism in Measurement and Execution
![Page 8: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/8.jpg)
Tomas Kalibera WOSP/SIPEW 2010
Non-determinism in Execution is Costly
Main() {
initialize();
warm-up();
for(i=0;i<nmeasurements, i++) {
before = getCurrentTime();
doOperation();
after = getCurrentTime();
results[i] = after – before;
}
print(results);
}
Cmd-line> ./benchmark
Cmd-line> ./benchmark
Cmd-line> ./benchmark
One measurement
One execution
Repeated with every execution
![Page 9: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/9.jpg)
Non-determinism in execution is caused by cache & virtual memory
![Page 10: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/10.jpg)
Tomas Kalibera WOSP/SIPEW 2010
Application Memory Layout (Linux)
Program code
Initialized data
Non-initialized data
Heap
StackCmd-line arguments
Environment
High virtual addresses
Low virtual addresses
Vir
tual
me
mo
ry
![Page 11: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/11.jpg)
Tomas Kalibera WOSP/SIPEW 2010
Cache and Addressing on Typical System
Page address Page offset
Cache set Line offset
Cachelines
data
data
Page address Page offset
Physical address
Virtual address
Decided by OS
![Page 12: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/12.jpg)
Tomas Kalibera WOSP/SIPEW 2010
• Operating system assigns colors to pages• Data from pages of different colors do not collide
in the cache
Page/Cache Color
Page address Page offset
Page address Page offset
Physical address
Virtual address
Decided by OS
Page Color (Cache Color)
![Page 13: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/13.jpg)
Could a cache-aware strategy forselecting page colors reduce non-
determinism in execution ?
![Page 14: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/14.jpg)
Tomas Kalibera WOSP/SIPEW 2010
Good Old Cache-aware Strategies
• Page Coloring
– Heuristic for “spatial locality”
– Adjacent pages have different color – do not collide
– Solaris, Windows, Free BSD
• Bin Hopping
– Heuristic for “temporal locality”
– Pages first accessed in sequence have different color
– Digital Unix
• No Support in Linux
![Page 15: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/15.jpg)
Tomas Kalibera WOSP/SIPEW 2010
• Linux Kernel extension for strategies
– Supports bin hopping and page coloring as modules
– Supports more: other strategies, application layer control, etc
• Large empirical study in Linux
– 4500 benchmark experiments
– Evaluation based on statistical methods
Our Contribution
![Page 16: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/16.jpg)
Tomas Kalibera WOSP/SIPEW 2010
• Mono (C#)– SciMark2 – FFT (numerical)– TCP/HTTP Ping (remote communication)– Rijndael (cryptography)
• SciMark2 (C, numerical)– FFT, Matrix Factorization, Monte Carlo, …
• Csibe (C/C++)– JPEG (multimedia compression)– GZIP, BZIP2, PNG (lossless compression)– Lexical analysis, abstract machine simulator, …
Benchmarks
![Page 17: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/17.jpg)
Tomas Kalibera WOSP/SIPEW 2010
• Executed about 4500 experiments
• Question for evaluation:– “Does page coloring or bin hopping provide lower
response time/non-determinism than the default kernel strategy ?”
• Metrics– Mean response time, impact factor of non-
determinism
• Quantitative Summary
• Qualitative Summary
Evaluation Methodology
![Page 18: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/18.jpg)
Tomas Kalibera WOSP/SIPEW 2010
Non-det. in Execution: Quantitative Summary
![Page 19: Reducing Performance Non-determinism via Cache-aware Page ... · Tomas Kalibera WOSP/SIPEW 2010 •Response time –Cache-aware strategies don’t help –Page coloring performs like](https://reader034.vdocuments.site/reader034/viewer/2022043017/5f39b7e02970a0231a3d40eb/html5/thumbnails/19.jpg)
Tomas Kalibera WOSP/SIPEW 2010
• Response time– Cache-aware strategies don’t help
– Page coloring performs like default, bin hopping is sometimes slightly slower
• Non-determinism– Cache-aware strategies reduce non-determinism
– Bin hopping sometimes reduces a bit more than page coloring
• Our kernel extension allows to select a strategy on application basis
Summary