performance is dead, long live performance · specint2006 cpu performance 6 0.01 0.10 1.00 10.00...
TRANSCRIPT
![Page 1: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/1.jpg)
Performance is Dead, Long Live Performance
Ben Zorn
Microsoft Research
![Page 2: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/2.jpg)
Outline
Good news
Bad news
Good news again!
Mystery…
2 Ben Zorn CGO 2010 Keynote
![Page 3: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/3.jpg)
1990s A Great Decade for Performance!
• Stock market booming
• Itanium processor shipping
• Processor performance growing exponentially (Moore’s Law)
• Compiler research booming
3 Ben Zorn CGO 2010 Keynote
![Page 4: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/4.jpg)
NASDAQ Booming
0
1000
2000
3000
4000
5000
6000
1/3/1995 1/3/1996 1/3/1997 1/3/1998 1/3/1999 1/3/2000 1/3/2001 1/3/2002 1/3/2003
4 Ben Zorn CGO 2010 Keynote
![Page 5: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/5.jpg)
New Processors Had High Expectations
Source: CNET Networks from data provided by Sun and IDC (12/7/2005)
Itanium Sales Forecasts
5 Ben Zorn CGO 2010 Keynote
![Page 6: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/6.jpg)
SPECint2006 CPU Performance
6
0.01
0.10
1.00
10.00
100.00
88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09
Year of Introduction
intel 486intel pentiumintel pentium 2intel pentium 3intel pentium 4intel itaniumAlpha 21064Alpha 21164Alpha 21264SparcSuperSparcSparc64MipsHP PAPower PCAMD K6AMD K7AMD x86-64IBM PowerSUN UltraSPARCIntel Core 2AMD Opteron
Numbers courtesy of Mark Horowitz, Ofer Shacham Ben Zorn CGO 2010 Keynote
![Page 7: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/7.jpg)
Performance Papers Dominate PLDI
0
5
10
15
20
25
30
35
Pap
ers
Pu
blis
he
d
Correctness
Other
Performance
8 Ben Zorn CGO 2010 Keynote
![Page 8: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/8.jpg)
Some Cynics: Proebsting’s Law
http://research.microsoft.com/en-us/um/people/toddpro/papers/law.htm
9
• Proebsting's Law: Compiler Advances Double Computing Power Every 18 Years “…This means that while hardware computing horsepower increases at roughly 60%/year, compiler optimizations contribute only 4%. Basically, compiler optimization work makes only marginal contributions.”
Ben Zorn CGO 2010 Keynote
![Page 9: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/9.jpg)
The Bubble Bursts
0
1000
2000
3000
4000
5000
6000
1/3/1995 1/3/1996 1/3/1997 1/3/1998 1/3/1999 1/3/2000 1/3/2001 1/3/2002 1/3/2003
10 Ben Zorn CGO 2010 Keynote
![Page 10: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/10.jpg)
Itanium Sales Lag
Source: CNET Networks from data provided by Sun and IDC (12/7/2005)
http://news.cnet.com/2300-1006_3-5873647.html 11 Ben Zorn CGO 2010 Keynote
![Page 11: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/11.jpg)
Uniprocessor Performance Flattens
12
0.01
0.10
1.00
10.00
100.00
88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09
Year of Introduction
intel 486intel pentiumintel pentium 2intel pentium 3intel pentium 4intel itaniumAlpha 21064Alpha 21164Alpha 21264SparcSuperSparcSparc64MipsHP PAPower PCAMD K6AMD K7AMD x86-64IBM PowerSUN UltraSPARCIntel Core 2AMD Opteron
Numbers courtesy of Mark Horowitz, Ofer Shacham
4%/year sounding pretty good
Ben Zorn CGO 2010 Keynote
![Page 12: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/12.jpg)
PLDI Performance Paper Decline
0
5
10
15
20
25
30
35
40
45
50
1986 1989 1991 1993 1995 1997 1999 2001 2003 2005 2007 2009
Pap
ers
Pu
blis
he
d
Correctness
Other
Performance
What Happened?
14 Ben Zorn CGO 2010 Keynote
![Page 13: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/13.jpg)
Performance is Dead
15 Ben Zorn CGO 2010 Keynote
![Page 14: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/14.jpg)
Code Red July 2001 359k hosts, 1 day
What Killed Performance?
Nimda September 2001 Became largest worm in 22 minutes
Slammer January 2003 Infected 90% of vulnerable hosts in < 10 minutes
16 Ben Zorn CGO 2010 Keynote
![Page 15: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/15.jpg)
Companies Shift Gears
• Correctness and security a major new focus
• Microsoft investments: – PREfix, PREfast, SDV (Slam), ESP
– Large code bases automatically checked for correctness errors (10+ million LOC)
• “Combined, the tools [PREfix and PREfast] found 12.5% of the bugs fixed in Windows Server 2003” – “Righting Software”, Larus et al., IEEE Software, 2004
17 Ben Zorn CGO 2010 Keynote
![Page 16: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/16.jpg)
Researchers Shift Gears
• Ben’s research agenda changes • 1990s
– Predicting object lifetime and locality (with David Barrett and Matt Seidl)
– Branch Prediction (with Brad Calder et al.) – Value Prediction (with Martin Burtscher)
• 2000s –tough sounding project names – DieHard – with Emery Berger, Gene Novark – Samurai – with Karthik Pattabiraman – Nozzle – with Ben Livshits
18 Ben Zorn CGO 2010 Keynote
![Page 17: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/17.jpg)
a
The New Threat: Exploitable Memory Corruptions
19
c
0 99
p1
0 99
p2
x
• Buffer overflow char *c = malloc(100);
c[101] = ‘a’;
• Use after free char *p1 = malloc(100);
char *p2 = p1;
free(p1);
p2[0] = ‘x’;
Ben Zorn CGO 2010 Keynote
![Page 18: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/18.jpg)
Strategies for Avoiding Memory Corruptions
• Rewrite in a safe language (Java, C#, JavaScript)
• Static analysis / safe subset of C or C++
– SAFECode [Adve], etc.
• Runtime detection, fail fast
– Jones & Lin, CRED [Lam], CCured [Necula], others…
• A New Approach: Tolerate Corruption and Continue
– Failure oblivious computing [Rinard] (unsound)
– Rx, Boundless Memory Blocks, ECC memory
– DieHard / Exterminator, Samurai
20 Ben Zorn CGO 2010 Keynote
![Page 19: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/19.jpg)
Correctness at What Cost?
• Heap implementations are/were maximally brittle for performance
• Space: packed as tightly as possible
• Time: reuse freed objects as soon as possible
– free = push malloc = pop
21
freelist freelist
Ben Zorn CGO 2010 Keynote
![Page 20: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/20.jpg)
DieHard Allocator in a Nutshell • With Emery Berger (PLDI 2006) • Existing heaps are brittle,
predictable – Predictable layout is easier for
attacker to exploit
• Randomize and overprovision the heap – Expansion factor determines how
much empty space – Semantics are identical – Allocator is easy to replace
• Replication increases benefits • Exterminator extended ideas
(PLDI 2007, Novark et al.)
Normal Heap
DieHard Heap
22 Ben Zorn CGO 2010 Keynote
![Page 21: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/21.jpg)
Of Course, Performance Matters
0
0.5
1
1.5
2
2.5
No
rma
lize
d E
xe
cu
tio
n T
ime
GNU libc Exterminator
allocation-intensive SPECint2000
23 Ben Zorn CGO 2010 Keynote
![Page 22: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/22.jpg)
DieHard Impact • DieHard (non-replicated)
– Windows, Linux version implemented by Emery Berger
– Works in FireFox distribution without any changes
– Try it right now! (http://www.diehard-software.org/)
• RobustHeap – Microsoft internal version implemented by Ted Hart
– Prototyped in Microsoft products
– Demonstrated to tolerate faults and detect errors
• Windows 7 Fault Tolerant Heap (FTH) – Inspired by ideas from DieHard/Robustheap
– Turns on when application crashes
24 Ben Zorn CGO 2010 Keynote
![Page 23: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/23.jpg)
A Benefit of Working at Microsoft…
One day I was trying to convince a security team that DieHard would improve security… They said “What about heap spraying?” And I said “What’s that?” (long pause) And they said “Look it up…”
25 Ben Zorn CGO 2010 Keynote
![Page 24: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/24.jpg)
Here’s What I Found…
Firefox 3.5 July 14, 2009
http://www.web2secure.com/2009/07/mozilla-firefox-35-heap-spray.html
Adobe Acrobat/Reader July 23, 2009
Adobe Acrobat / Reader February 19, 2009
Flash July 23, 2009
http://blog.fireeye.com/research/2009/07/actionscript_heap_spray.html
Common Element: All vulnerable applications support embedded scripting languages (JavaScript, ActionScript, etc.)
26 Ben Zorn CGO 2010 Keynote
![Page 25: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/25.jpg)
Drive-By Heap Spraying
Owned!
27 Ben Zorn CGO 2010 Keynote
![Page 26: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/26.jpg)
Drive-By Heap Spraying (2)
<HTML>
<SCRIPT language="text/javascript">
shellcode = unescape("%u4343%u4343%...'');
</SCRIPT>
<IFRAME
SRC=file://BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB …
NAME="CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC …
഍഍">
</IFRAME>
</HTML>
ok
bad
ok
Creates the malicious object
Triggers the jump
Program Heap ASLR prevents the
attack
PC
28 Ben Zorn CGO 2010 Keynote
![Page 27: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/27.jpg)
Drive-By Heap Spraying (3)
<SCRIPT language="text/javascript">
shellcode = unescape("%u4343%u4343%...'');
oneblock = unescape("%u0C0C%u0C0C");
var fullblock = oneblock;
while (fullblock.length<0x40000) {
fullblock += fullblock;
}
sprayContainer = new Array();
for (i=0; i<1000; i++) {
sprayContainer[i] = fullblock + shellcode;
}
</SCRIPT>
ok
bad
ok
Program Heap
bad
bad
bad
bad
bad
Allocate 1000s of malicious objects
29 Ben Zorn CGO 2010 Keynote
![Page 28: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/28.jpg)
Nozzle – Detecting Heap Spraying
• Joint work with Paruj Ratanaworabhan (Kasetsart University) and Ben Livshits (Microsoft Research)
• Insight:
– Spraying creates many objects with malicious content
– That gives the heap unique, recognizable characteristics
• Approach:
– Dynamically scan objects to estimate overall malicious content
30 Ben Zorn CGO 2010 Keynote
![Page 29: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/29.jpg)
Nozzle: Classifying Malicious Objects
Application Threads Nozzle Threads
Application Heap
new object
create object
scan object and classify
suspect object
Repeat
suspect object
benign object benign
object
benign object
suspect object benign
object
31 Ben Zorn CGO 2010 Keynote
![Page 30: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/30.jpg)
Local Malicious Object Detection
Code or Data?
Is this object dangerous?
• Is this object code?
– Code and data look the same on x86
• Focus on sled detection
– Majority of object is sled
– Spraying scripts build simple sleds
• Is this code a NOP sled?
– Previous techniques do not look at heap
– Many heap objects look like NOP sleds
– 80% false positive rates using previous techniques
• Need stronger local techniques
32
000000000000
000000000000
000000000000
000000000000
000000000000
000000000000
000000000000
add [eax], al
add [eax], al
add [eax], al
add [eax], al
add [eax], al
add [eax], al
add [eax], al
0101010101
0101010101
0101010101
0101010101
0101010101
0101010101
0101010101
and ah, [edx]
and ah, [edx]
and ah, [edx]
and ah, [edx]
and ah, [edx]
and ah, [edx]
and ah, [edx]
32
NOP sled
shellcode
Ben Zorn CGO 2010 Keynote
![Page 31: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/31.jpg)
Object Surface Area Calculation
• Assume: attacker wants to reach shell code from jump to any point in object
• Goal: find blocks that are likely to be reached via control flow
• Strategy: use dataflow analysis to compute “surface area” of each block
33 33
An example object from visiting google.com
Ben Zorn CGO 2010 Keynote
![Page 32: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/32.jpg)
Nozzle Effectiveness
Logical time (number of allocations/frees)
No
rmal
ized
Su
rfac
e A
rea Malicious Page
Normal Page
Application: Web Browser
34 Ben Zorn CGO 2010 Keynote
![Page 33: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/33.jpg)
Nozzle Performance
35 35 Ben Zorn CGO 2010 Keynote
![Page 34: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/34.jpg)
So, Performance is Dead…
36
How far can defect detection and runtime toleration go?
% A
ll “C
riti
cal”
Def
ects
Det
ecte
d
100
50
1970 1980 1990 2000 2010
How much headroom is left
for improvement?
Testing, code reviews
Testing automation, fuzzing, extreme programming…
Static analysis, verification
DART, safe languages, etc.
Future challenges: - Diminishing returns - Scaling verification - 3rd-party library code - Performance implications
0
Ben Zorn CGO 2010 Keynote
![Page 35: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/35.jpg)
What’s Happening Here? Browser Market Share Trends
37
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Ap
ril,
200
8
May
, 20
08
Jun
e, 2
008
July
, 20
08
Au
gust
, 20
08
Sep
tem
ber
, 20
08
Oct
ob
er, 2
00
8
No
vem
ber
, 20
08
Dec
emb
er, 2
00
8
Jan
uar
y, 2
00
9
Feb
ruar
y, 2
009
Mar
ch, 2
00
9
Ap
ril,
200
9
May
, 20
09
Jun
e, 2
009
July
, 20
09
Au
gust
, 20
09
Sep
tem
ber
, 20
09
Oct
ob
er, 2
00
9
No
vem
ber
, 20
09
Dec
emb
er, 2
00
9
Jan
uar
y, 2
01
0
Feb
ruar
y, 2
010
Mar
ch, 2
01
0
Other
FireFox
IE
Source: http://marketshare.hitslink.com/
Can we explain this?
Security?
Reliability?
Features?
Performance!
Ben Zorn CGO 2010 Keynote
![Page 36: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/36.jpg)
Long Live Performance!
38
“Safari dominates browser benchmarks”
http://news.zdnet.com/2100-9595_22-272792.html
“Browser faceoff: IE vs Firefox vs Opera vs Safari”
Kai Schmerer, ZDNet Germany on May 29th, 2008
http://www.favbrowser.com/chrome-vs-opera-vs-firefox-vs-internet-explorer-vs-safari/
“Browser Wars: Ultimate Browser Benchmark…”
Performance can make or break a platform
Ben Zorn CGO 2010 Keynote
![Page 37: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/37.jpg)
One Word:
39
JavaScript
Standard for scripting web applications
Fast JITs widely available
Support in every browser
Lots of code present in all major web sites
Ben Zorn CGO 2010 Keynote
![Page 38: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/38.jpg)
Goal: Measure JavaScript in real web applications Approach: Instrument IE runtime
Understanding JavaScript Behavior
Maps Maps
7 V8 programs:
• richards
• deltablue
• crypto
• raytrace
• earley-boyer
• regexp
• splay
8 SunSpider programs:
• 3-draytrace
• access-nbody
• bitops-nsieve
• controlflow
• crypto-aes
• date-xparb
• math-cordic
• string-tagcloud
JSMeter
With Paruj Ratanaworabhan and Ben Livshits
Benchmarks Real apps
40 Ben Zorn CGO 2010 Keynote
![Page 39: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/39.jpg)
Real Apps are Much Bigger
42
0
500
1000
1500
2000
2500
Sou
rce
siz
e (
kilo
byt
es)
Benchmarks Real apps
Gmail delivers more than 2 megabytes of source code to your browser
Ben Zorn CGO 2010 Keynote
![Page 40: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/40.jpg)
Real Apps have Interesting Behavior: Live Heap over Time (eBay)
43
Heaps repeatedly created, then discarded
Heap contains mostly functions
Ben Zorn CGO 2010 Keynote
![Page 41: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/41.jpg)
Real Apps have Different Architectures
Bing (Web 2.0)
Google (Web 1.0)
Code|Objects|Events
44
You stay on the same page during your entire visit Code loaded once Heap is bigger
Every transition loads a new page Code loaded repeatedly Heap is smaller
Ben Zorn CGO 2010 Keynote
![Page 42: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/42.jpg)
The Next 10 Years
• Reliability
• “Good enough” = cheap
• Energy
• Concurrency
45 Ben Zorn CGO 2010 Keynote
![Page 43: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/43.jpg)
46
Reliability Threats
Silicon Defects (Manufacturing defects and device wear-out)
Manufacturing Defects That Escape Testing
(Inefficient Burn-in Testing)
Parametric Variability (Uncertainty in device and environment)
Transient Faults due to Cosmic Rays & Alpha Particles
(Increase exponentially with number of devices on chip)
Increased Heating
Higher Transistor Leakage
Thermal Runaway
Higher Power Dissipation
N+ N+
Source DrainGate
P--+
-+
-+-+
-+
H/W and S/W Design Errors (Bugs are expensive and expose security holes)
Intra-die variations in ILD thickness
Slide courtesy of Todd Austin “Reliable Processor Research @ Umich” Ben Zorn CGO 2010 Keynote
![Page 44: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/44.jpg)
The “Good Enough” Revolution Source: WIRED Magazine (Sep 2009) – Robert Kapps
• Observation: People prefer “cheap and good enough” over “costly and near-perfect”
• Examples: Flip video cameras, Skype, etc. • Conclusion:
• Engineer for imperfect result at low cost • Projects: Green (Chilimbi, MSR), Perforation
(Rindard, MIT), Flicker (Pattabiraman, UBC)
47 Ben Zorn CGO 2010 Keynote
![Page 45: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/45.jpg)
Energy
48
“Estimating Total Power Consumption by Servers in the U.S. and the World”, Jonathan G. Koomey, LBL Report, Feb. 2007
1.2% of 2005 US electricity sales
0.8% of 2005 world electricity sales
Tota
l Ele
ctri
city
Use
(b
illio
ns
kWh
/yea
r)
World US
Coo
Cooling and auxiliary equipment High-end servers Mid-range servers Volume servers
Ben Zorn CGO 2010 Keynote
![Page 46: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/46.jpg)
Conclusions
• Performance was and continues to be critical – Correctness and security neglected until 2000s
• What is being optimized changes – Energy usage
– Concurrency
– Cost effectiveness
– Constrained devices
• Improvements in next 10 years harder – Proebsting’s Law: Accurate? Acceptable?
49 Ben Zorn CGO 2010 Keynote
![Page 47: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/47.jpg)
Acknowledgements
• CGO Organizers (especially Kim Hazelwood and David Kaeli)
• Todd Austin, U. Michigan
• Alex David, Deborah Robinson – Microsoft
• Mark Horowitz, Ofer Shacham – Stanford
• CJ Newburn, Shubu Mukherjee – Intel
• Karthik Pattabiraman – UBC
• DBLP Computer Science Bibliography – Universität Trier
50 Ben Zorn CGO 2010 Keynote
![Page 48: Performance is Dead, Long Live Performance · SPECint2006 CPU Performance 6 0.01 0.10 1.00 10.00 100.00 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 Year of Introduction](https://reader034.vdocuments.site/reader034/viewer/2022052010/601fbef3c6b8ad769c6a2c56/html5/thumbnails/48.jpg)
Questions?
51 Ben Zorn CGO 2010 Keynote