![Page 1: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/1.jpg)
The impact of Spectre and MeltdownBen Bromhead, CTO, InstaclustrJanuary 2018
![Page 2: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/2.jpg)
Agenda• An introduction to Speculative Execution, Spectre and
Meltdown• Implemented fixes• Some graphs on the impact
2
![Page 3: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/3.jpg)
Intro to Spectre and Meltdown
3
![Page 4: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/4.jpg)
Intro to Spectre and Meltdown
4
• Three exploits
• Target information leakage in speculative execution by CPUs.
• One is vendor specific (Intel).
• The other two impact most CPUs that implement speculative execution.
• Despite existing for 20+ years, 10+ years of theorised existence, bugs were
discovered independently by multiple teams within a 6 month window.
• Fundamentally the concepts exploit three related elements of CPU and OS
design:
• Kernel Memory Mapping
• CPU L1, L2 and L3 Cache behaviour
• CPU behavior to minimise stall time (Speculative execution and branch
prediction).
![Page 5: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/5.jpg)
Memory Protection
5
CPURing3
47 bit
64 bit
Kernel Space
32 bit
Mem Map area
Heap
Virtual Memory space (not including ASLR offsets)
![Page 6: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/6.jpg)
Memory Protection
6
![Page 7: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/7.jpg)
Memory Protection
7
CPURing0
47 bit
64 bit
Kernel Space
32 bit
Mem Map area
Heap
Process Virtual Memory space (not including ASLR offsets)
![Page 8: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/8.jpg)
Memory Protection
8
CPURing3
47 bit
64 bit
Kernel Space
32 bit
Mem Map area
Heap
Virtual Memory space (not including ASLR offsets)
Segmentation Fault!
![Page 9: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/9.jpg)
Out of order execution
9Courtesy - https://meltdownattack.com/meltdown.pdf
![Page 10: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/10.jpg)
Cache behavior
10Courtesy - https://medium.com/@mattklein123
![Page 11: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/11.jpg)
Putting it all together - Meltdown
11
1| raise_exception(); 2| // the line below is never reached 3| access(probe_array[data * 4096]);
![Page 12: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/12.jpg)
Putting it all together - Spectre
12
1| if (x < array1_size) 2| y = array2[array1[x] * 256];
Instructions
..{inst - x > array}
..{inst - x < array}
..{inst - x < array}
..{inst - x < array}
..{inst - x < array}
..{inst - x < array}
CPU Guesses wrong!
CPU
Correct values of x that train branch predictor to guess true
![Page 13: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/13.jpg)
Putting it all together
13
1| raise_exception(); 2| // the line below is never reached 3| access(probe_array[data * 4096]);
![Page 14: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/14.jpg)
Meltdown vs Spectre• Both leverage CPU speculative execution
• Meltdown - out of order, non dependent execution• Spectre - branch prediction
• Both exfiltrate data via timing of cache access• Meltdown can access protected memory (including that
of a hypervisor as well as kernel) due to Intel not checking the privilege bit upon instructions executed speculatively
14
![Page 15: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/15.jpg)
15Context Switching (Kernel mapped into all user process spaces)
Kernel Space
Google Chrome
Meltdown vs Spectre
1| raise_exception(); 2| // the line below is never reached 3| access(probe_array[data * 4096]);
1| if (x < array1_size) 2| y = array2[array1[x] * 256];
![Page 16: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/16.jpg)
The fixesSpectre variant 1 - Recompile everythingSpectre variant 2 - CPU microcode updates OR Google
RetpolineMeltdown - Linux patchset call KPTI (Kernel Page
Isolation)
16
![Page 17: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/17.jpg)
KPTIIsolates Kernel page table from user space (still in physical
memory, just not process page table)
17
Kernel Space
Google Chrome
![Page 18: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/18.jpg)
KPTI Performance impact• KPTI unmaps the majority of the kernel virtual memory
while user code is on CPU.• Performance impact of this swap estimated to be
between 5 - 30% depending on the number of system calls a process makes.
• How does this impact Cassandra?
18
![Page 19: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/19.jpg)
KPTI Performance impact
19
![Page 20: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/20.jpg)
KPTI Performance impact
20
![Page 21: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/21.jpg)
KPTI Performance impact
21
![Page 22: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/22.jpg)
KPTI Performance impact
22
![Page 23: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/23.jpg)
Wait what• During the first round of testing we noticed a slight
increase in performance AFTER upgrading. • We observed a similar increase in performance across
our fleet of managed nodes and concluded this was due to a patch made to the underlying AWS hypervisors and not related to the guest OS upgrades - the tests were repeated to confirm results.
23
![Page 24: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/24.jpg)
Impact after resolved AWS patches
24
![Page 25: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/25.jpg)
Impact after resolved AWS patches
25
![Page 26: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/26.jpg)
Real world impact after patches
26
Over our AWS fleet of 1500 nodes (m4.xl, r4.xl and i3.2xl) serving real world production workloads● Slight increase to no increase in CPU utilisation
(especially in the cloud)● Minimal impact on production environments ● Slight increase in base line latency● Most of this stuff appears to be within a certain
margin of error anyway
![Page 27: The impact of Spectre and Meltdown - Instaclustr · 2018-02-06 · 2. Intro to Spectre and Meltdown 3. Intro to Spectre and Meltdown 4 • Three exploits • Target information leakage](https://reader034.vdocuments.site/reader034/viewer/2022050416/5f8cadcccd874e2c913d1cd5/html5/thumbnails/27.jpg)
What about the other benchmarks?A number of companies conducted Cassandra benchmarking including:• The Last Pickle• Datastax• Scylla
All showed an impact at a throughput maximum. E.g. a cluster pushed to its absolute limit.
27