Fine grain Cross-VM Attacks on Xen and VMware are possible!

Gorka Irazoqui Apecechea, Mehmet Sinan Inci, Thomas Eisenbarth, Berk Sunar ∗

Worcester Polytechnic Institute{girazoki,msinci,teisenbarth,sunar}@wpi.edu

AbstractThis work exposes further vulnerabilities in virtualizedcloud servers by mounting Cross-VM cache attacks inXen and VMware VMs targeting AES running in thevictim VM. Even though there exists a rich literature oncache attacks on AES, so far only a single work, demon-strating a working attack on an ARM platform runninga L4Re virtualization layer has been published. Herewe show that AES in a number popular cryptographiclibraries including OpenSSL, PolarSSL and Libgcrypt

are vulnerable to Bernstein’s correlation attack when runin Xen and VMware (bare metal version) VMs, the mostpopular VMs used by cloud service providers (CSP) suchas Amazon and Rackspace. We also show that the vul-nerability persists even if the VMs are placed on differ-ent cores in the same machine. The results of this studyshows that there is a great security risk to AES and (dataencrypted under AES) on popular cloud services.

1 Motivation

In as little as a decade cloud computing and cloud en-abled applications have become mainstream running ITservices of many businesses and have even personal com-puting through popular applications such as cloud basedstorage, e.g. Dropbox, and media sharing and deliv-ery systems, e.g. Netflix. The biggest concern prevent-ing companies and individuals from further adopting thecloud is data security and personal privacy. Systems run-ning on the cloud use libraries designed for a single-userserver-world, where adversaries can only interact overwell-defined network interfaces. In contrast, in the cloud,malicious code may be running on the same machineas the crypto stack, separated only by a virtual machinemanager (VMM). Indeed, the main security principle inthe design and implementation of VMMs has been that

∗This material is based upon work supported by the National Sci-ence Foundation under Grant No. #1318919.

of process isolation achieved through sandboxing. How-ever, sandboxing has traditionally been defined in thesoftware space ignoring leakages of information throughsubtle side-channels shared by the processes running onthe same physical hardware. In non-virtualized environ-ments, a number of effective side-channel attacks wereproposed that succeeded in extracting sensitive data bytargeting the software layer only. For instance, the earlyproposal by Bernstein [5] (and later in [8, 12]) targetsthe time variation in accessing the cache to extract AESencryption keys while the proposal by Acıicmez targetsthe key dependent time variations in the hardware branchprediction unit [1].

Given the level of access to the server and control ofprocess execution required to realize a successful phys-ical attack, these attacks have been largely dismissed inthe cloud setting until fairly recently.

As early as in 2003, D. Brumley and Boneh [10]demonstrated timing side-channel vulnerabilities inOpenSSL 0.9.7 [22], in the context of sandboxed ex-ecution in VMMs. The authors ran several experimentswith processes running on the same processor and man-aged to recover RSA private keys from an OpenSSL-based web server using packet timing information. Asa consequence of this attack RSA implementations inOpenSSL use blinding to counter timing attacks. Nev-ertheless, as recently as in 2011, B.B. Brumley and Tu-veri [9] demonstrated that the ladder computation oper-ation in the ECDSA computation [6] is vulnerable to atiming attack. The authors were able to steal the privatekey used for server authentication in a TLS handshakeimplemented using OpenSSL 0.9.8o.

Later, in 2011, Suzaki et al. [20, 21] exploited anOS-optimization, namely KSM, to recover user data andsubsequently identify a user from a co-hosted guest OShosted in a Linux Kernel-based Virtual Machine (KVM).

In 2012, Ristenpart et al. [19] demonstrated that it ispossible to solve the logistics problems and extract sen-sitive data across VMs. Specifically, using the Amazon

EC2 service as a case study, Ristenpart et al. demon-strated that it is possible to identify where a particulartarget VM is likely to reside, and then instantiate newVMs until one becomes co-resident with the target VM.Even further, the authors show how cache contention be-tween co-located Xen VMs may be exploited to deducekeystrokes with high success probability. By solving theco-location problem, this initial result fueled further re-search in Cross-VM side-channel attacks.

Shortly later, Zhang et al. [27] presented an access-driven side-channel attack implemented across Xen VMsthat manages to extract fine-grain information from a vic-tim VM. More specifically, the authors manage to re-cover an ElGamal decryption key from a victim VM us-ing a cache timing attack. To cope with noise and therebyreduce the errors, the authors utilize a hidden Markovmodel. The significance of this work, is that for thefirst time the authors were able to extract fine grain in-formation across VMs, in contrast to the earlier work ofRistenpart et al. [19] who managed to extract keystrokepatterns. Yarom et al. [26] describe a trace-driven flushand reload attack and note that it could be applied in vir-tualized environments. In [24] Weiß et al for the firsttime present a cache timing attack on AES in a L4ReVM running on an ARM Cortex-A8 processor inside aFiasco.OC microkernel. The attack is realized usingBernstein’s correlation attack and targets several popu-lar AES implementations including the one in OpenSSL.The significance of this work is that it showed that it ispossible to extract even finer grain information (AES vs.ElGamal keys in [27]) inside a VM.

In this work, we push the envelope even further by uti-lizing Bernstein’s correlation attack (and its last roundvariation) to recover AES keys from popular crypto-graphic libraries in Xen and VMWare virtualized en-vironments. Our work shows that it is possible to re-cover fine grain information, i.e. AES keys, in com-monly used cryptographic software packages, runninginside and across Xen and VMware, even if the VMs arelocated on different cores on the same machine. We re-peat the experiments, by running the attack natively, in-side the same VM and Cross-VM—establishing that formost cases the attack success degrades only mildly whenthe target and victim processes are taken from native tovirtualized to Cross-VM execution.

2 Background

Our goal is to set up a testbed that allows us to evalu-ate whether fine-grain cross-VM attacks can be appliedin a real world scenario. As previously mentioned, thefirst fine-grained side-channel attack on a server systemwas Bernstein’s attack on AES [5]. It exploits data-dependent timing-behavior that is introduced by the un-

derlying cache architecture of the host system. Bern-stein’s attack has often been disregarded as impracticalwhen performed over a network (e.g. in [16]). How-ever, due to the quickly increasing popularity of cloudservices, timing noise stemming from the network con-nection is no longer an obstacle. In public clouds, de-termined adversaries are able to place malicious hosts onthe same machine as a target host [19]. The malicioushosts can then monitor and manipulate shared resources,including caches and other hardware resources in orderto extract critical information from the target. In the fol-lowing we will review the cache architecture as well asattacks that exploit its behavior. We will focus on Bern-stein’s attack, as it gives us a simple tool to study andcompare the viability of fine-grained cross-VM attacksin a realistic attack scenario.

2.1 Cache Side Channel Attacks

Cache Architecture The cache architecture consistsof a hierarchy of memory components located betweenCPU cores and RAM. Its purpose is to reduce the av-erage access time to the main memory. When the CPUneeds to fetch data from memory, it queries the cachememory first to check if the data is in the cache. If it is,then it can be accessed with much smaller delay and inthis case we say that a cache hit occurred. On the otherhand, if the data is not present in the cache, it needs tobe fetched from a higher-level cache or maybe even fromthe main memory which results in greater delays. We re-fer to this as a cache miss. When a cache miss occurs,the CPU retrieves the data from the memory and stores itinto the cache. This action is motivated by the temporallocality principle: recently accessed data is likely to beaccessed again soon.

However the CPU is going to take advantage of spa-tial locality as well: when a data is accessed, the valuesstored in close locations to the accessed data are likelyto be accessed again. Therefore, when a cache miss oc-curs it makes sense to load not only the missed data, butan entire block which includes data in nearby locationsalso. Therefore, the cache is organized into fixed sizecache lines, e.g of l bytes each. A cache line representsthe partitions of the data that can be retrieved or writtenat a time when accessing the cache.

When an entry of a table stored in memory is accessedfor the first time, the memory line containing the re-trieved data is loaded into the cache. If the algorithmaccesses data from the same memory line again, the ac-cess time is lower since the data is located in the cache,not in the memory, i.e. a cache hit occurs. Therefore theencryption time will depend directly on the accessed ta-ble positions, which in turn depend on the secret internalstate. This fact can be exploited to gain information of

2

the used secret key. In case, there are no empty (invalid)cache lines available, one of the data bearing lines needsto be reallocated to the incoming line. Therefore, cachelines not recently accessed are evicted from the cache.

The position that a certain memory line occupies inthe cache depends on the cache structure. There are threekinds of cache structures: fully-associative, direct map-ping and set-associative. The fully-associative structurestates that a memory line can occupy any of the cachelines of the cache. The direct mapping policy states thateach memory line has a fixed position in the cache. Fi-nally, the set-associative policy divides the cache into ssets, each containing k lines. With this structure a mem-ory line has to be loaded in one specific set, but it canoccupy any of the k lines in that set.

There are three main families of cache-based sidechannel attacks: time driven, trace driven, and accessdriven cache attacks. The difference between them arethe capabilities of the attacker. Time driven attacks arethe least restrictive ones. The only assumption is that theattacker can observe the aggregated timing profile of afull execution of a target cipher. Trace driven attacks areassumed to be able to catch the cache profile when thetargeted program is running. Finally, access driven at-tacks assume only to know which sets of the cache havebeen accessed during a execution of a program.

Cache Leakage for AES The run-time of fast softwareimplementations of ciphers like the AES often heavilydepends on the speed at which table look ups are per-formed. A popular implementation style for the AESis the T-table implementation of AES [11]. It combinesthe SubBytes, ShiftRows and MixColumns operationsinto one single table look up per state byte along withxor operations. Compared to standard S-boxes, T-tablebased implementations use more memory, but the en-cryption time is significantly faster, especially on 32-bitCPUs. Because of this almost all current software imple-mentations of AES for high-performance CPUs are T-table implementations. Please note that the index of theloaded table entry is determined by a byte of the cipherstate. Hence, information on which table values havebeen loaded into cache can reveal information about thesecret state of AES. Such information can be retrievedby monitoring the cache directly, as done in trace-drivencache attacks. Similar information can also be learnedby observing the timing behavior of an AES execution.Above we ascertained that a cache miss it takes moretime to access the data than a cache hit. Hence, the lookup of values from a table will take different time, de-pending on whether the accessed data has been loaded toa cache before or not.

Related Attacks on AES Kocher showed in his 1996study that data-dependent timing behavior of crypto-graphic implementations can be exploited to recover se-cret keys [13]. It took almost ten years for the crypto-graphic community to become aware of the fact that thatthe microarchitecture of modern CPUs can introduce ex-ploitable timing behavior into seemingly constant run-time implementations such as AES. Bernstein showedin [5] that timing-dependencies introduced by the datacache allow key recovery for the popular T-table imple-mentations of AES [11]. Subsequent papers exploredreasons [16], improved attack techniques such as colli-sion attacks [8, 2] and attacks targeting last round [7].In [17] new cache-specific attack styles, such as access-driven cache attacks are explored. Mowery et al. ana-lyze in [15] the effect of new microarchitectural featurescommonly present in modern CPUs on Bernstein’s at-tack. An example of access-driven cache attack in AESis the one proposed in [26, 12] where Gullasch et al flushthe desired data from the cache prior to the encryptionand then at some point of the encryption, they try to loadthe flushed data again. If the data has been accessed inthe encryption it takes less time to access it (since it is inthe cache).

Dag Osvik et al describe a trace driven attack calledprime+probe [17]. The prime and probe technique fillsthe cache before the encryption and after it checks whichsets have been accessed by reloading the data. Moreoverin the same paper we find a description of a time drivenattack called evict+time. The approach is simple: Thecache is evicted before the encryption and then the timeof the encryption is measured.

Only little work has been performed on the influenceof virtualization on cache-based side-channel leakage. Ina real virtualized scenario, we only know about MichaelWeiss research in [24], where they tried to attack AESon ARM processors. To the best of our knowledge, thiswork is the first to analyze the side-channel cache attackvulnerability of current AES implementations on modernx86-type CPUs in virtualized environment.

2.2 Bernstein’s attack

Bernstein’s attack [5] is a four step side-channel cache at-tack on the AES. Bernstein originally proposed his attackin the non-virtualized server-client setting, however, weadapt it here to the virtualized setting to serve the pur-poses of this paper. The attack mainly consists of fourstages:

• Profiling Stage In the profiling stage, the attackercreates a duplicate profile server either on theattacker’s own VM or on a third VM on the samephysical machine. After the server is run with

3

Figure 1: Output of Bernstein’s attack after the correlation phase.

a known key, the profile server encrypts theseplaintexts and sends back the timing information ofthe encryption to the attacker. The received timinginformation is put into a timing table T of size16×256 that holds the timing information for eachbyte of the key for all possible key values. Notethat the table is initialized with all zero values. Newvalues are added and then averaged for each entry.So, for the learning stage, t is calculated for eachi, j such that 0 ≤ i < l and 0 ≤ j < 16 , where l isthe number of plaintexts sent to the server. Thisway, the following equation is calculated contin-uously until the desired sample number l is reached.

T [ j][p j, i] := T [ j][p j, i]+ time(AES(pi,k)) (1)

• Attack Stage In the attack stage the attacker Csends known plaintexts to the target server S justlike in the study stage but with a slight difference.Unlike the study stage, the secret key k in this stagein unknown to the attacker. So during this stage,the attacker profiles the timings for known plain-texts with an unknown key and records the timinginformation for each plaintext.

• Correlation Stage In the correlation stage, timingprofiles from the first two stages are correlated anda new table is created. This new table holds a cor-relation coefficient for every possibility of each keybyte. Then the elements of this table are sorted start-ing from the possible key with the highest correla-tion coefficient to the one with the lowest correla-tion coefficient. Then using a threshold, low possi-bility key candidates are eliminated and the remain-ing key candidates are displayed. We can see anexample of such output correlation in Figure 1.

• Brute Force Stage In the final stage, all possiblekey combinations are tried using the key candidates

created at the end of the third stage. Since the pos-sible key space has been significantly reduced bythe correlation stage, this stage takes much less timethan a brute-force AES key search.

For different executions of AES, the attacker gets dif-ferent timings. These timings depend on deterministicsystem-based evictions. When an attacker requests anencryption for all possible key bytes with a known key(profiling stage), it profiles the performance of the cachefor a certain entry in the corresponding table look up.Then he profiles the cache performance for an unknownentry in the corresponding table look up, although heknows the plaintext. Since the entries for the table lookups in the first round are just an XOR operation betweenkey and plaintext, he can compute the correlation andfigure out what key has been used.

In a cloud environment the cache attack scenario doesnot change by much. In Figure 2 we see an example of atypical scheme in a cloud. Users set up virtual machineswith different OS and the VMM handles their accessesto the hardware. In this specific scheme we show a fourcore physical with three cache levels, where each corehas private L1 and L2 caches, but the third level cache isshared. Usually the last level cache is attacked since it iswhere the four cores share data, making it suitable for across-VM leakage [26].

2.3 Challenges & Opportunities in VMMemory Management Technologies

Here we briefly summarize key VM memory manage-ment technologies which were developed to improvememory utilization, and memory access performance ingeneral, and to enhance security. We also comment onhow they affect the performance of side-channel attacksin a virtualized server/cloud setting. As it turns out whilesome VM memory management technologies provide a

4

Core 1 Core 4Core 3Core 2

Guest OS Guest OS Guest OS Guest OS

L1 Cache L1 Cache L1 Cache L1 Cache

L2 Cache L2 CacheL2 CacheL2 Cache

L3 Cache

MAIN MEMORY

VMM: XEN, VMWARE, KVM

Figure 2: Hypervisor allocating different virtual ma-chines into the same physical machine

challenge to mounting a cache timing attack, others ac-tually do enable attacks.

Address Space Layout Randomization (ASLR):ASLR is a security technique implemented to avoidbuffer overflow attacks. In a nutshell, the ASLR tech-nique randomizes the memory positions of various mem-ory sections of the program (heap, stack, static and dy-namic libraries) each time the program is executed. Therandomization prevents an adversary from exploiting aweak function that can be exploited in the memory. Mostof the modern operating systems support ASLR and forthe majority of them, e.g. most Linux distributions, Mi-crosoft Windows, is enabled by default. However thereare still some operating systems such as FreeBSD whichdo not support ASLR.

Second Level Address Translation (SLAT): SLATschemes such as Intel’s Extended Page Tables (EPT)and AMD’s Nested Page Tables (NPT) as shown in Fig-ure 3 are used to manage the virtualized memory directlyfrom the processor. Using a larger Translation LookasideBuffer (TLB) with additional logic circuitry inside theprocessor, these schemes provide faster virtual machinememory management by eliminating the intermediarystep between the virtual memory address (VA) and thephysical memory address (PA). Instead of managing thevirtual machine memory through the host operating sys-tem by performing the VA to PA and then PA to MemoryAddress (MA) mapping at the software level, EPT andNPT technologies use a larger TLB that supports bothnative and virtual memory entries to be mapped to the

TLB Fill HardwareVA MA

TLB

VA-PA Mapping

PA-MA Mapping

Guest PT

Extended PT

Figure 3: Nested/Extended Page Tables

memory addresses directly from the guest and the hostoperating systems. As shown in Figure 3 the TLB tablehas the option that indicates if the received data is from avirtual machine or the native machine. Also, if the data isgenerated by a virtual machine, then it is tagged with thatspecific VM’s Address Space Identifier (ASID). Usingthis tag, the TLB can keep track of entries from differentvirtual machines in the physical machine. This methodprovides a significant performance improvement in VMmemory management but also introduces a security riskby giving direct memory access to the guest VMs.

Kernel SamePage Merging (KSM): KSM is the Linuxmemory deduplication scheme that was originally devel-oped to allow the execution of multiple virtual machineson the same physical machine but later was adopted tothe native machines as well and became a standard fea-ture with the Linux kernel v2.6.32. What KSM does issimply to search the memory contents for duplicate en-tries and merge the duplicate pairs into single page en-tries. But instead of going over the whole memory con-tinuously which would be CPU and memory consum-ing, KSM looks for duplicates using a more sophisticatedscheme. When an application is called, its assigned amemory space. If the memory space is marked as merge-able then the KSM adds it to its search space and startsto look for duplicates using two red-black trees method.Using the knowledge that some of the pages are sharedbetween processes and clients, one can steal sensitive in-formation. This information can be part of a process run-ning on the co-resident guest OS such as a visited web-site or even a possible secret key. Suzaki et al [20] havereported that by exploiting the KSM they were able todetect web browsers, visited websites and even a down-loaded file used by the co-hosted guest OS in a LinuxKernel-based Virtual Machine (KVM) system. In an-other study, using a secret memory marker, Suzaki et al

5

Physical Machine

Hypervisor

Private Pages

Shared (Common) Pages

Private PagesPrivate Pages

Virtual Machine Virtual Machine Virtual Machine

Figure 4: Memory Sharing Across VMs

[21] were able to identify a VM user on a processor in amulti-user cloud environment.

Transparent Page Sharing (TPS): Another importantsecurity risk for virtualization is TPS which is imple-mented in both the VMware and the XEN VMMs [23][4]. A high level depiction of the TPS is shown in Fig-ure 4. TPS continuously scans the memory for duplicate4KB sized data regardless of when, where or by whomthe data was created. Duplicate entries are found by firstmatching their hashes and then if there is a hash match,checking their actual data bit by bit. If a duplicate datain memory is found, the VMM automatically discardsone of the duplicates and allows multiple guests to usethe same data until the data is modified by one of theclients. With this sharing, TPS utilizes the system mem-ory in a more efficient manner but at a cost of compro-mising memory space isolation. Similar to the customSLAT case, one of the clients using the same physicalmachine can set traps and collect information on a neigh-boring VM through cache timing attacks. Even thoughthe shared data cannot be modified or corrupted by anadversary, it can be used to reveal secrets about the tar-get VM client. An adversary can fill the cache with thecandidate data used by the target VM to find out whichdata is shared with therefore also used by the other VMclient.

3 AES in Common Crypto Libraries

We believe that cache based side-channel leakage is abig concern for cryptographic libraries. In this sectionwe analyze the software implementations of AES forthree widely used crypto libraries, namely OpenSSL[22],

PolarSSL [18], and Libgcrypt [14]. All studied AESimplementations use the T-tables optimization. However,the implementations differ in the implementation of thelast round. Also, some of the implementations do al-ready contain methods to reduce or nullify cache-basedside channel leakage.

Due to the lack of the MixColumns operation, thelast round has to be handled differently to the precedingrounds. We have encountered three different implemen-tation styles: The classical approach is to have a special256×32-bit table T4; alternatives are a 256×8-bit S-boxor a reuse of one of the T-tables, followed by maskingand shift operations. While using T4 is very efficient, itcan introduce a particularly strong leakage since it is onlybeing used in the last round. The S-box usage for the lastround to the contrary, is more secure, since each mem-ory line is holding four times more data than the look uptable. Less leakage is introduced when reusing T-tables,since they are used in the 10 rounds of the encryption.

From now on, we will mention first and last round at-tack referring to Bernstein’s attack applied to first andlast round (described in [3]), respectively.

OpenSSL 0.9.7 OpenSSL implements the T-table im-plementation of AES, using T4 for the last round. Thislibrary has been a perfect target for researchers sinceit is highly vulnerable against cache attacks. Althoughoutdated, we decided to include it in our study as areference for an inappropriately protected implementa-tion. Some examples of researchers attacking this libraryare [5, 8, 3]. The advantage that this version of the li-brary gives to attackers is that the last round is a goodtarget, since T4 is being used. Therefore profiling thelast round with Bernstein’s attack should lead to mea-surements with less noise and hence better results. How-ever, we show in the following section that results for thefirst round attack on this libraby are very good as wellwhen comparing to other libraries.

Rounds 1-9: T0,T1,T2,T3.

Last Round: T4;

OpenSSL 1.0.1 The latest version of the OpenSSL li-brary features two different implementations of AES.One of them tries to completely prevent cache-based sidechannel leakage in the first and last rounds. The methodis simple: a simple and small s-box is used in the outer-most rounds. Prefetching is performed before the round.this means that the table is loaded to the cache before thementioned rounds, resulting in constant-time accessesfor all possible look ups.

Prefetch(T4).

First Round: T4.

6

Rounds 2-9: T0,T1,T2,T3.

Prefetch(T4).

Last Round: T4.

The other AES core file has more in common with theolder version. However, it differs in the last round, sinceinstead of using a single and different T-table, the tablesT0 to T3 are reused. Hence, there is no special leakagefor the last round. Instead, the additional round with thesame tables adds more noise to measurements, so we ex-pect the results to be worse than for the older version.This implies more possible candidates for the key, but nomitigation of cache attacks.

Rounds 1-10: T0,T1,T2,T3.

One might wonder which of the two implementationsis used by default? According to OpenSSL source com-ments, the leakage-shielded implementation one is stillexperimental. Second, the makefile in the library com-piles the vulnerable one by default. Finally, we reverseengineered the static library that is installed by default inUbuntu. It is, in fact, the vulnerable one.

PolarSSL 1.3.3 This library uses an implementationbased on the four T-tables for rounds 1-9 and a single S-box for the last round. Unlike T-tables, that hold 4 bytevalues, S-boxes hold 1 byte values andt he entire S-boxoccupies very few cache lines to be profiled. Since theBernstein’s attack is profiling cache lines, PolarSSL’simplementation makes the library a suitable target for thefirst round attack, but not for the last round attack forpractical purposes.

Rounds 1-9: T0,T1,T2,T3.

Last Round: S-box.

Libgcrypt 1.6 The implementation of AES inLibgcrypt also uses the T-table approach for the firstnine rounds. The last round uses only T1. This is a simi-lar scenario as with latest version OpenSSL, where a lastround attack is not likely to recover any key bytes, sincemany of T1 values are already in the cache. A first roundattack makes more sense for this implementation, butmore noise in our measurements comparing to OpenSSL

0.9.7.

Rounds 1-9: T0,T1,T2,T3.

Last Round: T1.

Note that the latest versions of all discussed crypto-graphic libraries support AES-NI instructions. AES-NIis an instruction set extension for Intel CPUs that ac-celerate the execution of AES. According to Intel [25],

Profile Server Attacker Target Server

Figure 5: Our attack setup

the performance is from 3 to 10 times faster when com-pared to state-of-the-art software implementations. AES-NI performs a full round of AES on a 128-bit regis-ter. Hence, data-dependent table look ups are no longernecessary, consequently preventing all known microar-chitectural side channel attacks, including cache attacks.Most of the virtual machine monitors allow a guest OS touse the AES-NI instructions to perform an AES encryp-tion. However, there are still numerous scenarios whereAES-NI is not available, accessible or the implementa-tion might not be aware of its presence.

4 Experiment Setup

Fine-grain cache-based side channel attacks work bestif the target and the attacker are on the same physicalmachine, making a cloud environment a promising andrealistic scenario for the attack. Two well-known andwidely used cloud service providers are the Amazon EC2and the Rackspace services. Both of these CSPs use theXEN VMM to provide virtualization. Also companiesare typically deploying VMware to consolidate a largenumber of servers into a few machines to reduce IT costs.Our study focuses on these two virtualization environ-ments. We assume the attacker to have a copy of thetarget server’s implementation so that the profiling stagecan be performed on in the attacker’s set of VMs. Thisassumption is practical, since there is typically a smallset of popular implementations of certain services. Inour experiments we used Ubuntu Linux, which comeswith OpenSSL and Libgcrypt preinstalled as dynamiclibraries. PolarSSL has to be installed manually, but itis also automatically installed as a dynamic library. Fig-ure 5 gives a visual description of our attack scenario.Attacker (the client in Bernstein’s attack) and the pro-file server are under the adversarys full control. The tar-get server is running in another virtual machine with anunknown secret key. We assume that the attacker hasalready solved the co-location problem [19]. Next, theadversary performs the profiling stage. Since both thetarget and profiling servers are in the same physical ma-chine, they profile the same cache architecture. During

7

the subsequent attack stage, the attacker queries the tar-get server and records plaintexts (or ciphertexts for thelast round attacks) together with the measured timings ofthe AES encryptions. As the final step, the attacker cor-relates both results and recovers the corresponding keybytes.

Timings were obtained using the Intel x86 RDTSC in-struction on the target server, as done in [5, 15, 24]. Al-though practical server implementations will not providetimestamps along with the response messages, the setuptranslates to a practical one for the following reasons:Unlike in classical over-the-network attacks, the inter-domain delay between two VMs in the same physicalmachine is insignificant and almost constant [24]. SinceBernstein’s attack algorithm uses correlation, the con-stant offset is irrelevant. The remaining jitter is expectedto only introduce very little additional noise. However,using the RDTSC instruction minimizes the sources ofnoise in our setup and thus allows us to quantify theamount of leaked information faster, reducing the timespent for each experiment.

Note that even though we are using a Linux based op-erating system and the Linux kernel utilizes the ASLR bydefault, it also automatically aligns dynamic libraries topage boundaries. Since Bernstein’s attack only dependson the page offset of the look up table positions, the ta-bles are aligned from one execution to the other. In otherwords, the attack works in spite of the enabled ASLR.

5 Experiment Results

This section details on our measurement setup and the re-sults obtained for different crypto libraries in different at-tack scenarios. Our main concern is the impact of Bern-stein’s attack when it is translated to a cross-VM attackscenario.

5.1 Results for Native ExecutionWe first analyzed whether the latest versions of the stud-ied libraries are still vulnerable against Bernstein’s at-tack. We attack each of the analyzed libraries with bothBernstein’s first round and last round attack. the resultsare given in the number of bits of the key recovered bythe attack as a function of the number of plaintexts sent tothe server. The measurements where done in two work-stations: a four core I5 2.6 GHz of speed under Ubuntuv13 and an eight core I7 3.2 GHz of speed under Ubuntuv12.04.

In Figure 6 we can observe the resilience of each li-brary against Bernstein’s attack. Note that for OpenSSL0.9.7 major parts of the key were recovered with thefirst round attack and the entire key with last round at-tack (with sufficient plaintexts). For the latest version of

Figure 6: Comparison of Bernstein’s attack for differentlibraries on native machine.

8

Figure 7: Comparison of Bernstein’s attack in native andsingle VM scenarios.

OpenSSL, we see a degradation in the first round attack,and a complete failure in the key recovery for the lastround attack. This fact is due to the design of AES withrespect to look up tables, as explained in Section 3.

We observe a similar behavior for PolarSSL andLibgcrypt. For both libraries the first round attackworks well (it recovers 25% to 50% of the key). How-ever, the last round attack is completely mitigated. Wefurther notice that between the latest versions of the threelibraries, PolarSSL is less vulnerable than OpenSSL andLibgcrypt.

5.2 Results in Single VMs

Next step was to analyze the degradation of Bernstein’sattack when it is moved to a virtual machine. Measure-ments were performed on two workstations, one of themrunning XEN hypervisor, and the other one running aVMware hypervisor. Both machines feature two IntelCore i5 3.20GHz CPUs. The operating system for our at-tacks was Ubuntu 12.04. At this point we only performedprofile and attack stages in the same virtual machine. Theresults are presented in Figure 7.

We can see that there is a degradation of the attackwhen moving it to a virtual machine. This is due to the

Figure 8: Comparation of Bernstein attack in native, sin-gle VM and cross VM scenarios for VMware

increased noise of the virtual machine environment. Incontrast to the case where we recovered almost the wholekey in native machine for OpenSSL0.9.7, we could onlyrecover up to 81 bits in the case of XEN and up to 61 bitsin the case of VMware. We believe that the differencebetween both of the results is due to external noise. Infact, the single VM attack on VMware was the only inthe study that was performed in a noisy environment withtypical user traffic in the background.

Indeed when we analyze the case for OpenSSL 1.0.1

we see a similar behavior for both hypervisors. For thislibrary we also decreased the amount of bits recoveredfrom 58 to 38 in the case of XEN and to 31 in the case ofVMware. Although we did not even recover half of thebits of the key, we believe that recovering 25% of the keyis a reason to be concerned.

5.3 Cross-VM Attack ResultsThe last set of experiments is performed in a cross-VMenvironment with the setup that we described in Sec-tion 4. The setup of our server is identical to the onein the single VM scenario (XEN and VMWARE hyper-visors), but with more VMs. All the VMs created for theprofiling server, target server and attacker have the samesettings and have the same OS, Ubuntu 12.04.

9

Figure 9: Comparation of Bernstein attack in native, sin-gle VM and cross VM scenarios for XEN

The results are presented in Figure 8 and Figure 9. Thefirst graph shows a comparison between the attack run-ning in native machine, single VM running in VMwareand cross-VM running in VMware. The second oneshows the same comparison for the attack in XEN.

In the first plot we can see that again we have moresources of noise in the virtual environment so the dif-ference to the native machine attack is easily observed.However we see similar results in the case of VMwarefor single VM and cross-VM attack. This means thatwe are not loosing any performance by moving the at-tack to a cross-VM scenario (indeed we recovered someadditional bits for cross-VM), although we are we havean increased latency due to not communicating via localsockets.

In the second plot we notice that there is a huge dropfrom single VM to cross VM for OpenSSL 0.9.7. How-ever, the results were better than expected when movingfrom native to virtual machine (where we only lost 10bits). We believe there was very little noise when runningOpenSSL 0.9.7 in single VM. Further work should bedone here to investigate noise sources. In the cross-VMscenario we were still able to recover around 50 bits forOpenSSL 0.9.7. In the case of OpenSSL 1.0.1 it canbe seen that for 2 million encryptions the results for sin-

gle VM and for cross-VM are similar. We conclude thatthe results depend more on different noise for differentruns than on the migration from single to cross-VM.

When comparing the behavior of XEN and VMwarefor the cross-VM attacks, we see similar results. For thelatest version of OpenSSL in both of them between 30and 40 bits were recovered once the number of encryp-tions is high enough.

We want to highlight that apart from discarding somekey byte candidates, the attack sorts the rest of them bytheir posterior likelihood. The attacker can take advan-tage of this additional knowledge by starting the exhaus-tive search of the remaining key space by checking themost likely values for the key bytes first. Figure 1 givesan example output obtained from Bernstein’s attack. Thefirst column refers to the remaining key possibilities afterthe attack; the second column refers to key byte number;in the third column, key byte candidates are sorted ac-cording to their posterior likelihood. In our experimentswe have observed that the correct key is either in the firstor the first two quartiles with a very high likelihood. Ta-ble 1 shows how often the correct key byte value waswithin the first, first two, and first three quartiles, de-pending on the number of remaining candidates. Thetable shows that if the attack was only able to narrowthe space down to 16 or more possibilities, the proba-bility of finding the correct key in the 50% most likelyvalues is of more than 95%. This means that even if aremaining key space seems to be too large to be searchedin its entirety (this usually means remaining lists of 16 ofmore for each key byte, or a remaining key space of morethan 264 possibilities), an attacker might still be able tonarrow down the search space to manageable sizes with-out significantly sacrificing the success probability of theattack.

5.4 Experiment Outcomes

The conclusions we made after analyzing the experi-ments are the following:

• Last round mitigated: We did not obtain any keybyte from the last round attacks in latest versions ofthe libraries.

• First round vulnerable: The first round of thethree libraries analyzed is still vulnerable to cacheattacks.

• Noise in VM scenario: We observed that there isa drop when moving from native machine to virtualmachine. We believe that it is due to more layersthat are introduced (i.e. VMM) in a cloud environ-ment between the guest OS and the hardware.

10

Table 1: Probability of the correct key byte having a high rank within the sorted remaining candidates.

Remaining key byte space Correct key in 1st quartile Correct key in 2nd quartile Correct key in 3rd quartile

1-8 43% 65% 86%8-16 36% 61% 75%16-32 57% 95% 95%32-64 66% 100% 100%64-128 71% 96% 100%128-256 65% 100% 100%

• No degradation when moving to cross-VM: Wedid not observe a degradation in the results due tomoving from a single VM to a cross-VM scenario.

• Noise dependent results: We obtained cleaner re-sults when we were not running other processes to-gether with the attack ( i.e web browsing).

• Bit recovery: We recovered a minimum of 30 bitsfor the cross-VM scenarios, without any optimiza-tion of the exhaustive remaining key search.

6 Countermeasures

It is possible to prevent the data leakage and the resultingcross-vm cache attacks on cloud environment with sim-ple precautions. The countermeasures that we proposeare easy solutions to avoid a problem that can be verypainful when working with sensitive data.

• AES-NI Setting up the virtual machine on a com-puter with AES-NI support and use AES-NI instruc-tions instead of the software implementation. Us-ing AES-NI mitigates completely the cache side-channel attacks since it does not use the mem-ory. Moreover is nearly 10 times faster than thesoftware implementation [25]. Even though thismight seem like an obvious countermeasure, a quicksearch shows that there are still some VMs offeredby popular CSPs that do not have AES-NI hardwaresupport.

• Cache Prefetching Prefetch the tables in to thecache prior to first and last round. When the datais prefetched there is no possibility of attacking ei-ther the first or last rounds. Although this leads toa less noisy environment for a second round attack,the fact that it is much more difficult to implementthe second round attack still makes this a viable op-tion. We already know that the OpenSSL developersare aware of the cache attack vulnerability and weencourage them to work with the experimental filethat they already implemented and included in theOpenSSL 1.0.1‘ distribution. As for Libgcrypt

and PolarSSL a new prefetching scheme for AESshould be implemented to provide protection fromAES cache attack.

• Using AES-256 Use 256-bit key AES encryptioninstead of 128-bit and 196-bit versions. This willlead in more rounds, 12 for the 192-bit and 14 forthe 256-bit versions, and will introduce more noiseto the side channel measurements. We believe thatthe study samples required to analyze a 256-bit AESkey would deem the cache attack impractical andthe amount of the key bytes recovered in such casewould be drastically low.

• Preventing Cartography As noted earlier in [19]another countermeasure would be making the servercartography harder. If the attacker cannot locate thetarget, it is possible to prevent the attack before iteven begins since the attacker has to be on the samephysical machine to perform the attack. Frequentcore-migration might be used as a primary mecha-nism to prevent cartography.

• Avoiding Co-location This countermeasure wascited in [19]. Although this option is not perfor-mance efficient and goes against the idea of cloudbased systems, using separate physical machines foreach customer/client would make the side channelattacks impossible. Even if an adversary discov-ers the target’s IP address and connects to the targetusing this IP address, she still can not perform thecache attack since she won’t have a study file (revisethis) to correlate with the attack timings.

7 Conclusion

We showed that popular cryptographic libraries, namelyOpenSSL, Libgcrypt and PolarSSL have little to noprotection against a cache type attack such as Bernstein‘sattack if the Intel AES-NI instructions are not used. Weshowed that all of the libraries are still susceptible to firstround cache attacks, while last round attacks seem tohave been mitigated in the current versions. When the

11

attack is moved to a virtualized setting, i.e. co-locatedguest OS in Xen and VMware, the attack still succeeds.While cross-VM attacks are noisier, the degradation ismild, and it is still possible to recover fine-grain infor-mation, i.e. an AES encryption key, from a co-locatedguest OS.

References[1] ACIICMEZ, O. Yet another microarchitectural attack:: Exploiting

i-cache. In Proceedings of the 2007 ACM Workshop on ComputerSecurity Architecture (New York, NY, USA, 2007), CSAW ’07,ACM, pp. 11–18.

[2] ACIICMEZ, O., SCHINDLER, W., AND CETIN K. KOC. Cachebased remote timing attack on the aes. In Topics in CryptologyCT-RSA 2007, The Cryptographers Track at the RSA Conference2007 (2007), Springer-Verlag, pp. 271–286.

[3] ATICI, A. C., YILMAZ, C., AND SAVAS, E. An approach forisolating the sources of information leakage exploited in cache-based side-channel attacks. In Software Security and Reliability-Companion (SERE-C), 2013 IEEE 7th International Conferenceon (June 2013), pp. 74–83.

[4] BARHAM, P., DRAGOVIC, B., FRASER, K., HAND, S.,HARRIS, T., HO, A., NEUGEBAUER, R., PRATT, I., ANDWARFIELD, A. Xen and the art of virtualization. ACM SIGOPSOperating Systems Review 37, 5 (2003), 164–177.

[5] BERNSTEIN, D. J. Cache-timing attacks on AES, 2004. URL:http://cr.yp.to/papers.html#cachetiming.

[6] BLAKE, I. F., SEROUSSI, G., AND SMART, N. Elliptic curvesin cryptography, vol. 265. Cambridge university press, 1999.

[7] BONNEAU, J. Robust final-round cache-trace attacks against aes.

[8] BONNEAU, J., AND MIRONOV, I. Cache-collision timing at-tacks against aes. In Cryptographic Hardware and Embed-ded Systems—CHES 2006 (2006), vol. 4249 of Springer LNCS,Springer, pp. 201–215.

[9] BRUMLEY, B. B., AND TUVERI, N. Remote timing attacks arestill practical. In Computer Security–ESORICS 2011. Springer,2011, pp. 355–371.

[10] BRUMLEY, D., AND BONEH, D. Remote timing attacks are prac-tical. Computer Networks 48, 5 (2005), 701–716.

[11] DAEMEN, J., AND RIJMEN, V. The Design of Rijndael.Springer-Verlag, 2002.

[12] GULLASCH, D., BANGERTER, E., AND KRENN, S. Cachegames – bringing access-based cache attacks on aes to practice.In Proceedings of the 2011 IEEE Symposium on Security and Pri-vacy (Washington, DC, USA, 2011), SP ’11, IEEE Computer So-ciety, pp. 490–505.

[13] KOCHER, P. C. Timing attacks on implementations of Diffie-Hellman, RSA, DSS, and other systems. In Advances in Cryp-tology — CRYPTO ’96 (1996), N. I. Koblitz, Ed., vol. 1109 ofLecture Notes in Computer Science, Springer Verlag, pp. 104–113.

[14] LIBGCRYPT. The libgcrypt reference manual. http://www.

gnupg.org/documentation/manuals/gcrypt/.

[15] MOWERY, K., KEELVEEDHI, S., AND SHACHAM, H. Are aesx86 cache timing attacks still feasible? In Proceedings of the2012 ACM Workshop on Cloud Computing Security Workshop(New York, NY, USA, 2012), CCSW ’12, ACM, pp. 19–24.

[16] NEVE, M., AND SEIFERT, J.-P. Advances on access-drivencache attacks on aes. In Selected Areas in Cryptography, E. Bi-ham and A. Youssef, Eds., vol. 4356 of Lecture Notes in Com-puter Science. Springer Berlin Heidelberg, 2007, pp. 147–162.

[17] OSVIK, D. A., SHAMIR, A., AND TROMER, E. Cache at-tacks and countermeasures: The case of aes. In Proceedingsof the 2006 The Cryptographers’ Track at the RSA Conferenceon Topics in Cryptology (Berlin, Heidelberg, 2006), CT-RSA’06,Springer-Verlag, pp. 1–20.

[18] POLARSSL. PolarSSL: Straightforward,secure communication.www.polarssl.org.

[19] RISTENPART, T., TROMER, E., SHACHAM, H., AND SAVAGE,S. Hey, you, get off of my cloud: Exploring information leakagein third-party compute clouds. In Proceedings of the 16th ACMConference on Computer and Communications Security (NewYork, NY, USA, 2009), CCS ’09, ACM, pp. 199–212.

[20] SUZAKI, K., IIJIMA, K., YAGI, T., AND ARTHO, C. Mem-ory deduplication as a threat to the guest os. In Proceedings ofthe Fourth European Workshop on System Security (2011), ACM,p. 1.

[21] SUZAKI, K., IIJIMA, K., YAGI, T., AND ARTHO, C. Softwareside channel attack on memory deduplication. SOSP POSTER(2011).

[22] THE OPENSSL PROJECT. OpenSSL: The open source toolkitfor SSL/TLS. www.openssl.org, April 2003.

[23] WALDSPURGER, C. A. Memory resource management invmware esx server. ACM SIGOPS Operating Systems Review36, SI (2002), 181–194.

[24] WEISS, M., HEINZ, B., AND STUMPF, F. A cache timing at-tack on aes in virtualization environments. In 14th InternationalConference on Financial Cryptography and Data Security (Fi-nancial Crypto 2012) (2012), Lecture Notes in Computer Sci-ence, Springer.

[25] XU, L. Securing the enterprise with intel aes-ni.White Paper, September 2010. http://www.intel.

com/content/www/us/en/enterprise-security/

enterprise-security-aes-ni-white-paper.html.

[26] YAROM, Y., AND FALKNER, K. E. Flush+reload: a high reso-lution, low noise, l3 cache side-channel attack. IACR CryptologyePrint Archive 2013 (2013), 448.

[27] ZHANG, Y., JUELS, A., REITER, M. K., AND RISTENPART,T. Cross-vm side channels and their use to extract private keys.In Proceedings of the 2012 ACM Conference on Computer andCommunications Security (New York, NY, USA, 2012), CCS ’12,ACM, pp. 305–316.

12