ndh2k12 cloud computing security

Hacking Cloud Computing infrastructures for fun and for profit

Matthieu BOUTHORS

Wednesday, July 11, 12

Plan

• Introduction to cloud computing

• Focus on IaaS

• Targeting data snapshots

• Targeting data dedup

2


* as a Service

3


Characteristics

• Agility / Elasticity : Everything could be scaled up/down

• APIs : Everything could be automatized for more agility

• Reliability / Low maintenance : sharing provide access to state-of-the-art architectures at low costs

• Simplicity : Everything is divided in simple pieces (needed by previous points)

4


Focus on IaaS

• Orchestration from user to behind the OS :– Networking (Firewall rules, Elastic IPs, DNS,

Load-Balancing, ... )– Storage (RAW Volumes creation/deletion,

Mounts, Snapshots, ...)– Hypervisor (VM lifecycle, sizing, RO access to

COM Port, ...)• Sound likes something new and

interesting5


IaaS stack

6


Cloud Solutions

• Custom orchestration other Xen/Qemu-KVM hypervisor : AWS/Rackspace/...

• Closed source all-in-one solutions : VMWare (vsphere + cloud director), MS (Hyper-V + System Center)

• Open source all-in-one solutions : OpenStack, Eucalyptus

7


IaaS Hackers’ objectives

• Almost unlimited network/compute/storage resources for free

• Access to IaaS customers’ sensitive piece of data

• Access to IaaS customers’ VMs

8


IaaS attack vectors

• From the Internet– easiest to perform– limited attack surface– hard to find cloud specific vectors

• Cloud APIs–Webservices available from all the Internet–Most secured part– Complete access to each IaaS functionality

9


IaaS attack vectors

• From a VM inside the cloud :– behind almost all conventional layers of

defense– simple and RAW access to underneath

resources– almost all tool / custom code could be

used on the VM– restricted to public cloud

10


“10$ Hack” principles

• Pretend to be a legitimate customer• Spend a minimum of money thanks to

elastic billing• Abuse a maximum of functionality only

accessible to customers

• Very effective against all provider with shared infrastructures

11


Get close to the target

• Thousands of physical hosts but you hope to be in the same as your target

• Host is assignation is only algorithm tuned with different goals :– limit host under-utilization– provide constant performance among all

the hosts– avoid capacity issues

12


Abuse Host assignation

• VMWare DRS :• spawn a lot of VMs and generate huge charge

inside these VMs• VMs will be migrated to hosts with low usage • Stop generating load on VMs (VMs won’t be

migrated but their hosts will be able to accept new busy VMs again)• Start DoS on targeted VMs• Targeted VMs will probably be migrated on a

host where you already have a VM

13


Attacking Storage

14


Commonly used technologies

• High performance NAS/SAN

• Custom file format for RAW disks storage :– VMDK : VMWare– QED : qemu– TODO Hyper-V

• Sometimes custom FS (VMWare VMFS)

15


Snapshots

• Users tends to create VM from only a few subset of images

• Duplicating a RAW disk image on VM creation on each VM creation takes ages

• Users need way to easily save RAW disk images

• Snapshots solve these problems

16


Snapshot principles

• “Current” disk is marked as read only• New disk is created with the “current”

disk as snapshot ancestor• New writes are performed on new disk• Reads are performed on new disk only

if block has been re-writed since snapshot

17


Exploiting snapshots

• Control the value of the ancestor

• Restore the snapshot

• Read the content of the chosen ancestor

18


qemu-kvm

• anyfile can be used as an ancestor with standard qemu-img binary :

•

19

$ qemu-img create -f qcow2 -b /etc/passwd nasty.qed

$ strings nasty.qed

/etc/passwd

$ qemu-img convert -O raw nasty.qed nasty.raw && head -n 3 nasty.raw

# $FreeBSD: release/9.0.0/etc/master.passwd 218047 2011-01-28 22:29:38Z pjd $

#

root:*:0:0:Charlie &:/root:/usr/local/bin/bash


VMWare VMDK

• No sexy tool like qemu-img• Virtual disk is usually separated into

several .vmdk files• *-flat.vmdk files containing the raw

data• .vmdk containing useful informations

including the references to *-flat.vmdk files

20


Normal .vmdk

21

# Disk DescriptorFileversion=1encoding="UTF-8"CID=b5892a19parentCID=ffffffffisNativeSnapshot="no"createType="vmfs"

# Extent descriptionRW 33554432 VMFS "debian-test-flat.vmdk"

# The Disk Data Base#DDBddb.virtualHWVersion = "8"ddb.longContentID = "312b4792544af9a8c959d20eb5892a19"ddb.uuid = "60 00 C2 9b 32 5e f1 d0-54 b3 29 04 95 51 99 3c"ddb.geometry.cylinders = "2088"ddb.geometry.heads = "255"ddb.geometry.sectors = "63"ddb.adapterType = "lsilogic"


Modifying the extends

• Works with -flat.vmdk files from other disk images

• Does not seems to work on local text files

• Seems to work on local gz files

22


.vmdk modified #1

23


# Extent descriptionRW 33554432 VMFS "debian-test-flat.vmdk"RW 0 VMFS "debian-test_1-flat.vmdk"



.vmdk modified #2

24


# Extent descriptionRW 33554432 VMFS "debian-test-flat.vmdk"RW 0 VMFS "/bootbank/state.tgz"



Retrieving data

• Extend is accessible at the end of the disk device

• We only have to skip the begining of the disk to retrieve the needed piece of data

• In our case :

25

dd if=/dev/sda of=leaked_img skip=33554432 bs=51225+0 records in25+0 records out12800 bytes transferred in 0.020000 secs (640001 bytes/sec)


Caveats

• Hard to guess other VMs .vmdk filenames and paths

• .gz are truncated to the closest multiple of 512 bytes

26


Real-World attack possibilities

• Unlikely to be able to forge these “snapshots” from API

• Should be really useful when user is able to upload his own images

27


Deduplication

• Snapshot is not enough for optimizing storage usage

• Filer (or file-systems like ZFS) provides deduplication of redundant blocks

• Each time two redundant blocks are found, a block is replaced by a link

• If a block is rewrited, the entire block need to be written

28


Identify deduplication

• If only one byte is written, all the deduplicated block needs to be written

• Sounds like a write penalty

• We will conduct timed attacks in order to identify this penalty

29


Caveat #1

• Deduplication may be asynchronous

• Attack needs to wait for the deduplication process to occurs

• Since deduplication is only on the Filer, VM does not need to run waiting for deduplication, only volume need to remains during the wait

30


Caveat #2

• Deduplication is filer-centric

• Some filers work on NFS export at file level => no attack possible

• Blocks may be misaligned between the VM Filesystem and the actual Filer => unlikely to be seen since this means a severe performance issue, the cloud provider will work for us and avoid this case

31


Caveat #3

• OS / Filesytems Cache

• Empty Linux cache before doing anything else

• Use direct IO for writing

32


What exploitation looks like

• Create files that may be deduplicated and random files on your own VM

• Wait (maybe 24 hours if dedup is a daily cron job)

• Flush Linux file caches• Write one byte per block on each bloc

(usually 4KB)• Compare timing on random files vs

deduplicated files

33


Flush Linux file cache

• Cache behaviour of Linux unpredictable

• we need a tool to show and flush Linux file cache

• VMTouch is this tools (thx to hoytech)

• https://github.com/hoytech/vmtouch34


https://github.com/hoytech/vmtouch

https://github.com/hoytech/vmtouch

Writes and time measurement

• Avoiding python/ruby/java/...• Needs to use direct IOs• Needs to be far more precise than

UNIX time command (ms precision)

• Solution : simple C program

35


.c POC

36

#include <sys/types.h>#include <sys/stat.h>#include <fcntl.h>#include <time.h>#include <stdio.h>

int64_t timespecDiff(struct timespec *timeA_p, struct timespec *timeB_p){ return ((timeA_p->tv_sec * 1000000000) + timeA_p->tv_nsec) - ((timeB_p->tv_sec * 1000000000) + timeB_p->tv_nsec);}

int main(int argc, char **argv) { int f; int t = "A"; f = open(argv[1], O_WRONLY | O_DIRECT); struct timespec start, end; clock_gettime(CLOCK_MONOTONIC, &start); write(f,&t,1); clock_gettime(CLOCK_MONOTONIC, &end); printf("%d\n",timespecDiff(&end, &start)); close(f); return 0;}


Attacking ZFS dedup

37


Limitations of this POC

• Only focus on first block of a file

• Lack of automation

• Lack of block size guessing (needed in order to correct first limitation)

38


Impacts / Counter-measure

• Could be use to find if your cloud neighborhood have :

• unpatched binaries on their VMs• files that belongs to you (dataleak detection)

• Not limited to VMs, Provider may also have hypervisor’s data on the filer

• Counter-measure : encrypt all your stuff in order to have different RAW blocks for same piece of data

39


Corrupting dedup

• Work In Progress :– generating a collision on dedup table– when user try to write legitimate file, they are deduped to your

backdoored version

• Difficulties :– Each Filer / Filesystem have is own checksum algorithm : you

need to gather information about the storage infrastructure– Filer / Filesystems using weak checksum usually perform bit per

bit comparaison before dedup (for instance NetApp)– Finding usable collision on strong checksum algorithm is really

hard (for instance ZFS uses AES-256)

40


Denial of Service via Dedup

• Not tested yet

• Only usable with solutions with weak checksum + bit-per-bit comparison

• Attack scenario :– generate a maximum of collisions thanks

to the weak algorithm

41


Solutions : Provider side

• Innovative solutions need to be security assessed more than usual

• Internal security actions need to be publicly known and mesurable

• Solutions :– internal skilled security teams – certifications and external audit policies

42


Solutions : Customer/User side

• Customer could not blindly trust any cloud provider

• Counter-measures are available on customer side :– Avoid storing sensitive data unencrypted– Add firewall / IDS / IPS directly on VMs– Ensure the quality and reliability of the

selected provider

43


Conclusions

• New technologies bring new threats• Other attacks available :

• VM memory overlapping Host Memory (VMSA-2012-0009.2)• Xen SYSRET instruction (CVE-2012-0217)• ...

• New threats does not always mean new way to fight these threats

44


Questions ?

• Thanks : @hackerzvoice & @outscale

• Contacts : @majin_boo

45

CrashFR will never die in our living memories


ndh2k12 cloud computing security

Technology