(stg403) amazon ebs: designing for performance

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Marc Olson, Senior Software Engineer

October 2015

Amazon Elastic Block Store

Designing for Performance

STG403

Amazon EBS overview

What is EBS?

• Network block storage as a service

• EBS volumes attach to any Amazon EC2 instance in the

same Availability Zone

• Designed for five nines of availability

• 2 million volumes created every day

EBS volume types

Magnetic General purpose (SSD) Provisioned IOPS (SSD)

EBS volume types

IOPS: Typically 100, best effort

Throughput: 40-90 MB/s

Latency: Read 10-40ms, Write 2-10ms

Best for infrequently accessed data

Magnetic

EBS volume types

IOPS Baseline: 100-10,000 (3 / GiB)

IOPS Burst: 30 minutes @ 3,000

Throughput: Up to 160 MB/s

Latency: Single-digit ms

Performance consistency: 99%

Most workloadsGeneral purpose (SSD)

EBS volume types

IOPS: 100-20,000 (customer provisioned)

Throughput: Up to 320 MB/s

Latency: Single-digit ms

Performance consistency: 99.9%

Mission Critical workloadsProvisioned IOPS (SSD)

Performance

Queuing theory

Little’s law is the foundation for performance tuning theory

• Mathematically proven by John Little in 1961

𝑾 =𝑳

𝑨W = Wait time = average wait time per request

L = Queue length = average number of requests waiting

A = Arrival rate = the rate of requests arriving

EBS performance is related to this law

Performance optimization is measured by:

IOPS: Read/write I/O rate (IOPS)

Latency: Time between I/O submission

and completion (ms)

Throughput: Read/write transfer rate

(MB/s); throughput = IOPS X I/O size

Key components to performance

EC2 instance

I/O

EBS

Network link

A day in the life of an I/O

A day in the life: I/O

All I/O must pass through I/O domain

Requires “grant mapping” prior to 3.8.0

Grant mappings are expensive operations due to TLB flushes

EBS

Grant

mappingread(fd, buffer,

BLOCK_SIZE)

I/O domainInstance

A day in the life: I/O (continued)

Responses

Requests

Instance I/O domain

READ

8KB @ 1234

Request queue is a single memory page

Each I/O request has 11 grant references (4KiB/reference)

Maximum data in queue = 1408 KiB

3.8.0+ Kernels – Persistent grants

Grant mappings are setup in a pool once

Data is copied in and out of the grant pool

Copying is significantly faster than remapping

EBS

Grant

poolread(fd, buffer,

BLOCK_SIZE)

I/O domainInstance

3.8.0+ Kernels – Indirect grants

Responses

Requests

Instance

READ

8KB @ 1234

I/O domain

Each I/O request has grant references that contain grant references

Maximum data in queue = 4096 KiB (default)

Instance I/O: Before 3.8.0

0 1 2 3 4 5 30 31

128KiB

44KiB 44KiB 40KiB

Instance I/O: Linux 3.8.0+

0 1 2 3 4 5 30 31

128KiB

Tip: Use 3.8+ kernel

Amazon Linux 2013.09 or later

Ubuntu 14.04 or later

RHEL7 or later

Etc.

Queue depth

An I/O operation

EBS

After it’s gone, it’s gone

EC2

Queue depth is the pending I/O for a volume

Workload/

software

Typical block

size

Random/

Seq?

Max EBS @ 500

MB/s instances

Max EBS @

1 GB/s instances

Max EBS @ 10 GB/s

instances

Oracle DB Configurable:2 KB –

16 KB

Default 8 KB

random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS

Microsoft SQL

Server

8 KB w/ 64 KB

extents

random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS

MySQL 16 KB random ~4,000 IOPS ~7,800 IOPS ~48,000 IOPS

PostgreSQL 8 KB random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS

MongoDB 4 KB sequential ~15,600 IOPS ~31,000 IOPS ~48,000 IOPS

Apache

Cassandra

4 KB random ~15,600 IOPS ~31,000 IOPS ~48,000 IOPS

GlusterFS 128 KB sequential ~500 IOPS ~1,000 IOPS ~6,000 IOPS

Cheat sheet: Sample storage workloads on AWS

Example workload

Transaction (OLTP)

Examples: eCommerce website, metadata storage

Benchmark: MySQL + sysbench

Tip: Workload

Where possible, use real

production workloads for

performance testing

Baseline configuration

Availability Zone: US West (Oregon)

Instance type: m2.4xlarge

vCPU: 8

Memory: 68.4GiB

EBS-optimized

Data volume: 500GiB EBS magnetic

OS: Amazon Linux 2015.03.1

Optimization: Increase parallelism

MySQL threads

Tra

nsa

ctions (

n)

Baseline

2 n

Tip: Parallelism

Increase parallelism of your system


EC2 instance

I/O

EBS

Network link

m2.4xlarge

CPU: Intel Xeon

vCPU: 8

Memory: 68.4 GiB

Price: $0.98/hour*

Instance selection

r3.2xlarge

CPU: Intel Xeon E5-2670 v2

vCPU: 8

Memory: 61 GiB

Enhanced networking

Price: $0.70/hour*

* All pricing from US West (Oregon)

EBS optimized instances

• Most instance families support the EBS-optimized flag

• EBS-optimized instances now support up to 4 Gb/s

• Drive 32,000 16K IOPS or 500 MB/s

• Available by default on newer instance types

• EC2 *.8xlarge instances support 10 Gb/s network

• Max IOPS per node supported is ~48,000 IOPS @ 16K I/O

Tip: Use EBS-optimized instances

Use EBS-optimized instances

for consistent EBS performance

Updated configuration: Instance type


Instance type: r3.2xlarge

vCPU: 8

Memory: 61 GiB

EBS-optimized

EBS volume: 500GiB magnetic


25%

Optimization: Current generation instances

MySQL threads

Tra

nsactions (

n)

Baseline

r3.2xlarge

2 n

Tip: Instance selection

Use the right instance family for

your workload

Use current generation instances


EC2 instance

I/O

EBS

Network link

Volume selection

EBS magnetic

Latency:

Read: 10-40ms

Write: 2-10ms

SSD backed

Latency:

Read/Write: Single-digit ms

File systems

Use a modern, journaled filesystem

ext4, xfs, etc.

Ensure partitions are aligned on 4KiB boundaries

Pre-warming

Volume initialization

Newly created volumes

• Just attach, mount, and go!

• Pre-warming is no longer recommended

Volumes restored from snapshots

• You can use your volume right away

• Accelerate data loading by reading

Updated configuration: EBS volumes


Instance type: r3.2xlarge

vCPU: 8

Memory: 61 GiB

EBS-optimized

Boot volume: 8 GiB – EBS general purpose

Data volume: 500 GiB – EBS general purpose


Optimization: Volume selectionT

ransactions (

n)

19% 50%

MySQL threads

Baseline

r3.2xlarge

r3.2xlarge gp2

2 n

Tip: Volume selection

Use SSD backed volumes when

performance matters

EBS IOPS vs. Throughput

20,000 IOPS

PIOPS volume

20,000 IOPS

320 MB/s

throughput

You can achieve 20,000 IOPS when

driving smaller I/O operations

You can achieve up to 320 MB/s

when driving larger I/O operations

EBS IOPS vs. Throughput

8,000 IOPS

PIOPS volume

8,000 IOPS

320 MB/s

throughput

8,000 x 8 KB = 64 MB/s

8,000 x 16 KB = 128 MB/s

8,000 x 32 KB = 256 MB/s

16,000 x 8 KB = 128 MB/s

8,000 x 64 KB=512 MB/s

5,000 x 64 KB = 320 MB/s

Striping

Increases performance, or capacity, or both

Don’t mix volume types

Typically RAID 0 or LVM stripe

Avoid RAID for redundancyEBS

EC2

Striping: Snapshots

Quiesce I/O

1. Database: FLUSH and LOCK tables

2. Filesystem: sync and fsfreeze

3. EBS: snapshot all volumes

When snapshot API returns,

it is safe to resume

EBS-optimized instance

Four key components: Balanced

EC2

A “boatload” of I/O

Right-sized EBS

Monitoring

Amazon CloudWatch

Important Amazon CloudWatch metrics:

• IOPS and bandwidth

• Latency

• Queue depth

All EBS metrics are prefixed with “Volume”

CloudWatch: Instance bandwidth

m4.2xlarge

Instance: 128MB/s

m4.4xlarge

Instance: 256MB/s

m4.10xlarge

Volume: 320MB/s

Iostat

Don’t trust svctm or %util on parallel storage devices and systems!

Perfmon

Summary

Monitor

performance

Increase I/O

parallelism

Select the right

instance for your

workload

Select the right

volume for your

workload

Related Sessions

BDT323 - Amazon EBS and Cassandra:

1 Million Writes Per Second on 60 Nodes

Thank you!

(stg403) amazon ebs: designing for performance

Technology