IT-SDC : Support for Distributed Computing

M. Hellmich, O. Keeble, F. Furano, A. Manzi, A. Ayllon, I. Calvet, R. Rocha, M. Regala

CHEP 2013Amsterdam

DPM - efficient storage in diverse environments

CHEP2013 2IT-SDC

Experiment with storage

Scaling the Architecture Raspberry Pi Virtual Machines Bare Metal Servers

Evaluate the outcomes Performance per Watt, Dollar

15 Oct 2013

CHEP2013 3IT-SDC

Storage

15 Oct 2013

CHEP2013 4IT-SDC

This talk is a reaction

Change in our software

Change in computing services

Change in available hardware

15 Oct 2013

CHEP2013 5IT-SDC

Our software: DPM

15 Oct 2013

MD

Disk Disk Disk Disk

DB

CHEP2013 6IT-SDC

Our software: New DPM

15 Oct 2013

MD

DiskCeph

HDFSGW GW

anyDB

S3

CHEP2013 7IT-SDC

Computing Services

15 Oct 2013

MD

Disk HDFSGW GW

anyDB

S3ceph

Metadata Reqs network CPU memory

Gateway Reqs >> network CPU memory

CHEP2013 8IT-SDC

Available Hardware: ARM

Low power consumption also when idle

15 Oct 2013

CHEP2013 9IT-SDC

So we focused on Metadata

15 Oct 2013

MD

GW

anyDB

S3ceph

MDMD

CHEP2013 10IT-SDC

The Metric

requests/sec served, pure and simple Stat requests for files Deputy for other calls

Tested over http

15 Oct 2013

CHEP2013 11IT-SDC

The Setup

15 Oct 2013

client

Disk

client

client

client

client

MD

MD

MD

DB

t

t

t

t

CHEP2013 12IT-SDC

Performance ComparisonRaspi CERN

Openstack VMEC2 medium VM

Bare metal 4 core

Requests/sec 25.8 394.18 162.27 1701.23

15 Oct 2013

3 x Raspi 3 x CERN Openstack VM

Requests/sec 71.30 1087.0

Scaling 2.76 2.75

CHEP2013 13IT-SDC

Scaling up with Memcached

*VMs are OpenStack VMs

15 Oct 2013

Raspi 3 x Raspi

Requests/secdirect

25.8 71.30

Requests/sec with Memcached

58.47 152.16

VM* 3 x VM*

Requests/secdirect

394.18 1087.0

Requests/sec with Memcached

764.02

CHEP2013 14IT-SDC

Total Cost of Ownership

TCO = Cost Machines + Cost Running Machines

Cost Running Machines = Cost Maintenance + Cost Replacement +Cost Power + Cost External Components

Cost SoftwareDev

15 Oct 2013

CHEP2013 15IT-SDC

Performance per Watt

Raspi CERN OpenStack VM

Requests/sec 25.8 394.18

Consumption [W] 3.5 30

Requests/sec/watt 7.37 13.1

15 Oct 2013

CHEP2013 16IT-SDC

Performance per Euro

15 Oct 2013

Raspi VM EC2 Medium

Requests/sec 25.8 394.18 162.27

Buying Price 25 4282 -

Power Cost over 3 years12

4.89 Euros 41.94 -

Cost over 3 years1 29.89 Euros (+100) 469.94 (+100) 1297.2

500 Request/secover 3 years

750 940 3891

1 Buying Price + Power Costs or EC2 Pricing (reserved instance)2 12VMs running on DELL PowerEdge M620 24 core 16GB RAM, price from website

Note Rasp external equipment is much more expensive!

CHEP2013 17IT-SDC

Working with a Raspberry

15 Oct 2013

Smallest possible machine

But very good for testing

Runs EPEL software*

CHEP2013 18IT-SDC

Summary

Evaluation of a DPM metadata server VMs do well Especially with memcached

Raspberries work Are even competitive

ARM may become very useful for storage

15 Oct 2013

CHEP2013 19IT-SDC

THANK YOU VERY MUCHQuestions?

15 Oct 2013