what is a ceph (and why do i care). openstack storage - colorado openstack meetup october 14 2014

14 OCT 2014Colorado OpenStack Meetup

HISTORICAL TIMELINE

RHEL-OSP Certification FEB 2014

MAY 2012Launch of Inktank

OpenStack Integration 2011

2010Mainline Linux Kernel

Open Source 2006

2004 Project Starts at UCSC

Production Ready Ceph SEPT 2012

2012CloudStack Integration

OCT 2013Inktank Ceph Enterprise Launch

Xen Integration 2013

APR 2014Inktank Acquired by Red Hat

10 years in the making

OPENSTACK USER SURVEY, 05/2014

DEV / QA PROOF OF CONCEPT PRODUCTION

A STORAGE REVOLUTION

PROPRIETARY HARDWARE

PROPRIETARY SOFTWARE

SUPPORT & MAINTENANCE

COMPUTER

DISKCOMPUTE

COMPUTER

STANDARDHARDWARE

OPEN SOURCE SOFTWARE

ENTERPRISEPRODUCTS &

SERVICES

COMPUTER

DISKCOMPUTE

COMPUTER

ARCHITECTURE

ARCHITECTURAL COMPONENTS

RGWA web services

gateway for object storage, compatible

with S3 and Swift

LIBRADOSA library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby,

RADOSA software-based, reliable, autonomous, distributed object store comprised ofself-healing, self-managing, intelligent storage nodes and lightweight monitors

RBDA reliable, fully-distributed block device with cloud

platform integration

CEPHFSA distributed file

system with POSIX semantics and

scale-out metadata management

APP HOST/VM CLIENT

RGWA web services

with S3 and Swift

APP HOST/VM CLIENT

OBJECT STORAGE DAEMONS

btrfsxfsext4zfs?

RADOS CLUSTER

APPLICATION

RADOS CLUSTER

RADOS COMPONENTS

OSDs: 10s to 10000s in a cluster One per disk (or one per SSD, RAID

group…) Serve stored objects to clients Intelligently peer for replication & recovery

Monitors: Maintain cluster membership and state Provide consensus for distributed decision-

making Small, odd number These do not serve stored objects to

clients

WHERE DO OBJECTS LIVE?

??APPLICATION

OBJECT

A METADATA SERVER?

APPLICATION

CALCULATED PLACEMENT

FAPPLICATION

EVEN BETTER: CRUSH!

CLUSTER

OBJECTS

0110 10 01

PLACEMENT GROUPS(PGs)

CRUSH IS A QUICK CALCULATION

RADOS CLUSTER

OBJECT

0110 10 01

CRUSH: DYNAMIC DATA PLACEMENT

CRUSH: Pseudo-random placement algorithm

Fast calculation, no lookup Repeatable, deterministic

Statistically uniform distribution Stable mapping

Limited data migration on change Rule-based configuration

Infrastructure topology aware Adjustable replication Weighting

OBJECT

10 10 01 01 10 10 01 11 01 10

hash(object name) % num pg

CRUSH(pg, cluster state, rule set)

OBJECT

10 10 01 01 10 10 01 11 01 10

CLIENT

RGWA web services

with S3 and Swift

APP HOST/VM CLIENT

ACCESSING A RADOS CLUSTER

APPLICATION

RADOS CLUSTER

LIBRADOS

OBJECT

socket

LIBRADOS: RADOS ACCESS FOR APPS

LIBRADOS: Direct access to RADOS for applications C, C++, Python, PHP, Java, Erlang Direct access to storage nodes No HTTP overhead

RGWA web services

with S3 and Swift

APP HOST/VM CLIENT

THE RADOS GATEWAY

RADOS CLUSTER

RADOSGWLIBRADOS

socket

RADOSGWLIBRADOS

APPLICATION APPLICATION

RADOSGW MAKES RADOS WEBBY

RADOSGW: REST-based object storage proxy Uses RADOS to store objects API supports buckets, accounts Usage accounting for billing Compatible with S3 and Swift applications

RGWA web services

with S3 and Swift

APP HOST/VM CLIENT

STORING VIRTUAL DISKS

RADOS CLUSTER

HYPERVISORLIBRBD

SEPARATE COMPUTE FROM STORAGE

RADOS CLUSTER

HYPERVISORLIBRB

VM HYPERVISORLIBRB

KERNEL MODULE FOR MAX FLEXIBLE!

RADOS CLUSTER

LINUX HOSTKRBD

RBD STORES VIRTUAL DISKS

RADOS BLOCK DEVICE: Storage of disk images in RADOS Decouples VMs from host Images are striped across the cluster

(pool) Snapshots Copy-on-write clones Support in:

Mainline Linux Kernel (2.6.39+) Qemu/KVM, native Xen coming soon OpenStack, CloudStack, Nebula,

Proxmox

Export snapshots to geographically dispersed data centers▪Institute disaster recovery

Export incremental snapshots▪Minimize network bandwidth by only sending

changes

RBD SNAPSHOTS

RGWA web services

with S3 and Swift

APP HOST/VM CLIENT

SEPARATE METADATA SERVER

LINUX HOST

RADOS CLUSTER

KERNEL MODULE

datametadata 0110

SCALABLE METADATA SERVERS

METADATA SERVER Manages metadata for a POSIX-compliant

shared filesystem Directory hierarchy File metadata (owner, timestamps,

mode, etc.) Stores metadata in RADOS Does not serve file data to clients Only required for shared filesystem

CALAMARI

CALAMARI ARCHITECTURE

CEPH STORAGE CLUSTER

MASTER

CALAMARI

ADMIN NODE

MINION

USE CASES

WEB APPLICATION STORAGE

WEB APPLICATION

APP SERVER

CEPH STORAGE CLUSTER(RADOS)

CEPH OBJECT GATEWAY

CEPH OBJECT GATEWAY(RGW)

APP SERVER

S3/Swift S3/Swift S3/Swift S3/Swift

MULTI-SITE OBJECT STORAGE

WEB APPLICATIONAPP

SERVER

CEPH OBJECT GATEWAY

(US-EAST)

WEB APPLICATIONAPP

SERVER

CEPH OBJECT GATEWAY

(EU-WEST)

ARCHIVE / COLD STORAGE

APPLICATION

CACHE POOL (REPLICATED)

BACKING POOL (ERASURE CODED)

ERASURE CODING

OBJECT

REPLICATED POOL

ERASURE CODED POOL

COPY COPY

OBJECT

31 2 X Y

Full copies of stored objects Very high durability Quicker recovery

One copy plus parity Cost-effective durability Expensive recovery

ERASURE CODING: HOW DOES IT WORK?

OBJECT

ERASURE CODED POOL

CACHE TIERING

CEPH CLIENT

CACHE: WRITEBACK MODE

BACKING POOL (REPLICATED)

Read/Write Read/Write

CACHE TIERING

CEPH CLIENT

CACHE: READ ONLY MODE

BACKING POOL (REPLICATED)

Write Write Read Read

WEBSCALE APPLICATIONS

WEB APPLICATION

APP SERVER

NativeProtocol

ARCHIVE / COLD STORAGE

APPLICATION

CACHE POOL (REPLICATED)

BACKING POOL (ERASURE CODED)

Site A Site B

CEPH STORAGE CLUSTER CEPH STORAGE CLUSTER

CEPH BLOCK DEVICE (RBD)

DATABASES

MYSQL / MARIADB

LINUX KERNEL

NativeProtocol

WHAT ABOUT CEPH AND OPENSTACK?

CEPH AND OPENSTACK

RADOSGWLIBRADOS

RADOS CLUSTER

OPENSTACK

KEYSTONE CINDER GLANCE

NOVASWIFTLIBRB

DLIBRB

HYPER-

VISORLIBRBD

JUNO Enable Cloning for rbd-backed ephemeral disks

KILO Volume Migration from One Backend to Another Implement proper snapshotting for Ceph-based

ephemeral disks Improve Backup in Cinder

OPENSTACK ADDITIONS

Future Ceph Roadmap

CEPH ROADMAP

Giant Hammer I-Release

RBD df

Object Versioning

Performance Improvements

RBD Mirroring

Calamari

Alternative Web Server for RGW

Object Expiration

NEXT STEPS

NEXT STEPSWHAT NOW?

• Read about the latest version of Ceph: http://ceph.com/docs

• Deploy a test cluster using ceph-deploy: http://ceph.com/qsg

Getting Started with Ceph

Most discussion happens on the mailing lists ceph-devel and ceph-users. Join or view archives at http://ceph.com/list

IRC is a great place to get help (or help others!) #ceph and #ceph-devel. Details and logs at http://ceph.com/irc

Getting Involved with Ceph

• Deploy a test cluster on the AWS free-tier using Juju: http://ceph.com/juju

• Ansible playbooks for Ceph: https://www.github.com/alfredodeza/ceph-ansible Download the code: http:

//www.github.com/ceph The tracker manages bugs and

feature requests. Register and start looking around at http://tracker.ceph.com

Doc updates and suggestions are always welcome. Learn how to contribute docs at http://ceph.com/docwriting

THANK YOU!

Ian ColleGlobal Director of Software Engineering

icolle@redhat.com303.601.7713

@ircolle

what is a ceph (and why do i care). openstack storage - colorado openstack meetup october 14 2014

storage nodes

storage revolution

storage of disk images

rados components10osds

rados access forapps28librados

cluster state

cluster membership

cluster pool snapshots

Software

introduction into ceph storage for openstack

ceph bluestore and openstack

july triangle openstack meetup

openstack and ceph: the winning pair

20130325 openstack-meetup

ceph storage in openstack

openstack with ceph

ceph and mirantis openstack

ceph day santa clara: deploying ceph and openstack with juju

ceph and openstack - sébastien han _ ceph - liberty.pdf ·...

ceph performance on openstack - barcelona summit

[viet openstack] cloud computing - openstack meetup v2

seattle openstack meetup

suse openstack and ceph · openstack cinder and ceph •...

triangle openstack meetup

déployer facilement openstack et ceph avec bright openstack

rally - benchmarking_as_a_service - openstack meetup

private cloud mit ceph und openstack

openstack storage buddy ceph

ceph and openstack