gigaom structure sf - june 2012

Inktank Delivering the Future of Storage

ME ME ME ME ME ME.

I made a slide today. It’s all about me.

ME ME ME ME ME ME.

I made a slide today. It’s all about me.

Ross Turk VP Community, Inktank ross@inktank.com @rossturk inktank.com | ceph.com

RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes

LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP

RBD A reliable and fully-distributed block device, with a Linux kernel client and a QEMU/KVM driver

CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE

RADOSGW A bucket-based REST gateway, compatible with S3 and Swift

APP APP HOST/VM CLIENT

Let’s Start With a Good, Old-Fashioned Origin Story JD Hancock, Flickr / CC BY 2.0 6

The Evolution of Storage A brief history of information storage technology

Cave Paintings: The Earliest Form (maybe) of Information Storage Chico.Ferreira, Flickr / CC BY 2.0 8

Technology Review: Cave Painting The good •  Low cost per smudge • Multitouch

The bad •  Limited storage capacity

•  10 caveman ideas per wall

• No support for CIFS

= + HUMAN WRITING

Technology Review: Books and Libraries The good •  Cost per scroll is high

•  Can be eased w/slave labor

The bad • No automatic replication

•  Must complete backups before Caesar’s invasion of Egypt!

Books (Strahov, Prague Library) Moyan_Brenn, Flickr / CC BY-ND 2.0 12

Printing Press FateDenied, Flickr / CC BY 2.0 13

+ magnet = tape magnetic tape

IBM System 360 Tape Drives Erik Pitti, Wikipedia / CC BY-ND 2.0

HUMAN COMPUTER TAPE

HUMAN ROCK

11101011 10110110 10110101 10101001 00100100 01001001 10100100 10100101 01011010 01101010 10101010 10101010 01010110 01010011

Tape Is Stupid Mrs. Gemstone, Flickr / CC BY-SA 2.0 18

Computers Need Programmers (and Operators) USDAgov, Flickr / CC BY 2.0 19

HUMAN COMPUTER TAPE

Throughput Becomes Important rfduck, Flickr / CC BY-ND 2.0 21

Hard Drive Jeff Kubina, Flickr / CC-BY-SA 2.0 22

Hard Drives Are Totally Better

amazing spinny hard drives sucky stupid tape

111010

010 111

da 000

110 001

010 011 db

owner: rturk created: aug12

last viewed: aug17 size: 42025 perms: 644 11101011 10110110 10110101

10101001 00100100 01001001 10100100 10100101 01011010 01101010 10101010 10101010

111010

010 111

da 000

110 001

010 db 01 10

Humanity Outgrows the Hard Drive Mr. T in DC, Flickr / CC BY 2.0 27

HUMAN COMPUTER

What Happens When Two HUMANs Need Access to the Same Resource? wFourier, Flickr / CC BY 2.0 29

HUMAN COMPUTER

(COMPUTER)

HUMAN HUMAN

HUMAN (actually more like this…)

DISK COMPUTER

texture: crunch flavor: smoke, salt

nutrition: none color: bacon

11101011 10110110 10110101 10101001 00100100 01001001 10100100 10100101 01011010 01101010 10101010 10101010

object

111010

010 111

da 000

110 001

010 011 db X

DISK COMPUTER

COMPUTER

DISK COMPUTER

COMPUTER

DISK COMPUTER

COMPUTER

The Current State of Storage How people store information today, and why it’s still not perfect yet

How Much Store Things All Human History!! Time-scale: Roughly logarithmic. Content: Whatever the opposite of “scientific” is. 39

Writing

Computers

Shared storage

Distributed storage

Cloud computing

Painting

DISK COMPUTER

Storage Hardware Michael Moll, Wikipedia / CC BY-SA 2.0 44

6.4 Million Square Feet of Expensive Factory Buildings Dude94111, Flickr / CC BY 2.0 45

Storage Hardware Vendors Have Bills to Pay CarbonNYC, Flickr / CC BY 2.0 46

…Which Means That Customers Do Too 401K 2012, Flickr / CC BY-SA 2.0 47

Technology Is Becoming a Commodity RaeAllen, Flickr / CC-BY 2.0 48

Commodity Prices Fluctuate

May-07 May-08 May-09 May-10 May-11 May-12

Growing With Hardware Appliances First PB •  Proprietary

storage hardware • Well-known

storage vendor $14 b’zillion

Second PB •  Proprietary storage

hardware •  Same storage

vendor Another $14 b’zillion

Dedicated Hardware Appliances Are OLD TECHNOLOGY Paul Keller, Flickr / CC BY 2.0 51

52 Source: http://www.cpubenchmark.net/high_end_cpus.html

FLAGSHIP PRODUCT

“I'm sick of paying for hardware with a three-year-old proc in it!” Mel B., Flickr / CC BY 2.0 54

Hardware Appliances are Mysterious Black Boxes Abode of Chaos, Flickr / CC BY 2.0 55

HUMAN [DEVELOPER]

Give More Money To The Big Proprietary Vendors It will make them very, very happy. 59

Storage Should Be Better

People need storage solutions that… • …are open • …are easy to manage • …satisfy their requirements

•  performance •  functional •  financial

The Birth of a New Storage Solution We think our roots are showing

DreamHost 62

Sage Weil Co-founder of DreamHost Inventor of Ceph CEO of Inktank

DreamHost DreamHost is staffed by extraordinarily hip people 64

New Monthly Code Commits

2004-06 2005-07 2006-07 2007-07 2008-07 2009-07 2010-07 2011-07

Ceph Starts Popping Up

OPEN SOURCE

philosophy design

Open Source is the Best Way to Spread Ideas orchidgalore, Flickr / CC BY 2.0 69

OPEN SOURCE

COMMUNITY-FOCUSED

philosophy design

All of Us Are Smarter Than Some of Us rturk, Linkedin Inmap 71

OPEN SOURCE

COMMUNITY-FOCUSED

SCALABLE

philosophy design

Ceph is Built to Scale Time-scale: Roughly logarithmic. Content: Whatever the opposite of “scientific” is. 73

Too much for a book

Too much for a drive

Too much for a computer

Too much for a room

Too much for a cave

OPEN SOURCE

COMMUNITY-FOCUSED

SCALABLE

NO SINGLE POINT OF FAILURE

philosophy design

Ariolimax Californicus aroid, Flickr / CC BY 2.0 75

The Octopus (A Metaphor) I love speaking in metaphors. 76

single point of failure

replicated replicated

The Beehive (A Better Metaphor) blumenbiene, Flickr / CC BY 2.0 77

OPEN SOURCE

COMMUNITY-FOCUSED

SCALABLE

SOFTWARE BASED

philosophy design

C++ ✔

OPEN SOURCE

COMMUNITY-FOCUSED

SCALABLE

SOFTWARE BASED SELF-

MANAGING

philosophy design

Hard Drives Are Tiny Record Players and They Fail Often jon_a_ross, Flickr / CC BY 2.0 82

55 times / day

x 1 MILLION

Enter: Ceph An architectural and functional overview of the Ceph system

DISK DISK

OSD OSD OSD OSD

FS FS FS FS btrfs xfs ext4

Monitors: • Maintain cluster map •  Provide consensus for

distributed decision-making • Must have an odd number •  These do not serve stored

objects to clients

OSDs: • One per disk (recommended) •  At least three in a cluster •  Serve stored objects to

clients •  Intelligently peer to perform

replication tasks •  Supports object classes

LIBRADOS

socket

LIBRADOS •  Provides direct access to

RADOS for applications •  C, C++, Python, PHP, Java • No HTTP overhead

LIBRADOS RADOSGW

socket

RADOS Gateway: •  REST-based interface to

RADOS •  Supports buckets,

accounting •  Compatible with S3 and

Swift applications

LIBRADOS LIBRBD

VIRTUALIZATION CONTAINER

LIBRADOS

LIBRBD CONTAINER

LIBRADOS LIBRBD

CONTAINER VM

LIBRADOS

KRBD (KERNEL MODULE) HOST

RADOS Block Device: •  Storage of virtual disks in

RADOS •  Allows decoupling of VMs and

containers •  Live migration!

•  Images are striped across the cluster

•  Boot support in QEMU, KVM, and OpenStack Nova

•  Mount support in the Linux kernel

CLIENT

data metadata

Metadata Server • Manages metadata for a

POSIX-compliant shared filesystem •  Directory hierarchy •  File metadata (owner,

timestamps, mode, etc.) •  Stores metadata in RADOS •  Does not serve file data to

clients • Only required for shared

filesystem

What Makes Ceph Unique? Part one: CRUSH

APP ??

How Long Did It Take You To Find Your Keys This Morning? azmeen, Flickr / CC BY 2.0 107

Dear Diary: Today I Put My Keys on the Kitchen Counter Barnaby, Flickr / CC BY 2.0 108

I Always Put My Keys on the Hook By the Door vitamindave, Flickr / CC BY 2.0 110

HOW DO YOU FIND YOUR KEYS

WHEN YOUR HOUSE IS

INFINITELY BIG AND

ALWAYS CHANGING?

The Answer: CRUSH!!!!! pasukaru76, Flickr / CC SA 2.0 112

10 10 01 01 10 10 01 11 01 10

hash(object name) % num pg

CRUSH(pg, cluster state, rule set)

10 10 01 01 10 10 01 11 01 10

CRUSH •  Pseudo-random placement

algorithm •  Ensures even distribution •  Repeatable, deterministic •  Rule-based configuration

•  Replica count •  Infrastructure topology •  Weighting

CLIENT

What Makes Ceph Unique Part two: thin provisioning

LIBRADOS

LIBRBD VIRTUALIZATION CONTAINER

HOW DO YOU SPIN UP

THOUSANDS OF VMs INSTANTLY

AND EFFICIENTLY?

0 0 0 0

instant copy

CLIENT

CLIENT read

What Makes Ceph Unique? Part three: clustered metadata

Metadata for a POSIX-Compliant Filesystem Barnaby, Flickr / CC BY 2.0 127

CLIENT

one tree

three metadata servers

DYNAMIC SUBTREE PARTITIONING

And Now: Backpedaling

ALMOST EVERYTHING

NEARLY AWESOME

AWESOME AWESOME

AWESOME

LAN SCALE!! *

* OR REALLY REALLY SCARY FAST WAN

What is Inktank? I really like your polo shirt, please tell me what it means!

Who? •  Ceph’s inventor and (most) developers

Why? •  To ensure the long-term success of Ceph

•  To help companies adopt Ceph through services, support, training, and consulting

When? •  Founded: December 28, 2011

•  Brand Launched: April 2012

What do we want from you?? •  Try Ceph! Tell us what you think. Ask if

you need help. Help others if you can!

•  Are you a company? Consider dedicating dev resources to the project.

Questions?

Ross Turk VP Community, Inktank ross@inktank.com @rossturk inktank.com | ceph.com

gigaom structure sf - june 2012

flickr cc

wikipedia cc

human history

future of storage

intelligent storage

storage hardware vendors

storage hardwaremichael

human computer tape20

Technology

gigaom research - work horses and dark horses

summary of gigaom research sector roadmap report on sdn

"preliminary findings from the ict developers survey" by...

data science day new york: gigaom big data market overview

gigaom structure 2013 to look into the future (slides)

gigaom radar for enterprise object storage

phil hendrix location epicenter of mobile innovation 2010...

spice works vmware cloud strategy gigaom report

gigaom radar for cloud observability

"sizing the eu app economy" by david card and mark mulligan,...

cambriadevelopment...elevator equipment room 44 sf total...

gigaom structure 2014 launchpad featuring nimboxx

google's mobile strategy, gigaom research roundtable

gigaom -...

sketchnotes-sf meetup :: round 22 :: structure-a-go-go [wed...

saltstack - gigaom structure launchpad 2013

gigaom radar for network operating systems

the future of mobile a segment analysis by gigaom pro

riverbed deck - gigaom 2013 events

for lease - loopnet › d2 ›...