OSDC 2013 1

Distributed Storage with GlusterFS

Dr. Udo SeidelLinux-Strategy @ Amadeus

OSDC 2013 2

Agenda

● Introduction● High level overview● Storage inside● Use cases● Summary

OSDC 2013 3

Introduction

LinuxCon Europe 05-07/11 2012 4

Me ;-)

● Teacher of mathematics & physics● PhD in experimental physics● Started with Linux in 1996● Linux/UNIX trainer● Solution engineer in HPC and CAx environment● Head of the Linux Strategy team @Amadeus

OSDC 2013 5

Storage: History

● Reviewing storage task responsibilities● Block allocation● Space management

● Extension of SCSI standard● Object based storage● Meta-Data handling separated from data

management

OSDC 2013 6

Object based storage

● Storage objects quite general● Partition, file, ...● Unique identifier

● OSD (Object based Storage Device)● Hardware -> original trigger● Software -> common implementation● Main component of distributed file systems

OSDC 2013 7

Distributed storage: Paradigm changes

● Block -> Object● Central -> Distributed

● Few -> Many● Big -> Small

● Server <-> Storage

OSDC 2013 8

Distributed File Systems

● 'Recent' attention on distributed storage● Cloud hype● Big Data

● See also CEPH talk the other day

OSDC 2013 9

Distributed storage – Now what?!?

● Several implementations● Different functions● Support models● Storage vendors initiatives● Relation to Linux distributions

Here and now ==> GlusterFS

OSDC 2013 10

High level overview

OSDC 2013 11

History

● Gluster founded in 2005● Gluster = GNU + cluster● Acquisition by Red Hat in 2011● Community project

● 3.2 in 2011● 3.3 in 2012

● Commercial product: RedHat Storage Server

OSDC 2013 12

The Client

● Native● 'speaks' GLUSTERFS● Not part of the Linux Kernel● FUSE-based

● NFS● Normal NFS client stack

● S3/Swift compatible● Proxy needed

OSDC 2013 13

The Server

● Data● Bricks● Translators● Volumes -> exported/served to the client

● Meta-Data● No dedicated instance● Distributed hashing approach

OSDC 2013 14

The picture

OSDC 2013 15

Storage inside

OSDC 2013 16

The Brick

● Trust each other● Interconnect via TCP/IP or RDMA/Infiniband● Dedicated file systems on GlusterFS server

● XFS recommended, EXT4 works too● Extended attributes a must

● Two main processes/daemons● glusterd ● glusterfsd

OSDC 2013 17

The Translator

● One per purpose● Replication● POSIX● Quota● I/O behaviour

● Chained -> brick graph● Technically: configuration

OSDC 2013 18

The Volume

● Service unit● Layer of configuration

● 'RAID level' -> distributed, replicated, striped, combinations

● NFS● Cache● Permissions● ....

OSDC 2013 19

Meta Data

● 2 kinds● More of local file system style● Related to distributed nature

● Some stored in backend file system ● Permissions● Time stamps● Distribution/replication

● Some calculated on the fly● Brick location

OSDC 2013 20

Elastic Hash Algorithm

● Based on file names● Name space divided● Full brick handled via relinking● Stored in extended attributes● Client needs to know topology

OSDC 2013 21

Self-Healing

● On demand vs. Scheduled● File based● Based on extended attributes● Split-brain

● Quorum function● Sometimes: manual intervention

OSDC 2013 22

Geo replication

● Asynchronous● Based on rsync/ssh● Master-Slave ● If needed: cascading ● One way street● Clocks in sync!

OSDC 2013 23

From files to objects

● Introduced with version 3.3● Hard links with some hierarchy

● Re-uses GFID (inode number)

● UFO● Unified File and Object● Combination with RESTful API● S3 and swift compatible

OSDC 2013 24

Operations:Growth, shrinkage .. failures

● A Must!● Easy● Rebalance!● Order of servers important

OSDC 2013 25

What else ...?

● Encryption :-|● Compression :-(● Snapshots :-(● Hadoop connector :-)● Locking granularity :-|● File system statistics :-)

OSDC 2013 26

Use cases

OSDC 2013 27

NAS replacement

● NFS as 1:1● Server: GlusterFS● Client: NFS

● NFS as such● Server: GlusterFS● Client: GlusterFS

OSDC 2013 28

SAN replacement

● Not quite advanced (yet)● New translator needed

● Development started● Presenting GlusterFS as block device

● Additional items needed● Locking● ...

OSDC 2013 29

Storage back-end for KVM and Co

● Stacked (indirect)● Not smart● Workable for main hypervisors

● Direct● QEMU● libvirt● oVirt/RHEV

OSDC 2013 30

Summary

OSDC 2013 31

Take aways

● Thin distributed file system layer● Modular architecture● Operationally ready● Still some surprises● Active development and community

OSDC 2013 32

References

● http://www.gluster.org● http://www.sxc.hu (pictures)

OSDC 2013 33

Thank you!

OSDC 2013 34

Distributed Storage with GlusterFS

Dr. Udo SeidelLinux-Strategy @ Amadeus