gluster.community.day.2013

Bricks and Translators: GlusterFS as SWIFT replacement in OpenStack

Dr. Udo SeidelLinux-Strategy @ Amadeus

GlusterFS Day London 2013

Agenda

● Introduction● High level overview● Storage inside● Team up with OpenStack● Summary


Introduction


Me ;-)

● Teacher of mathematics & physics● PhD in experimental physics● Started with Linux in 1996● Linux/UNIX trainer● Solution engineer in HPC and CAx environment● Head of the Linux Strategy team @Amadeus


Distributed Storage Systems

● 'Recent' attention on distributed storage● Cloud hype● Big Data

● See also ● CEPH● XtreemFS● FhGFS● ...


Distributed storage: Paradigm changes

● Block -> Object● Central -> Distributed

● Few -> Many● Big -> Small

● Server <-> Storage


Distributed storage – Now what?!?

● Several implementations● Different functions● Support models● Storage vendors initiatives● Relation to Linux distributions

Here and now ==> GlusterFS


High level overview


History

● Gluster founded in 2005● Gluster = GNU + cluster● Acquisition by Red Hat in 2011● Community project

● 3.2 in 2011● 3.3 in 2012● 3.4 in 2013

● Commercial product: RedHat Storage Server


The Client

● Native● 'speaks' GLUSTERFS● Not part of the Linux Kernel● FUSE-based

● NFS● Normal NFS client stack

● libgfapi (since 3.4)


The Server

● Data● Bricks● Translators● Volumes -> exported/served to the client

● Meta-Data● No dedicated instance● Distributed hashing approach


The 'traditional' picture


Storage inside


The Brick● Trust each other● Interconnect

● TCP/IP and/or RDMA/Infiniband

● Dedicated file systems on GlusterFS server● XFS recommended, EXT4 works too● Extended attributes a must

● Two main processes/daemons● glusterd and glusterfsd


The Translator

● One per purpose● Replication● POSIX● Quota● I/O behaviour● ...

● Chained -> brick graph● Technically: configuration


The Volume

● Service unit● Layer of configuration

● distributed, replicated, striped, ...● NFS● Cache● Permissions● ....


The Distributed Volume


The Replicated Volume


The Distributed-Replicated Volume


And more ...

● Striped● Striped-Distributed● Striped-Replicate● ....


Meta Data

● 2 kinds● More of local file system style● Related to distributed nature

● Some stored in backend file system ● Permissions● Time stamps● Distribution/replication

● Some calculated on the fly● Brick location


Elastic Hash Algorithm

● Based on file names & paths● Name space divided● Full brick handled via relinking● Stored in extended attributes● Client needs to know topology


Distributed Hash Tables


Self-Healing

● On demand vs. Scheduled● File based● Based on extended attributes● Split-brain

● Quorum function● Sometimes: manual intervention


Geo replication

● Asynchronous● Based on rsync/ssh● Master-Slave ● If needed: cascading ● One way street● Clocks in sync!● Coming: Parallel GeoRep


From files to objects

● Introduced with version 3.3● Hard links with some hierarchy

● Re-uses GFID (inode number)

● UFO● Unified File and Object● Combination with RESTful API● S3 and Swift compatible


Operations:Growth, shrinkage .. failures

● A Must!● Easy● Rebalance!● Backup?!?● Order of servers important


What else ...?

● Encryption :-|● Compression :-(● Snapshots :-(● Hadoop connector :-)● Locking granularity :-|● File system statistics :-)● Monitoring :-(


Team up with OpenStack


OpenStack

● Infrastructure as as Service (IaaS)● 'Opensource version' of AWS :-)● New versions every 6 months

● Previous (2013.1.3) is called Grizzly● Current (2013.1.4) is called Havana

● Managed by Openstack Foundation


OpenStack Architecture


OpenStack Components

● Keystone - identity● Glance – image● Nova - compute● Cinder – block storage● Swift – object storage● Quantum - network● Horizon - dashboard


About Swift

● Replace Amazon S3● Scalable ● Redundant

● Openstack object store● Proxy● Object ● Container ● Account● Auth


Why GlusterFS in the first place?

● Scale out storage● HA + self healing● Easy integration on O/S level● Already exposed to similar workloads


Replacing Swift – a story in 3 acts

● Sneaking in● Step in and throw out● Teaming up


Replacing Swift – the 'lazy' way

● Before 3.3● Just mounting● Works for Glance too ...● ... and even for Cinder


Replacing Swift – the next version

● With 3.3● Changes to Swift code● UFO (Unified File and Object)● Mainly proxy server● Helper tools● Meta data -> Extended attributes ● One volume per tenant


Replacing Swift – limitations

● Dependent on Swift release● Missing/late new development● Packaging


Replacing Swift – Gone!

● Since 3.4 and Grizzly● RedHat big player in Openstack● UFO renamed to G4O


And Cinder?

● Block Storage (since Folsom)● Integration similar to Swift

● Mounting

OR● Since Grizzly & GlusterFS 3.4:

– /etc/cinder/cinder.conf

– /etc/cinder/shares.conf– Still fuse mounted volume


What else?

● Storage for VM images (Glance)● 'Lazy' mode -> one volume● Better way with future versions

● NAS type


New with Havana release

● Glance can point to Cinder interface● Disk space savings

● Cinder can use libgfapi● Performance

● Nova integration via libgfapi● Performance● QEMU assisted snapshotting


Down the road

● Manila● OpenStack's shared FS service● Different possible frontends

● Savanna● Elastic Hadoop on OpenStack● Analytics as a Service


Reviewed: Why GlusterFS?

● Previous arguments still valid● One way to cover different

storage entities● Modular usage● Separation of duties possible● Co-location with other/foreign

workloads


Summary


Take aways

● Thin distributed file system layer● Modular architecture● Operationally ready● Good integration in Openstack● Active development and community


References

● http://www.gluster.org● http://www.sxc.hu (pictures)


Thank you!


Bricks and Translators:GlusterFS as SWIFT replacement in OpenStack

Dr. Udo SeidelLinux-Strategy @ Amadeus

gluster.community.day.2013

Travel

glusterfs glusterfs

amadeusglusterfs day

topologyglusterfs day

openstackglusterfs day

cinderglusterfs day

dashboard glusterfs

tenant glusterfs day

glusterfs server xfs