Indiana Jonesand the Temple of NFS Ganesha HA

Kaleb KeithleyGluster Summit, Berlin

6 October, 2016

These aren't the droids you're looking for

pacemaker

glusterd

glusterfsd

pacemaker

glusterd

glusterfsd

pacemaker

glusterd

glusterfsd

pacemaker

glusterd

glusterfsd

Setup Pacemaker, Gluster, and create volume

But I was going into Tosche Station to pick up some power converters!

pacemaker

glusterd

glusterfsd

pacemaker

glusterd

glusterfsd

pacemaker

glusterd

glusterfsd

pacemaker

glusterd

glusterfsd

My HA Cluster

Create HA Cluster

Don't call me a mindless philosopher, you overweight glob of grease.

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

Create Resource Agents

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

I find your lack of faith disturbing.

Set Constraints

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha-active ?

grace-active ?

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha-active ?

grace-active ?pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha-active ?

grace-active ?

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha-active ?

grace-active ?

Fast ship? You've never heard of the Millennium Falcon?

Start ganesha.nfsd

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

If there's a bright center to the universe, you're on the planet that it's farthest from.

ganesha_mon notices ganesha.nfsd is running

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha

ganesha-activeganesha-grace

Help me, Obi-Wan Kenobi; you're my only hope.

Ganesha.nfsd dies

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha

ganesha-activeganesha-grace

I used to bullseye womp rats in my T-16 back home, they're not much bigger than two meters.

ganesha_mon deletes grace-active

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha

ganesha-active

In my experience, there's no such thing as luck.

surviving ganesha.nfsds placed into NFS-GRACE

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha

ganesha-active

Who's the more foolish? The fool, or the fool who follows him?

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha

ganesha_mon deletes ganesha-active

You came in that thing? You're braver than I thought.

VIP fails over

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsdblock

unblock

ganesha

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

ganesha-activeganesha-grace

VIP2

You can't win, Darth. If you strike me down I will become more powerful than you could possibly imagine.

Next Steps

● Switch to storhaug (anyone fancy a trip to Norway?)

● Fencing?

I've got a bad feeling abou this.

● https://github.com/linux-ha-storage/storhaug● http://clusterlabs.org/

I've got to rest before I fall apart

Indiana Jonesand the Temple of NFS Ganesha HA

Kaleb KeithleyGluster Summit, Berlin

6 October, 2016

These aren't the droids you're looking for

pacemaker

glusterd

glusterfsd

pacemaker

glusterd

glusterfsd

pacemaker

glusterd

glusterfsd

pacemaker

glusterd

glusterfsd

Setup Pacemaker, Gluster, and create volume

But I was going into Tosche Station to pick up some power converters!

Four node clusterPacemaker + Corosync runningGlusterd running + four brick volume is started

pacemaker

glusterd

glusterfsd

pacemaker

glusterd

glusterfsd

pacemaker

glusterd

glusterfsd

pacemaker

glusterd

glusterfsd

My HA Cluster

Create HA Cluster

Don't call me a mindless philosopher, you overweight glob of grease.

Glusterd invokes ganesha-ha.sh –setupSetup create the pacemaker cluster, plus some other

pacemaker voodoo magic, e.g.+ authenticates w/ all members+ creates cluster w/ name and members+ starts cluster+ waits for cluster to elect a domain controller (DC)+ sets quorum policy if < 3 members+ disables stonith...

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

Create Resource Agents

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

I find your lack of faith disturbing.

Creates resources (resource agents)+ Floating (or virtual) IPs (IPaddr) and

portblock/unblock+ ganesha_mon, ganesha_grace, ganesha_nfsd

What they do:+ ganesha_mon checks to see if ganesha.nfsd is

running or not+ ganesha_grace sends dbus signals to put

ganesha.nfsd into grace+ ganesha_nfsd fixes up /var/lib/nfs/. (but storehaug)

Then lots of pacemaker constraint location voodoo to make VIPs fail over in a deterministic manner

Set Constraints

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha-active ?

grace-active ?

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha-active ?

grace-active ?pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha-active ?

grace-active ?

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha-active ?

grace-active ?

Fast ship? You've never heard of the Millennium Falcon?

VIPs want to be where ganesha-active attribute existsganesha_grace run when ganesha-grace attribute

exists.

ganesha_grace uses a pacemaker feature called “notify” available to clone resources. If ganesha_grace stops on node2, the other ganesha_grace RAs are notified.

Start ganesha.nfsd

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

If there's a bright center to the universe, you're on the planet that it's farthest from.

After everything is set up, gluster/glusterd starts the ganesha.nfsds on all the nodes

ganesha_mon notices ganesha.nfsd is running

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha

ganesha-activeganesha-grace

Help me, Obi-Wan Kenobi; you're my only hope.

ganesha_mon sees that ganesha.nfsd is running and creates ganesha-active and ganesha-grace attributes.

VIP RAs get happy, clients mount using the Floating IP addrs.

Time passes…...

Ganesha.nfsd dies

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha

ganesha-activeganesha-grace

I used to bullseye womp rats in my T-16 back home, they're not much bigger than two meters.

Oops, one of the ganesha.nfsds died.

ganesha_mon deletes grace-active

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha

ganesha-active

In my experience, there's no such thing as luck.

ganesha_mon notices and deletes the ganesha-grace attr.

surviving ganesha.nfsds placed into NFS-GRACE

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha

ganesha-active

Who's the more foolish? The fool, or the fool who follows him?

Pacemaker notifies the other ganesha_grace RAs, which send their ganesha.nfsd a dbus message putting them into NFS-GRACE

The ganesha.nfsds do whatever they do in grace, e.g. migrate locks, etc.

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP2

block

unblock

ganesha

ganesha_mon deletes ganesha-active

You came in that thing? You're braver than I thought.

Then ganesha-mon deletes the ganesha-active attr….

VIP fails over

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsdblock

unblock

ganesha

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP1

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP3

block

unblock

ganesha

ganesha-activeganesha-grace

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP

block

unblock

pacemaker

glusterd

glusterfsd

mon

nfsd

grace

VIP4

block

unblock

ganesha

ganesha-activeganesha-grace

VIP2

You can't win, Darth. If you strike me down I will become more powerful than you could possibly imagine.

Which causes the floating IP addr to fail-over to the next node based on the location constraints.

When the operator restarts the ganesha.nfsd, everything “unwinds”. The ganesha.nfsds are put into grace, and the floating IP fails back to the node.

In storehaug (Jose’s version) gluster/glusterd don’t run ganesha-ha.sh –setup, that’s done by the admin (or some other thing TBD. gluster/glusterd doesn’t start the ganesha.nfsds. Instead they’re started by the ganesha_nfsd RA along with the Samba server after /var/lib/nfs is setup.

Next Steps

● Switch to storhaug (anyone fancy a trip to Norway?)

● Fencing?

I've got a bad feeling abou this.

● https://github.com/linux-ha-storage/storhaug● http://clusterlabs.org/

I've got to rest before I fall apart