leadership election with spring cloud cluster
TRANSCRIPT
Locks and Leaders with SpringIntegration
Dave Syer, 2016 Twitter: @david_syer (see also @artem_bilan, @gprussell) Email: [email protected]
AgendaWhat is a distributed lock?
What is a leader election?
What is it good for?
How do I do it?
Some tools for thinking about failure
Warning: Here be Dragons!
A Simple Distributed Operation
Two Clients
Shuffle!
Shuffle!
Shuffle!
Shuffle!
Shuffle!
Shuffle!
Scary, Eh?
LocksExample code using java.util.concurrent.locks.Lock:
boolean acquired = false; try acquired = lock.tryLock(10, TimeUnit.SECONDS); if (acquired) // Do something unique! catch (InterruptedException e) Thread.currentThread().interrupt(); throw new RuntimeException("Interrupted"); finally if (acquired) lock.unlock();
Spring Integration: LockRegistrypublic interface LockRegistry Lock obtain(Object lockKey);
Locks with Spring IntegrationExample code using LockRegistry:
boolean acquired = false; try acquired = lock.tryLock(10, TimeUnit.SECONDS); if (acquired) // Do something unique! catch (InterruptedException e) Thread.currentThread().interrupt(); throw new RuntimeException("Interrupted"); finally if (acquired) lock.unlock();
(same code)
Locks and LeasesA distributed lock nearly always has a shelf life (it expires).
Technically, that makes it a "lease".
Without expiry system can't make progress when a lock holder dies.
Dragons(At least) two problems are lurking:
1. Acquiring a lock requires consensus.
2. Leases expire and holder can never be sure when that happens.
Read this: http://martin.kleppmann.com/2016/02/08/howtododistributedlocking.html
TL;DR If using a lock for "correctness" not "efficiency" be very careful.
Important: you can tune the system to adjust the probability, or how long it lasts, but fundamentally you cannotprevent the system from ever allowing more than one holder of a lock.
Leader Elections
Simple idea: if you hold a lock you are the leader.
What can you do with it?
Highly available globally unique things, often with messages
sequences
message aggregation
scheduling, e.g. cron service
Spring Integration: Leader InitiatorImplementations of leader election need to be able to start an election and fire events on granted and revoked.
Zookeeper
Hazelcast
Etcd(*)
Generic (lockbased)
For a user it looks like this (create a new bean which is a SmartLifecycle):
@Bean public LeaderInitiator leaderInitiator(CuratorFramework client, Candidate candidate) return new LeaderInitiator(client, candidate);
(*) No support for etcd v3. Probably dead.
Spring Integration: CallbacksCallbacks on leadership events:
public interface Candidate void onGranted(Context ctx) throws InterruptedException; void onRevoked(Context ctx); ...
See also:
@EventListener(OnGrantedEvent.class) public void start()
@EventListener(OnRevokedEvent.class) public void stop()
ClosingWhen to use locks and leaders: HA active/passive failover
How to avoid it: latency from restarting a failed app might be fine
Otherwise, be idempotent. For correctness, deduplication store has to be ACID.
Relational databases can be really useful, as is Zookeeper.
Careful with the physics.
Linkshttp://presos.dsyer.com/decks/locksandleaders.html
Sample code: https://github.com/SpringOnePlatform2016/dsyerlocksandleaders
Spring Integration: https://github.com/springprojects/springintegration
Spring Cloud Cluster: https://github.com/springcloud/springcloudcluster