Download - Redis Meetup NYC November 2014
Bill Anderson, Redis Mad Scientist
Beyond Simple Failover
Using Sentinel for more than just managing a master/slave pair.
Who Am I?
❖ Rackspace’s Resident Redis Mad Scientist!❖ Redis since 1.x !❖ github.com/therealbill/libredis and others!❖ Currently focused heavily on large scale deployments
Topics
❖ Overview / Review!❖ Alternate Uses of Sentinel!❖ Dynamic Sentinel!❖ Client Requirements!❖ Operational Perspectives!❖ Red Skull
Overview / Review
❖ Sentinel: A Redis option for managing which node in a Redis pod is master and which are slaves.!
❖ Can call scripts on events to trigger external events!❖ Publishes events to a PUBSUB channel for better system
integration.!❖ Uses a Quorum based election and promotion system
"So I have a Redis cluster..."
❖ Redis Pod!❖ One Master with at least 1 slave, no limit to slave count
or configuration!❖ Sentinel Constellation!
❖ Any number of Sentinel instances which manage a Pod!❖ Sentinels can overlap
Alternate Uses of Sentinel
❖ Sentinel as a monitoring tool!❖ Sentinel as a Service discovery mechanism!❖ Coming: info aggregator
Sentinel for Monitoring
❖ Replication Lag!❖ Track and Profile SDOWN events!
❖ Network insight !❖ System insight!❖ Redis delays - such as those caused by BGSAVE or
SAVE
Dynamic Sentinel❖ Sentinel API!
❖ Add, remove, reconfigure a pod!❖ Get current master!❖ Get current slaves!❖ Get other sentinels*!
❖ Useful for integration w/config management systems and removing Redis information from client code/configuration
Client Requirements
❖ API Usage: Any Redis Client or client/library which supports the sentinel command or allows a ”raw” command to be sent.!
❖ Client Usage: !❖ Option 1: Client which allows SENTINEL command,
implement connection management & control yourself!❖ Option 2: Client library which has “native” support for
using Sentinel.
Sentinel Event Handling
❖ Pubsub!❖ Each event has a dedicated channel!❖ masterchange!❖ +SDOWN !❖ -SDOWN!❖ Use to react or record events in your constellation and
pods.
Auditing Configuration
❖ Do you have enough sentinels to form a quorum?!❖ Do you have too many?!❖ Do all your masters have slaves?!❖ Are they promotable?!❖ Do the slaves have enough memory to handle the
master?
Operational Perspectives❖ Using Sentinel for discovery means more flexible
configuration for clients and client code!❖ Using Sentinel over config API improves integration with
system management and avoids file based management mechanisms (Chef, Puppet, etc.)!
❖ Visitor discovery of Redis through API provides snapshot information!
❖ Comparing discovered configuration to mandatory allows for automated audits
The Fine Print❖ When using failover quorum should be an odd number representing half+1
of your constellation.!❖ number_other_sentinels is not “live” - it is a high water mark. If you use it,
validate it.!❖ Same with sentinels in the “sentinel sentinels <master>” command. If you
use it, validate it.!❖ Sentinels must be able to talk to each other, the masters, and all of the
slaves.!❖ Sentinel is unauthenticated!❖ MONITOR/SENTINEL REMOVE must be done on all members of the
constellation
Sounds great. But ...
❖ Constellations, pods, nodes, sentinels, Oh my!!❖ Most changes have to be repeated across the
constellation!❖ I don't want to deal with all that fine print. !❖ I don't blame you. Neither do I. !❖ But I did!❖ And you don't have to
What does it do?
❖ Manages a cluster of sentinels!❖ Provides an API to handle constellation-wide actions in a
single call!❖ Audits key aspects of a constellation config !❖ Can correct some of the issues found in auditing!❖ Provides a window into your cluster via a web based
interface
Deployment
❖ Runs on each sentinel you operate!❖ Bootstraps itself from the config file for that sentinel!❖ One binary plus an HTML directory!❖ Easy docker container