How we cooked Elasticsearch, Consul,Haproxy and DNS-recursor

In comic strip

Authors:Shcherbakov S.

Sin A.Tokarev O.

DNS

I resolve my elasticsearch location by DNS

And send request to it

Gimme my data!

Whe

re is

my

Ela

stic

sear

ch?

What happens if Elasticsearch down?

DNSW

here

is m

y E

last

icse

arch

?

Ok. I will hide my Elasticsearch(es) behind HAproxy!!!

Data request

Service monitoring

DNS

Ok. Now I can access my data even if one of elasticsearch is down!

Gimme my data!

Whe

re is

my

Ela

stic

sear

ch?

DNSW

here

is m

y E

last

icse

arch

?

^#%$*&#%*%@*&%##*&$%399493

Ok. Let's add extra balancer for redundancy!

DNS

Ok. Now DNS points to one of balancer.But what will happen

if one of balancer down??

Gimme my data!

{OR}

Whe

re is

my

Ela

stic

sear

ch?

DNSW

here

is m

y E

last

icse

arch

?

Gimme my data!

DNS

Whe

re is

my

Ela

stic

sear

ch?

Gim

me

my

data

!

DNSW

here

is m

y E

last

icse

arch

?

Gimme my data!

It makes me crazythat DNS continues give me

An address of brokenserver

How to make DNS forget broken balancer serverASAP?!?

Make Consul cluster monitor your balancers healthand inform your NS about IPs of health ones

(Yep, Consul can speak in DNS language)

DNS

Forward zone

Consul cluster

Ok! Now if one of my balancer downConsul will return IPs of health ones only

Consul is authoritative for a zonewith my critical services

Not enough thermometers???

DNS

Consul clusterNotify Admin!

Retrieve services IPs

Three safety loops

Node level

Disaster notifications

(zabbix)

Application lo-level

Automated application cluster load balancing

(HAproxy)

Application hi-level

Automated application address discovering

(Consul, DNS recursor)

Do you thing that it is the DNS server that is the Single Point of Failure?

ZABBIX!!!

But it does not affect any component of production system directlyNeither customers are affected

As for DNS servers they can be readily deployed in required amountwith resolv.conf updated with new servers IPs

DNS

Still works..NOT BAD!!

HAproxy

/etc/haproxy/haproxy.cfg

frontend elasticsearch bind *:9200 default_backend elasticsearch

backend elasticsearch server es01 123.123.123.1:9200 check inter 1000 rise 10 fall 5 server es02 123.123.123.2:9200 check inter 1000 rise 10 fall 5 server es03 123.123.123.3:9200 check inter 1000 rise 10 fall 5 option httpchk GET /_cluster/health http-check expect ! string "status":"red"

Consul agent (on HAproxy nodes)/opt/consul/config/srv-es-lb.json

{ "service": { "name": "es-lb", "port": 9200, "checks": [ { "interval": "5s", "http": "http://localhost:9200", "timeout": "1s" } ] }}

Consul looks like as NS from outside

[root@consul01 ~]# dig @consul01 -p 8600 es-lb.service.martyshka +short185.66.251.35185.66.251.34[root@consul01 ~]#

PDNS recursor

/etc/pdns-recursor/recursor.conf

packetcache-ttl=0

/etc/pdns-recursor/conf.d/forward-zones.conf

forward-zones=martyshka=consul01:8600;

Resolver (all nodes)

/etc/resolv.conf

search openprovider.nl hosting-concepts.nl nameserver <recursor 1 IP>nameserver <recursor 2 IP>nameserver <provider NS 1 IP>nameserver <provider NS 2 IP>nameserver 8.8.8.8

DNS

Consul clusterNotify Admin!

Retrieve services IPs

● Cloud● Cluster

● Fail-over● Redundancy

● Fault tolerance

“fashionable stylish for youth”