cloud party 2014 - deploy your infrastructure with saltstack - salt cloud with openstack

CloudParty 2014Application and Infrastructure deployment

on Cloud Providersby /

- www.corley.it

Walter Dal Mut @walterdalmut

Corley S.r.l.

The ProblemYour web application never scale

It is really scalable?

Our daily goal?

A simple scalable web app

Why so many layers?Manage every layer by it-selfOptimize every layer by it-selfScale every layer by it-selfMonitor every layer by it-selfSecurize every layer by it-self...

The CloudAdd more resources when we need in secondsRemove resources when we don't need them anymoreReduce time to marketTurn a fixed cost into a variable costs

pay only for what you use

Deploy & OrchestationAutomate your infrastructure and deployment

Distributed Application

Split out your InfrastructureNetworksSubnetworks

Security GroupsWe will assign a different security-group for every group of VMs. In that

way we can apply our security policy in a simple and powerful way.

Security MapDatabase layer (MySQL)

PORT 3306 <- from web layerCache layer (Memcached)

PORT 11211 <- from web layerWeb layer (Apache2)

PORT 80 <- from proxy layerProxy layer (Nginx)

PORT 80 <- from everywhere

Security Groups

We will use Salt-CloudIt means that we also need to allow SSH connections from the "master"Every "minion" has also another security-group "salt-minion" that allows

SSH connections from the "salt-master" instance

Salt TOP.SLSbase:    '*':        ‐ basedev:    ...prod:    'proxy.milan.enter.*.prod':        ‐ nginx    'web.milan.enter.*.prod':        ‐ webserver        ‐ webapp    'cache.milan.enter.*.prod':        ‐ memcached    'rdb.milan.enter.*':        ‐ mysql        ‐ mysql.master    'srdb.milan.enter.*':        ‐ mysql

What about "salt-cloud"Salt cloud is made to integrate Salt into cloud providers in a clean way

so that minions on public cloud systems can be quickly and easilymodeled and provisioned. http://salt-cloud.readthedocs.org/en/latest/

salt-cloud for OpenStackWe need a Provider definition

enter‐openstack‐config:  minion:    master: 111.111.111.111

  identity_url: https://api‐legacy.entercloudsuite.com:5000/v2.0/tokens  compute_name: nova  protocol: ipv4

  compute_region: ItalyMilano1

  user: name@user.tld  password: YourPassword  tenant: name@user.tld

  provider: openstack

salt-cloud for OpenStackWe need VMs profiles

rdb:    provider: enter‐openstack‐config    size: e1standard.x4    image: GNU/Linux Ubuntu Server 12.04 LTS Precise Pangolin x64    ssh_username: ubuntu    ssh_key_file: /root/private‐key.pem    ssh_key_name: 'private‐key‐name'    ssh_interface: public_ips    security_groups: salt‐minion,mysql    networks:        ‐ fixed:            ‐ xxxxxxxx‐xxxx‐xxxx‐xxxx‐xxxxxxxxxxxxweb:    ...

Automatic IP management

Proxies need Web instaces IPs and so on...

Enable Peer communicationallow Salt minions to pass commands to each other

peer:  .*:    ‐ .*

We will use this features to share IP addressesYou can use "grains" or "mines" instead

Let's go

Database layerCreate a new resource with salt-cloud

salt‐cloud ‐p PROFILE VM‐NAME

salt‐cloud ‐p rdb rdb.milan.enter.1.prod

Deploy the Master RDBsalt 'rdb.*' state.highstate

Create a group of slavesCan we paralelize all VM creation?

salt‐cloud ‐Pp PROFILE VM‐NAME VM‐NAME ...

"-P" option means "parallel"salt‐cloud ‐Pp srdb \    srdb.milan.enter.1.prod \    srdb.milan.enter.2.prod \    srdb.milan.enter.3.prod

Deploy Slave RDBsalt 'srdb.*' state.highstate

Prepare all databasesNow we have a Master instance and 3 slaves

We have to prepare Read-ReplicasCHANGE MASTER TO    MASTER_HOST='xxx.xxx.xxx.xxx',    MASTER_USER='repl‐user',    MASTER_PASSWORD='repl‐pass',    MASTER_LOG_FILE='mysql‐bin‐xxxxx',    MASTER_LOG_POS=xxx

Execute MySQL commandssalt 'rdb.milan.enter.1.prod' mysql.query mysql 'show master status'

salt 'srdb.*' mysql.query mysql '    CHANGE MASTER TO    MASTER_HOST="xxx.xxx.xxx.xxx",    MASTER_USER="repl‐user",    MASTER_PASSWORD="repl‐pass",    MASTER_LOG_FILE="mysql‐bin‐xxxxx",    MASTER_LOG_POS=xxx    '

Now we have our databases

Now the cache layer

Add cache resourcessalt‐cloud ‐Pp cache \  cache.milan.enter.1.prod \  cache.milan.enter.2.prod \  cache.milan.enter.3.prod \  cache.milan.enter.4.prod

salt 'cache.*' state.highstate

Memcached will help us withcaching and session

management

When we distribute the loadacross a group of VMs all

information should be availableto the group otherwise we have

connectivity problemsCache warm up, disconnected users, and more...

Distribute the load

The Web TierAll Web VMs need to know DB and Cache nodes addresses

Master DB addressSlaves DB addressesSession VMs addressesCache VMs addresses

Distribute the load

Memcached Session handler# php.inisession.save_handler = memcachedsession.save_path = "192.168.0.5, 192.168.0.6, 192.168.0.7, 192.168.0.8"

{% set memcached_servers = [] %}{% for server,ip in salt['publish.publish']('cache.*', 'network.interfaces').items() %}{% set m = memcached_servers.append(ip.eth0.inet[0].address) %}{% endfor %}

session.save_handler = memcachedsession.save_path = "{{ memcached_servers|join(", ") }}

DatabaseThe problem: no default multiple connections

MySQLi($host, $username, $password);//PDO, ...

How to handle multiple connections? Write-Read and Read only?Master/Slave Async Replication

MySQL_NDReplace libmysql driver

Default connector in PHP 5.4

MySQL_ND Master/Slave (plugin)

MySQLND_MS Configuration{    "myapp": {        "master": {            "master_0": {                "host": "localhost",            }        },        "slave": {            "slave_0": {                "host": "192.168.2.27",            }        }    }}

Configuration in Salt{    "cwitter.db": {        "master": {            {% for server,ip in salt['publish.publish']('rdb.*', 'network.interfaces').items() %}            "master_{{ server[0] }}": {                "host": "{{ ip.eth0.inet[0].address }}"            }            {% endfor %}        },        "slave": {            {% for server,ip in salt['publish.publish']('srdb.*', 'network.interfaces').items() %}                "slave_{{ server[0] }}": {                "host": "{{ ip.eth0.inet[0].address }}"            }{% if not loop.last %},{% endif %}            {% endfor %}        },        "trx_stickiness": "master"    }}

Transaction AwareBy default MySQLND_MS is not transaction aware

trx_stickiness: master

BEGIN TRANSACTION

INSERT INTO ...DELETE FROMSELECT u1, u2, ... FROM ...UPDATE FROM

COMMIT

Now the application distributeuser sessions and DB queries

Distribute also HTTP requests!

Proxy HTTP/s requests

Proxy configuration needsPubic IPs

proxy:    provider: enter‐openstack‐config    size: e1standard.x1    image: GNU/Linux Ubuntu Server 12.04 LTS Precise Pangolin x64    ssh_username: ubuntu    ssh_key_file: /root/test.pem    ssh_key_name: 'my key name'    ssh_interface: public_ips    security_groups: salt‐minion,proxy    networks:        ‐ fixed:            ‐ xxxxxxxx‐xxxx‐xxxx‐xxxx‐xxxxxxxxxxxx        ‐ floating:            ‐ yyyyyyyy‐yyyy‐yyyy‐yyyy‐yyyyyyyyyyyy

NGINX as a proxyupstream app {    server 192.168.0.10:80;    server 192.168.0.11:80;    server 192.168.0.12:80;    # web server list}

server {    listen 80;

    location / {        proxy_pass http://app;    }}

NGINX proxy with Saltupstream app {    {% for server,ip in salt['publish.publish']('web.*', 'network.interfaces').items() %}    server {{ ip.eth0.inet[0].address }}:80;    {% endfor %}}

server {    listen 80;

    location / {         proxy_pass http://app;    }}

Use DNS round-robin feature inorder to resolve proxies's IP

The app is ready!

Thanks for listeningWalter Dal Mut

Github: Twitter: Linkedin:

wdalmut@walterdalmut

Walter Dal Mut