building cloud stack at scale

www.shapeblue.com

Building CloudStack at Scale

Paul AngusCloud Architect ShapeBlue

[email protected]: @ShapeBlue

www.shapeblue.com

Overview

The design phaseRedundancy & resilienceAutomationDocumentation

Building CloudStack at Scale

www.shapeblue.com

The Design Phase

Tips for the design phase

www.shapeblue.com

Get everyone in the room

The Design Phase

www.shapeblue.com

Know the fine print

The Design Phase

www.shapeblue.com

Understand your work loads

The Design Phase

www.shapeblue.com

Understand your bottlenecks

Design Phase

www.shapeblue.com

Redundancy & Resilience

Designing for Redundancy and

Resiliency

www.shapeblue.com

CloudStack Management Server

MySQL Servers

Virtualised Management Farm


www.shapeblue.com

CloudStack Management Server

Very easy to set up additional management servers

Load balancing required to give high availabilty


www.shapeblue.com

MySQL

Master / Slave is ‘standard’

Alternatives includeMySQL Proxy (Mirroring)Galera ClusterMMM


www.shapeblue.com


Load Balance

r

MySQL

Galera

MySQL

MySQL

Load Balancer

CS Man

CS Man

www.shapeblue.com


F5

MySQL

Galera

DC1 DC2

MySQL

MySQL

CS Man

CS Man

F5

F5

MySQL

Galera

MySQL

MySQL

CS Man

CS Man

F5

www.shapeblue.com

Server ‘pairs’

MySQL masters and slavesCS Man & MySQL mastervCenter & MS SQL serverAny other redundant servers i.e. DNS


www.shapeblue.com

Laws of probability….

Same failure probability as raid 0If management server OR MySQLmaster fails – downtime for the whole management system.


www.shapeblue.com

Automation

Automation of the infrastructure build

www.shapeblue.com

Uses:Why automateAutomation toolsExamples

Configuring management VMsBuild a CloudStack 4.0 management serverDeployment of hostsConfiguration of CloudStack (through API)

Automation

www.shapeblue.com

Work smarter, not harder

Automation

www.shapeblue.com

Automation gives you:

SpeedEfficiencyRepeatabilityAccuracy

Automation

www.shapeblue.com

Tools:

Chef, Puppet

KickStart, Python

Shell scripting

Automation

www.shapeblue.com

Example:

Automation

Base Build of Management Servers using Shell

Scripting

www.shapeblue.com

Automation – Management VM Configuration

Create Deployment VM

CentOS (Minimal) + wget

Download CSV & hostconfig script

Make it executable,run it.

www.shapeblue.com

The CSV:


VMhostname Searchdomain PrimaryNICIPPrimaryNIC

PrefixPrimaryNICG

atewayPrimaryNIC

Network SecondaryNICIPSecondaryNIC

PrefixSecondaryNIC

Network DNS1NTPSERVERIP

1

CSMan1 cs4.net 10.14.16.67 27 10.14.16.65 Tools 10.144.72.21 21 Mgmt 10.128.9.9 64.238.19.20

CSMan2 cs4.net 10.14.16.68 27 10.14.16.65 Tools 10.144.72.22 21 Mgmt 10.128.9.9 64.238.19.20

MySQL1 cs4.net 10.14.1.35 27 10.14.1.33 Database 10.141.163.4 26 Mgmt 10.128.9.9 64.238.19.20

MySQL2 cs4.net 10.14.1.36 27 10.14.1.33 Database 10.141.163.5 26 Mgmt 10.128.9.9 64.238.19.20

www.shapeblue.com

#!/bin/bash

echo "Please enter hostname [without domain]: ie CSMan1"

read VMHOSTNAME

SearchDomain=`grep \$VMHOSTNAME hostconfig.csv | awk -F ',' '{print \$2}'`

PrimaryNICIP=`grep \$VMHOSTNAME hostconfig.csv | awk -F ',' '{print \$3}'`

PrimaryNICIP=`grep \$VMHOSTNAME hostconfig.csv | awk -F ',' '{print \$3}'`

PrimaryNICPrefix=`grep \$VMHOSTNAME hostconfig.csv | awk -F ',' '{print \$4}'`

PrimaryNICGateway=`grep \$VMHOSTNAME hostconfig.csv | awk -F ',' '{print \$5}'`

PrimaryNICNetwork=`grep \$VMHOSTNAME hostconfig.csv | awk -F ',' '{print \$6}'`


www.shapeblue.com

echo "DEVICE=\"eth0\"BOOTPROTO=noneNM_CONTROLLED=\"no\"ONBOOT=yesTYPE=\"Ethernet\"IPADDR=$PrimaryNICIPPREFIX=$PrimaryNICPrefixGATEWAY=$PrimaryNICGatewayDEFROUTE=yesIPV4_FAILURE_FATAL=yesIPV6INIT=noNAME=\"$PrimaryNICNetwork\" "> /etc/sysconfig/network-scripts/ifcfg-eth0

echo "# ShapeBlue Script created file

search $SearchDomain

nameserver $DNS1

nameserver $DNS2

" > /etc/resolv.conf

echo "# ShapeBlue Script created file

NETWORKING=yes

HOSTNAME=$VMHOSTNAME

" > /etc/sysconfig/network

echo "$NTPSERVERIP1

$NTPSERVERIP2" > /etc/ntp.conf

Reboot


www.shapeblue.com

# if vm management use this one

if [ "$SecondaryNICNetwork" == “Mgmt" ]; then

echo "ADDRESS0=10.128.9.9

NETMASK0=255.255.255.255

GATEWAY0=10.14.16.1

ADDRESS1=10.128.3.13

NETMASK1=255.255.255.255

GATEWAY1=10.14.16.1

ADDRESS2=64.238.199.202

NETMASK2=255.255.255.255

GATEWAY2=10.141.163.1

ADDRESS3=213.212.65.202

NETMASK3=255.255.255.255

GATEWAY3=10.141.163.1

ADDRESS4=213.212.68.168

NETMASK4=255.255.255.248

GATEWAY4=10.141.163.1

ADDRESS5=213.212.69.0

NETMASK5=255.255.255.0 GATEWAY5=10.141.163.1" > /etc/sysconfig/network- scripts/route-eth1

fi


www.shapeblue.com

Example:

Automation

Build a CS4 management server

www.shapeblue.com

Automation – CS4 Management Server



Set static IP address and ensure it has an

FQDN

Download the script, make it executable,

run it.

www.shapeblue.com

#!/bin/bash

setenforce permissive

sed -i "/SELINUX=/ c\SELINUX=permissive" /etc/selinux/config

echo "

[cloudstack]

name=cloudstack

baseurl=http://cloudstack.apt-get.eu/rhel/4.0/

enabled=1

gpgcheck=0" > /etc/yum.repos.d/cloudstack.repo

yum update -y

yum install ntp cloud-client mysql-server -y

sed -i -e '/datadir/ a\innodb_rollback_on_timeout=1' -e '/datadir/ a\innodb_lock_wait_timeout=600' -e '/datadir/ a\max_connections=350' -e '/datadir/ a\log-bin=mysql-bin' -e "/datadir/ a\binlog-format = 'ROW'" /etc/my.cnf

chkconfig ntpd on

chkconfig mysqld on

chkconfig nfs on

chkconfig rpcbind on

service ntpd restart

service mysqld restart

service rpcbind start

service nfs start

/usr/bin/mysqladmin -u root password 'password'

cloud-setup-databases cloud:cloud@localhost --deploy-as=root:password

cloud-setup-management

Automation – CS4 Management Server

www.shapeblue.com

Example:

Automation

Host deployment server build using shell scipting

www.shapeblue.com



Download to VM Hypervisor installation media (inc. XenServer

Updates)

Download build script,make it executable,

run it.

Automation – Host Deployment

www.shapeblue.com

Deployment server (VM)

The script downloads, builds and configures:DHCPPXE (TFTP)HTTP server

Script writes the scripts needed to for PXE boot of XenServer & ESXi hostsScript also writes the answer files and post installation scripts to configure XenServer and ESXi hosts


www.shapeblue.com

Automation – Host Deployment# write DHCP config file

echo "# DHCP for PXE

ddns-update-style interim;authoritative;log-facility local7;default-lease-time 600;max-lease-time 7200;option subnet-mask $DHCPNETMASK;option broadcast-address $DHCPBROADCAST;option routers $DHCPGATEWAY;option domain-name-servers $DNS1;option domain-name \"$DOMAIN\";

subnet $DHCPSUBNET netmask $DHCPMASK {range $DHCPSTART $DHCPFINISH; }

allow booting;allow bootp;use-host-decl-names true;option option-128 code 128 = string;option option-129 code 129 = text;option space gpxe;option gpxe-encap-opts code 175 = encapsulate gpxe;option gpxe.bus-id code 177 = string;

next-server $PXESERVERIP;filename \"/gpxelinux.0\";

" > /etc/dhcp/dhcpd.conf

www.shapeblue.com

# create pxelinux.cfg/default file

echo "# script built pxeconfig

DEFAULT menu.c32

menu title ShapeBlue Host Deployment

timeout 0

LABEL ESXi5.0

KERNEL http://$PXESERVERIP/esxi/mboot.c32

APPEND -c http://$PXESERVERIP/esxipxeboot/boot.cfg pxebooting +++

IPAPPEND 1

MENU LABEL ESXi-5.0.0

" > /tftpboot/pxelinux.cfg/default


www.shapeblue.com

# Create xenserver answerfile

echo "<?xml version=\"1.0\"?>

<installation srtype=\"ext\">

<primary-disk gueststorage=\"yes\">sda</primary-disk>

<keymap>uk</keymap>

<root-password>P@ssword</root-password>

<source type=\"url\">http://$PXESERVERIP/xenserver/</source>

<script stage=\"filesystem-populated\" type=\"url\">http://$PXESERVERIP/xenpxeboot/postxeninstall.sh</script>

<admin-interface name=\"eth0\" proto=\"dhcp\" />

<timezone>Europe/Dublin</timezone>

<time-config-method>ntp</time-config-method>

<ntpservers>$NTPSERVER1</ntpservers>

<nameserver>$DNS1</nameserver>

</installation>

" > /var/www/html/xenpxeboot/xenanswerfile


www.shapeblue.com

MAC IP HOSTNAME STORAGEIP

80:c1:6e:6a:b5:9c 10.14.73.9 XenHost1 10.14.97.9

80:c1:6e:6a:b4:0c 10.14.73.10 XenHost2 10.14.97.10

80:c1:6e:6a:f3:fc 10.14.73.11 XenHost3 10.14.97.11

80:c1:6e:6a:f0:ec 10.14.73.12 ESXiHost1 10.14.97.12

80:c1:6e:6a:b5:a4 10.14.73.13 ESXiHost2 10.14.97.13


www.shapeblue.com

echo “# sample XenServer post install script

wget http://$PXESERVERIP/hostconfig.csv

MACADD=\`ifconfig | grep eth0 | awk '{print \$NF}'\`“

IPADDR=\’grep -i \$MACADD /tmp/arp_table.csv | awk -F ',' '{print \$2}‘\’

XENHOSTNAME =/`grep -i \$MACADD /tmp/arp_table.csv | awk -F ',' '{print \$3}‘/`

STORAGEIP=/`grep -i \$MACADD /tmp/arp_table.csv | awk -F ',' '{print \$4}‘/`

STORAGENICUUID=/’xe pif-list | awk '\$0 ~ str{print b}{b=\$0}' str="eth3" | awk '{print \$NF}‘/`

xe host-list params=uuid --minimal > /tmp/tmphostuuid

xe pif-list management=true params=uuid --minimal > /tmp/tmpnicuuid

read NICUUID < /tmp/tmpnicuuidread HOSTUUID < /tmp/tmphostuuid

xe host-set-hostname-live host-uuid=\$HOSTUUID host-name=\$XENHOSTNAME

xe host-param-set name-label=\$XENHOSTNAME uuid=\$HOSTUUID

xe pif-reconfigure-ip uuid=\$NICUUID mode=static IP=\$IPADDR netmask=$HOSTNETMASK gateway=$HOSTGATEWAY DNS=$DNS1,$DNS2

xe pif-param-set uuid=\$NICUUID other-config:domain=$DOMAIN

" > /var/www/html/xenpxeboot/firstfullbootxenscript.sh


www.shapeblue.com

echo "#!/bin/bash

# Install XenServer Updates

xe host-list params=uuid --minimal > /tmp/tmphostuuid

files=\$(ls /tmp/*.xsupdate 2> /dev/null | wc -l)

if [ \"\$files\" != \"0\" ]; then

read HOSTUUID < /tmp/tmphostuuid

cd /tmp

for updatefile in /tmp/*.xsupdate; do

PATCHUUID=\$(xe patch-upload file-name=\$updatefile)

xe patch-apply host-uuid=\$HOSTUUID uuid=\$PATCHUUID

rm -f \$updatefile

PATCHACTION=\$(xe patch-list uuid=\$PATCHUUID params=after-apply-guidance --minimal)

echo \"Patch Action:\" \$PATCHACTION

if [ \"\$PATCHACTION\" == \"restartXAPI\" ]; then

xe-toolstack-restart

else

reboot

exit

fi

done

fi

# Disable boot script for subsequent reboots

rm -f /etc/rc3.d/S99postinstall

# Final Reboot

reboot

" > /var/www/html/xenpxeboot/xenupdatescript.sh


www.shapeblue.com

Automation

Utilising the API when building the infrastructure

www.shapeblue.com

The API

What it is

How can you use it

Examples

Automation

www.shapeblue.com

What is the API

The API is the real engine of CloudStack

The web GUI is simply making API calls

Port 8096 by default

Automation – The API

www.shapeblue.com

Using the API

CloudStack GUIBrowserWord, ExcelUsing Firebug/IE Developer Tools with CloudStack


www.shapeblue.com

API commands directly through a browser

Immediate response


www.shapeblue.com

API calls from a Word document or Excel spread sheet


www.shapeblue.com

Using Firebug / IE Developer Tools

IE press F12 or ‘view developer tools’

Firefox install Firebug add-on


www.shapeblue.com

www.shapeblue.com

Global Settings

http://csman:8096/client/api?command=updateConfiguration&name=vmware.management.portgroup&value=svc-console

http://csman:8096/client/api?command=updateConfiguration&name=allow.user.create.projects&value=false

http://csman:8096/client/api?command=updateConfiguration&name=allow.public.user.templates&value=false

http://csman:8096/client/api?command=updateConfiguration&name=apply.allocation.algorithm.to.pods&value=true

http://csman:8096/client/api?command=updateConfiguration&name=cpu.overprovisioning.factor&value=2

http://csman:8096/client/api?command=updateConfiguration&name=vm.allocation.algorithm&value=random


www.shapeblue.com

Getting informationhttp://csman1:8096/client/api?command=listServiceOfferings

http://csman1:8096/client/api?command=listTemplates&templatefilter=featured


www.shapeblue.com

Deploying an instance

Base command:http://csman1:8096/client/api?command=deployVirtualMachine

The required options:

The Service Offering and Template IDs&serviceofferingid=XXX&templateid=XXX&zoneid=XXX&domainid=XXX&account=XXX

Optional options:&displayname=xxx


www.shapeblue.com

Deploying an instance

http://csman1:8096/client/api?command=deployVirtualMachine&serviceofferingid=XXX&templateid=XXX

Can be used to create a large number of instances very quickly


www.shapeblue.com

Documentation

A word on documentation

www.shapeblue.com

Dull, boring, tedious, slow – Crucial.

Write what you’re going to doFollow what you wroteUpdate itWith redundant servers; follow it againWith scripts ‘snapshot’ and start again

Documentation

www.shapeblue.com

Paul [email protected]: @ShapeBluewww.shapeblue.com

Any Questions ?

www.shapeblue.com

Thank you

Paul

building cloud stack at scale

Technology