how we used ruby to build locaweb's cloud (
TRANSCRIPT
MOTIVATION
WORKING AT LOCAWEBon the past 3 years
A LOT OF KNOWN PEOPLE
WHICH PROBLEM WE WANTED TO SOLVE
CLOUD SERVER
CLOUD SERVER TECHNOLOGIESPerlJavaVMware
VMWARE ARCHITECTURE
PROBLEMSfor both technical and product sides
CLOUD SERVER PRO
CLOUD SERVER PRO TECHNOLOGIESRubyXenserver
XEN ARCHITECTURE
XENSERVER, XENAPI AND XMLRPCwith a python example
XENAPI EXAMPLE - GET DISKS
reference:
vm_ref = session.VM.get_by_uuid(virtual_machine.uuid)vm_vbds_refs = session.VM.get_record(vm_ref)["VBDs"]disks = vm_vbds_refs.inject({}) do |disks, vm_vbd_ref| vm_vbd_record = session.VBD.get_record(vm_vbd_ref) if vm_vbd_record["type"] == "Disk" disks[vm_vbd_ref] = vm_vbd_record enddisks
Xenserver API documentation
XENAPI-RUBY HELPER FOR THIS CODE# Get all disks for a VMdisks = virtual_machine.vbds
XENAPI-RUBY PROVIDES SOME HELPERSBut the way is still validXenAPI
STARTING INFRASTRUCTUREOne pool, one firewall (providing NAT) and one DHCP server
SOMETHING LIKE THIS...
ASYNC ALL OVER THE PLACEstarted with and ActiveMQ Stomp
DIFFERENT QUEUE IMPLEMENTATIONSXMPP, ActiveMQ (+ Stomp), Resque
RESQUE IMPLEMENTATION EXAMPLE
app/consumers/resque/consumer.rb
require "consumers/resque/configuration"
module Resque class Consumer
class << self def queue(name) @queue = name end
...
XMPP IMPLEMENTATION EXAMPLE
app/consumers/xmpp/consumer.rb
require 'xmpp4r'
module XMPP class Consumer
class << self attr_reader :queue_name, :queued_process, :timeout_value
def queue(name) @queue_name = "#{name}@localhost" end
...
EASILY CHANGEABLE
app/consumers/consumer.rb
# -*- coding: UTF-8 -*-Consumer = Resque::Consumer
RESQUE WONBTW
VIRTUAL MACHINE INSTALLATION PROCESShttp://github.com/locaweb/bpmachine
BPMACHINE EXAMPLEprocess :of => :install do must_be :machine_created
transition :select_ips, :from => :machine_created, :to => :ips_selected
transition :queue_dhcp_for_install, :from => :ips_selected, :to => :dhcp_synchronized
...
AND THE STEPS
app/steps/install_steps.rb
module InstallSteps def select_ips log_activity(:info, :id => id, :status => 'starting') create_ip(primary=true) unless primary_ip_pair log_activity(:info, :id => id, :status => 'done') end
def queue_dhcp_for_install ... end
GET THE NAME AND IPA different app to handle these
DHCP CONFIGURATIONisc-dhcp
DUMMY DHCP EXPLANATION
ISC DHCP FILE EXAMPLE# cpro0007host 3ea808bd-5ef0-4227-a617-f5b0694e408c{ hardware ethernet 00:25:22:bd:d1:20; fixed-address 186.202.1.7; option host-name "cpro0007";}
# cpro0051host 3ea808bd-5ef0-4227-a617-cf5b0694e408 { hardware ethernet 00:25:22:bd:d1:21; fixed-address 186.202.1.8; option host-name "cpro0051";}
ERB FILE<% vms.each do |vm| %># <%= vm.name %>host <%= vm.uuid %> { hardware ethernet <%= vm.mac %>; fixed-address <%= vm.private_ip.address %>; option host-name \"<%= vm.name %>\";}<% end %>
SIMPLE SCRIPT FOR DHCP# Prepare isc dhcp fileecho -e "<%= dhcp_config %>" > /etc/dhcp/<%= Config[:dhcp_conf_file] %>
# Restart DHCPsudo /etc/init.d/isc-dhcp-server restart
NET::SSH AS TRANSPORT LAYERNet::SSH.start(@host, @user, ssh_options) do |ssh| ssh_result = ssh.open_channel do |channel| channel.request_pty do |chn, success| raise "Could not obtain pty from ssh" unless success chn[:out] = "" chn.exec command
...
FIREWALL CONFIGURATIONSdefault DROP
ADDING SOME FIREWALL RULESiptables -A <%= internal_address %>/32 \-p <%= rule.filter_protocol %> \-s <%= rule.filter_address %> \-d <%= internal_address %> \--dport <%= rule.filter_port %> \-j ACCEPT
CONFIGURING NAT# Configuring NAT...
iptables -t nat -A PREROUTING -d <%= rule.external_address %> \-j DNAT --to-destination <%= rule.internal_address %>
iptables -t nat -A POSTROUTING -s <%= rule.internal_address %> \-j SNAT --to-source <%= rule.external_address %>
...
EXAMPLE XVP FILEPOOL XENSERVERPOOL1 DOMAIN "" MANAGER root proxy_password_encrypted_hash HOST 10.20.30.40 VM 7890 4344dc8f-1bdd-4b65-812a-a0dc9b27256e console1_encrypted_pass VM 7891 c5b2d28e-4c11-492f-9d09-33670876cb4a console2_encrypted_pass
GENERATED FROM XVP BINARY# for hosts/bin/echo -e "proxy_password_here" | /usr/sbin/xvp -x
# for each console/bin/echo -e "console_password_here" | /usr/sbin/xvp -e
WAIT...XVP IS OPEN SOURCE
RUBY VERSION
coisa linda!
def encrypt_vnc(password) key = [0xc1, 0x24, 0x08, 0x99, 0xc2, 0x26, 0x07, 0x05] des = OpenSSL::Cipher::Cipher.new("des-ecb") des.key = key.map(&:chr).join des.encrypt des.update(password).unpack('H*').firstend
POST INSTALL PROCESSVM already shipped by Xen
WINDOWS? LINUX?Specialized post install
DONE BY VIRTUAL MACHINE TYPEdef post_installers { :windows2003 => PostInstall::Windows, :windows2008 => PostInstall::Windows, :linux => PostInstall::Linux }end
...
def post_install post_installers[code.to_sym].newend
SIMPLIFIED, OF COURSE... WE HAVE:centos5, centos6, ubuntu9.04, ubuntu9.10, ubuntu10.04,
ubuntu10.10, debian5, debian6, windows2003, ...
AND MANAGED ONES!linux + cpanel, linux + plesk, windows + plesk, ...
LINUX POST INSTALLSSH and bash scripts
PARTITIONING DISKS EXAMPLE for Linux and for WindowsLVM Diskpart
MORE AND MORE STEPSAdded basic monitoringConfigure DNSVM cleanupand so on...
AND WE'RE DONE!production ready
SHIPPED WITH A SIMPLE ARCHITECTURE
MULTI POOL"all your pools are belong to us"
MULTI POOL
MATRIX MACHINESand their maintenance
A WILD WINDOWS MATRIX MACHINEAPPEARED!
SECURITY FIX RELEASED
SECURITY FIX APPLIED
OK...BUT WE HAVE MORE!
WE CAN DEAL WITH IT!
BUT....WAT!
NUMBERS!~ 7000 VMs~ 25 pools~ 300 hosts
XEN IMPORT AND EXPORTnot so well documented
USING HTTP!def export(vm_uuid, options = {}) options = {:to => "/tmp/export_file"}.merge(options) file = File.open(options[:to], "wb") session_ref = self.key task_ref = self.task.create "export vm #{vm_uuid}", "export job"
path = "/export?session_id=#{session_ref} ... " uri = URI.parse "http://#{master_address}#{path}"
Net::HTTP.get_response(uri) do |res| res.read_body {|chunk| file.write chunk } end options[:to]ensure file.close rescue nil self.task.destroy(task_ref) rescue nilend
MULTI STORAGEor...datasets!
MULTI FIREWALLavoiding package overflow
SO BASICALLY...WE HAVE THIS:
TIME TO BREAK RESPONSABILITIESby improving the infrastructure
NEW NETWORK INFRASTRUCTUREdedicated servers for firewall and dhcp
BEFORE
NOW
QUANTUM + OPENVSWITCH to use Locaweb quantum plugin openvswitch
NOT SSH ANYMOREQueueing and consuming using RabbitMQ
NEW FIREWALL AND DHCP ARCHITECTURE
NEW HYPERVISORS TO SUPPORTKVM, VMware
A PROXY FOR HYPERVISORS
SIMPLESTACK INSTEAD OF XENAPI DIRECTLY
Using
# Old xenapi codesession.VM.get_by_uuid(virtual_machine.uuid)
# Simplestackpool.on_simplestack.guests.find(virtual_machine.uuid)
ruby simplestack client
FEATURES WORTH MENTIONING
AUTOSCALE AND ALERTSdriven by monitoring
THE FLOW
JUST MESSAGING
BLATHER AND DAEMON TOOLSrequire 'blather/client'DaemonKit::Application.running!
message :chat?, :body do |m| begin if Weatherman::Message.should_process?(m) message = JSON.parse(m.body, :allow_nan => true)["results"]
unless message["event"].nil? Weatherman::Message.process(message["event"]) end end rescue => e DaemonKit.logger.error "Invalid message #{m.body}" DaemonKit.logger.error e.backtrace endend
HETEROGENEOUS ENVIRONMENTnew features and support for all versions!
XEN POOL KNOWS ITS NETWORK VERSION# -*- coding: UTF-8 -*-module Network module Manager extend self
def for(pool) pool.network_version.constantize.new end endend
DIFFERENT BEHAVIORS FROM VERSION 1# -*- coding: UTF-8 -*-module Network class Ver1 def internal_vlan? true end
def installation_ip(vm) vm.public_ip.address end endend
TO NETWORK VERSION 2# -*- coding: UTF-8 -*-module Network class Ver2 def internal_vlan? false end
def installation_ip(vm) vm.on_simplestack.ip end endend
MULTIPLE IMPLEMENTATIONS FOR:network configurationsfirewall implementationspost install methods
LESSONS LEARNEDStart small and grow fastOOP helps a lot to keep the code organizedMultiple projectsMultiple languages
THANKS!BY WILLIAN MOLINARI (POTHIX)