cloudstack architecture
TRANSCRIPT
CloudStack Architecture
4/29/2012Chiradeep Vittal
Alex Huang
Outline
• Overview of CloudStack • Problem Definition• Feature set overview• System VMs• System Architecture & Context• Component View
•Secure, multi-tenant cloud orchestration platform– Turnkey platform for delivering
IaaS clouds– Hypervisor agnostic– Scalable, secure and open– Open source, open standards– Deploys on premise or as a hosted
solution
•Deliver cloud services faster and cheaper
What is CloudStack?
Build your cloud the way the world’s most successful clouds
are built
CloudStack Supports Multiple Cloud Strategies
Multi-tenantPublic Cloud
• Dedicated resources
• Security & total control
• Internal network• Managed by
Enterprise or 3rd party
• Mix of shared and dedicated resources
• Elastic scaling• Pay as you go• Public internet,
VPN access
Hosted Enterprise Cloud
• Dedicated resources
• Security• SLA bound• 3rd party owned
and operated
Private Clouds Public Clouds
On-premise Enterprise Cloud
Compute
CloudStack Provides On-demand Access to Infrastructure Through a Self-Service Portal
Network Storage
Admin
Users
Org A
Admin
Users
Org BUsers
End User
Admin
Open Flexible Platform
Compute
XenServer VMware KVMOracle VM Bare metal
Hypervisor
Storage
Local Disk iSCSI NFSFiber Channel Swift
Block & Object
Network
Network Type Isolation Load
balancerFirewall VPN
Network & Network Services
Primary Storage Secondary Storage
Problem Definition
• Offer a scalable, flexible, manageable IAAS platform that follows established cloud computing paradigms
• IAAS– Orchestrate physical and virtual resources to offer self-service
infrastructure provisioning and monitoring• Scalable
– 1 -> N hypervisors / VMs / virtual resources– 1 -> N end users
• Flexible– Handle new physical resource types
• Hypervisors, storage, networking
– Add new APIs– Add new services– Add new network models
Problem Definition (contd)
• Manageable– Hide complexity of underlying resources– Rich functional end-user and admin UI– Admin API to automate operations– Easy install, upgrade for small -> large clouds– Simple scaling, automated resilience
• Established Paradigms– EC2 –inspired
• Semantic variations based on cloud provider needs, hypervisor capabilities
End-User Experience
Self-service Portal to Create & Manage VMs
Select Operating System• Windows, Linux
Select Compute Offering• CPU & RAM
Select Disk Offering• Volume Size
Select Network Offering• Network & Services
Create VM
Create Custom Virtual Machines via Service Offerings
Dashboard Provides Overview of Consumed Resources
• Running, Stopped & Total VMs
• Public IPs
• Private networks
• Latest Events
Virtual Machine Management
Users
Start
Stop
Restart
Destroy
VM Operations Console Access
• CPU Utilized
• Network Read
• Network Writes
VM StatusChange
Service Offering
2 CPUs
1 GB RAM
20 GB
20 Mbps
4 CPUs
4 GB RAM
200 GB
100 Mbps
Volume & Snapshot Management
Volume
VM 1Add / Delete
Volumes
Schedule Snapshots
Hourly
Daily
Weekly
MonthlyNow
Create Templates from Volumes
Volume Template
View Snapshot History
….
Network & Network Services
• Create Networks and attach VMs
• Acquire public IP address for NAT & load balancing
• Control traffic to VM using ingress and egress firewall rules
• Set up rules to load balance traffic between VMs
Pod 1
….
Cluster N
Access Layer
Host 2
Cluster 1
CloudStack Deployment Architecture
Host 1
Hypervisor is the basic unit of scale.
Cluster consists of one ore more hosts of same hypervisor
All hosts in cluster have access to shared (primary) storage
Pod is one or more clusters, usually with L2 switches.
Availability Zone has one or more pods, has access to secondary storage.
One or more zones represent cloud
PrimaryStorage
Zone 1
….
L3 core
SecondaryStorage
Pod N
CloudStack
Management
Server
Internet
CloudStack Cloud Architecture
- Do Not Distribute
Zone1
Data Center 1
Cloud
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Data Center 3
Zone 4 CloudStack Cloud can have one or more Availability Zones (AZ).
Management Server Managing Multiple Zones
- Do Not Distribute
Zone1
Data Center 1
Cloud
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Data Center 3
Zone 4
Management
Server
Single Management Server can manage multiple zones
Zones can be geographically distributed but low latency links are expected for better performance
Single MS node can manage up to 5K hosts.
Multiple MS nodes can be deployed as cluster for scale or redundancy
Management Server Deployment Architecture
- Do Not Distribute
Management
ServerMySQL
DB
Back UpDB
InfrastructureResources
User API
Admin API
Load Balancer
Management
Server
Management
Server
Management
Server
MySQLDB
InfrastructureResources
User API
Admin API
Single-node Deployment
Multi-node Deployment
MS is stateless. MS can be deployed as physical server or VM
Single MS node can manage up to 10K hosts. Multiple nodes can be deployed for scale or redundancy
Commercial: RHEL 5.4+; FOSS: Ubuntu 10.0.4, Fedora 16
Replication
- Do Not Distribute
Pod 1
Host 2
Cluster 1
Host 1PrimaryStorage
L3 switch
SecondaryStorage
L2 switch
CloudStack Storage
• Configured at Cluster-level. Close to hosts for better performance
• Stores all disk volumes for VMs in a cluster
• Cluster can have one or more primary storages
• Local disk, iSCSI, FC or NFS
Primary Storage
• Configured at Zone-level
• Stores all Templates, ISOs and Snapshots
• Zone can have one or more secondary storages
• NFS, OpenStack Swift
Secondary Storage
• Hosts• Servers onto which services will be provisioned
• Primary Storage• VM storage
• Cluster• A grouping of hosts and their associated storage
• Pod• Collection of clusters
• Network• Logical network associated with service offerings
• Secondary Storage• Template, snapshot and ISO storage
• Zone• Collection of pods, network offerings and secondary
storage
• Management Server Farm• Responsible for all management and provisioning tasks
Core CloudStack Components
Zone
CloudStack Pod
Cluster
Host
HostNetwork
PrimaryStorage
VM
VM
CloudStack Pod
ClusterSecondary
Storage
• Primary Storage• Cluster level storage for VMs• Connected directly to hosts• NFS, iSCSI, FC and Local
• Secondary Storage• Zone level storage for template, ISOs and
snapshots• NFS or OpenStack Swift via CloudStack
System VM
• Templates and ISOs• Imported into CloudStack• Can be private or public
Understanding the Role of Storage and Templates
Zone
Secondary Storage
Pod
Cluster
Host
HostPrimary Storage
Template
1. User Requests Instance
2. Provision Optional Network Services
3. Copy instance template from secondary storage to primary storage on appropriate cluster
4. Create any requested data volumes on primary storage for the cluster
5. Create instance
6. Start instance
Provisioning Process
Zone
Secondary Storage
Pod
Cluster
Host
HostPrimary Storage
VM
Template
XenServer Resource Pool
• Integrates directly with XenServer Pool Master
• Snapshots at host level
• System VM control channel at host level
• Network management is host level
Citrix XenServerCloudStack Manager
XenServer Pool Master Host
XenServer Host
XenServer Host
XenServer Host
XenServer Host
• Integrates with ovs-agent
• Snapshots at host level
• System VM control channel at host level
• Network management is host level
• Does not use OVM Manager
• All templates must be from Oracle
• CloudStack configures ocfs2 nodes
• Requires “helper” cluster• XenServer, KVM or vSphere
Oracle VMCloudStack Manager
OVM Host
OVS Agent
OVM Host
OVS Agent
OVM Host
OVS Agent
OVM Host
OVS Agent
• Integrates with libvirt using Cloud Agent
• Snapshots at host level
• System VM control channel at host level
• Network management is host level
• Only RHEL 6, not RHEV• Also supports Ubuntu 10.04
RedHat Enterprise Linux (KVM)
KVM Host
Cloud Agent
Libvirt
KVM Host
Cloud Agent
Libvirt
CloudStack Manager
• Integration through vCenter
• System VM control channel via CloudStack private network
• Snapshot and volume management via Secondary Storage VM
• Networking via vSphere vSwitch
VMware vSphereCloudStack Manager
Data Center
vSphere Cluster
vSphere Host
vSphere Host
vSphere Host
vSphere Cluster
vSphere Host
vSphere HostvCenter
Management Server Interaction with Hypervisors
Management Server
XenServer
ESX
vCenter
KVM
Agent
OVM
Agent
XAPI HTTP
• XS 5.6, 5.6FP1, 5.6 SP2, 6.0
• Incremental Snapshots
• VHD
• NFS, iSCSI, FC & Local disk
• Storage over-provisioning: NFS
• ESX 4.1, 5.0 (coming)
• Full Snapshots
• VMDK
• NFS, iSCSI, FC & Local disk
• Storage over-provisioning: NFS, iSCSI
• RHEL 6.0, 6.1, 6.2 (coming)
• Full Snapshots (not live)
• QCOW2
• NFS, iSCSI & FC
• Storage over-provisioning: NFS
• OVM 2.2
• No Snapshots
• RAW
• NFS & iSCSi
• No storage over-provisioning
Multi-tenancy & Account ManagementCloud
• Domain is a unit of isolation that represents a customer org, business unit or a reseller
• Domain can have arbitrary levels of sub-domains
• A Domain can have one or more accounts
• An Account represents one or more users and is the basic unit of isolation
• Admin can limit resources at the Account or Domain levels
Admin
Org A
Admin
Reseller A
Domain
Domain
Admin
Org C
Sub-Domain
User 1
User 2
Group B
Account
Group A
Account
VMs, IPs, Snapshots…
VMs, IPs, Snapshots…
Resources
Resources
Router
L3 Core Switch
Access Layer
Switches
………… …
Availability Zone
Servers
CloudStack Mgmt Server Cluster
Secondary Storage
Pod 1 Pod 2 Pod 3 Pod N
MySQL
Load Balancer
Operations Admin and Cloud API
Users
Physical Network
…
DB Security Group
WebSecurity Group
Layer 3 cloud networking
… …
Web VM
Web VM
Web VM
Web VM
DB VM
Web VM
DB VM
Web VM
Guest Networks with L3 isolationGuest
1 VM 1
Guest 2 VM 1
Guest 1 VM 2
Guest 2 VM 2
Public Internet
10.1.0.1
Public IP address 65.37.141.1165.37.141.2465.37.141.3665.37.141.80
Guest address 10.1.0.2Guest address 10.1.0.3Guest address 10.1.0.4
Guest address 10.1.16.12
Load Balancer
Guest 2 VM 3
Guest 1 VM 3
Guest 1 VM 4
Guest address 10.1.16.21Guest address 10.1.16.47Guest address 10.1.16.85
L3 Core Switch
Pod 1 L2 Switch
Pod 3 L2 Switch
10.1.16.1
…
…10.1.8.1Pod 2 L2 Switch
Hypervisor 1
Hypervisor N
Hypervisor 8
Access Switch(es)
VM Traffic
…
Pod K
CLUSTER 1
…
CLUSTER 4
Core (L3) Network
…
Pod M Pod N
Virtual Networks (L2 isolation)
Hypervisor N+1
Public Traffic
Hypervisor
R
R V
VV
V
HypervisorV V
V
RTenant VMTenant Virtual Router
Guest virtual layer-2 network
Guest 1 VM 1
Guest 1 VM 2
Guest 1 VM 3
Guest 1 VM 4
Public Internet
Public Network
Guest Virtual Network 10.1.1.0/24
Gateway address 10.1.1.1
NATDHCPLoad BalancingVPN
Public IP address 65.37.141.1165.37.141.36
Guest address 10.1.1.2Guest address 10.1.1.3Guest address 10.1.1.4Guest address 10.1.1.5
Guest 1 Virtual Router
Guest 2 VM 1
Guest 2 VM 2
Guest 2 VM 3
Guest Virtual Network 10.1.1.0/24
Gateway address 10.1.1.1
NATDHCPLoad BalancingVPN
Guest address 10.1.1.2Guest address 10.1.1.3Guest address 10.1.1.4
Guest 2 Virtual Router
Public IP address 65.37.141.2465.37.141.80
Layer-2 Guest Virtual Network
Public Network/Internet
Guest Virtual Network 10.1.1.1/8VLAN 100
Gateway address 10.1.1.1
DHCP, DNSNATLoad BalancingVPN
Public IP 65.37.141.11
10.1.1.1Guest VM 1
10.1.1.3Guest VM 2
10.1.1.4Guest VM 3
10.1.1.5Guest VM 4
CSVirtual Router
Public Network/Internet
Guest Virtual Network 10.1.1.1/8VLAN 100
Private IP10.1.1.112
DHCP, DNS
Public IP 65.37.141.112
10.1.1.1Guest VM 1
10.1.1.3Guest VM 2
10.1.1.4Guest VM 3
10.1.1.5Guest VM 4
NetScalerLoad
Blancer
Private IP10.1.1.111
Public IP 65.37.141.111
Juniper SRX
Firewall
CS Virtual Router provides Network Services External Devices provide Network Services
CSVirtual Router
Layer-3 Guest Network
Public Network65.11.0.0/16
65.11.1.2Guest VM
1
Guest VM 2
Guest VM 3
Guest VM 4
Public Network/Internet
NetScaler
Load Blancer
Network Services Managed Externally Network Services Managed by CS
65.11.1.3
65.11.1.4
65.11.1.5
DHCP, DNS
CSVirtual Router
Security Group 1
Security Group 2
10.1.2.3Guest VM
1
Guest VM 2
Guest VM 3
Guest VM 4
10.2.12.4
10.5.2.99
10.1.2.18
DHCP, DNS
CSVirtual Router
Security Group 1
Security Group 2
EIP, ELB
65.11.1.265.11.1.3
65.11.1.4
L3 switch
• Cloud provider defines the feature set for guest networks
• Toggle features or service levels– Security groups on/off– Load balancer on/off– Load balancer software/hardware– VPN, firewall, port forwarding
• User chooses network offering when creating network
• Enables upgrade between network offerings
• Default offerings built-in– For classic CloudStack networking
Network Offerings
CloudStack System VMs
• System VMs optimize and scale the datapath on behalf of CloudStack– Stateless, can be destroyed and recreated from database state– Highly Available– Communicates with Management Server over management network– Usually have 3 interfaces: control, guest and public
• Console Proxy VM – Provides AJAX-style HTTP-only console viewer– Grabs VNC output from hypervisor– Scales out (more spawned) as load increases– Java-based server Communicates with MS over message bus
• Secondary Storage VM– Provides image (template) management services– Download from HTTP file share or Swift– Copy between zones– Scale out to handle multiple NFS mounts– Java-based server communicates with MS over message bus
CloudStack System VMs
• Virtual Router VM – Provides multiple network services– IPAM (DHCP), DNS, NAT, Source NAT, Firewall, PF, VPN– User-data, Meta-data, SSH keys and password change server– Redundancy via VRRP– MS configures VR over SSH
• Proxied via the hypervisor on XS and KVM
Virtual Router Information (applies to all Sys. VMs)
• Debian 6.0 ("Squeeze"), 2.6.32 kernel with the latest security patches from the Debian security APT repository. No extraneous accounts
• 32-bit for enhanced performance on Xen/VMWare• Only essential software packages are installed. Services such as, printing, ftp, telnet, X, kudzu,
dns, sendmail are not installed.• SSHd only listens on the private/link-local interface. SSH port has been changed to a non-
standard port (3922). SSH logins only using keys (keys are generated at install time and are unique for every customer)
• pvops kernel with Xen paravirt drivers + KVM virtio drivers + VMware tools for optimum performance on all hypervisors. Xen tools inclusion allows performance monitoring
• Template is built from scratch and is not polluted with any old logs or history• Latest versions of haproxy, iptables, ipsec, apache from debian repository ensures improved
security and speed• Latest version of jre from Sun/Oracle ensures improved security and speed
System VM contd
• SSH keys and password are unique to cloud installation
• Code can be patched by restarting system vm– Mounts a special ISO file with latest code at boot– If ISO contents differ, patch and reboot
• Same system vm works on XS, KVM, VMWare– Bootstrap step for the cloud is to install the template
for this system vm• Ready to be re-purposed for other specialized tasks
Service Management (Billing, Metering, Accounts, etc.)Service Management (Billing, Metering, Accounts, etc.)
Resource ManagementResource Management Dynamic Workload ManagementDynamic Workload Management
Availability and SecurityAvailability and Security
BackupBackup Load BalancingLoad Balancing High AvailabilityHigh Availability MonitoringMonitoring
Image LibrariesImage Libraries
Application CatalogApplication Catalog
Custom TemplatesCustom Templates
Operating System ISOsOperating System ISOs
Inte
grati
on A
PIIn
tegr
ation
API
Ope
ratio
n, A
dmin
istra
tion,
M
aint
enan
ce a
nd P
rovi
sion
ing User InterfaceUser Interface
AdministratorAdministrator End UserEnd User ConsoleConsole
Developer APIDeveloper API
AmazonAmazon OpenStackOpenStack CustomCustom
Architecture Components
Virtualization Layer
Compute Network Storage
Interactions
CloudStack
Cloud user{API client (Fog/etc)}
End User UI
Admin UI
MySQL
CloudStackClustered
CloudStackManagement
Server
Domain Admin
UI
CS Admin & End-user API
Cloud user{ec2 API client }
ec2 API
Monitoring CS API vSphere ClusterPrimaryStorage
vcenter
Cluster Mgmt
XS ClusterPrimaryStorage
vCenter API
XAPI
KVM ClusterPrimaryStorageJSON
OVM Cluster PrimaryStorage
XenApi
NetConf
Nitro APIJuniper SRX
Netscaler
Console Proxy VMConsole
Proxy VM
JSON
Cloud user
HTTPSAjax Console
VNC
Sec. StorageVM
NFS Server
NFSSec. Storage
VM
HTTP (Template Download)
HTTP (Template Copy)
HTTP (Swift)
NFS
Router VMRouter VM
Router VM
JSON
{Proxied} SSH
Inside a Management Server
APIServlet
AsyncJob
QueueMgr
CloudStackAPI
ServicesAPI
Commands
Responses
cmd.execute()
Kernel
PluginsPlugins
Plugins
Message Bus
Agent Manager
ResourcesAgentAPI(Cmds)
HypervisorNativeAPIs
LocalOrRemote
NetworkDeviceAPI
MySQL