openstack liberty summit 2015 vancouver - 01.orgdive into vm live migration openstack liberty summit...
TRANSCRIPT
Dive Into VM Live MigrationOpenStack Liberty Summit 2015
Vancouver
Michał Dulko
Michał Jastrzębski
Paweł Koniszewski
Why Bother?
o Imminent host failure
o Maintenance mode
o Optimal resource placement
Use Cases
o Cooling issues
o Storage problems
o Networking problems
o Your datacenter was struck by a flood
Imminent Host Failure
o Firmware upgrades
o Hardware upgrades
o Kernel upgrades
Maintenance Mode
o Reduce costs
o Move VMs closer to their storage to lessen network latency
o Stack more VMs on hosts to save power
o Increase resiliency
o Noisy neighbour separation
o Spread VMs across more hosts
Optimal Resource Placement
General Flow
o Live
o Consistent
o Transparent
o Minimal service disruption
Assumptions
Non-live migration (cold migration)o nova migrate <server>
True live migration (shared storage or volume-based)o nova live-migration <server> [<host>]
Block live migrationo nova live-migration --block-migrate <server> [<host>]
Migrations in OpenStack
Migration type Local storage Volumes Shared storage
Block LM ✓ ✗ ✗
True LM ✗ ✓ ✓
Block LM with read-only devices
✗ ✗ ✗
True LM with read-only devices
✗ ✗ ✓
Compatibility
o Pre-Migration
o Reservation
o Iterative pre-copy
o Stop and copy
o Commitment
Live Migration Process
Active VM on physical host A, host B selected by scheduler or preselected.
Pre-migration
Compute node A Compute node B
VM AActive
Pre-migration
Reservation
Iterative pre-copy
Stop and copy
Commitment
Confirm availability of resources on host B; reserve a new VM.
Reservation
Compute node A Compute node B
VM
A
VM
A
ACTIV
E
Compute node A Compute node B
VM AReserved
VM AActive
Pre-migration
Reservation
Iterative pre-copy
Stop and copy
Commitment
Memory is transferred from A to B and next dirtied pages are iteratively copied.
Iterative pre-copy
Pre-migration
Reservation
Iterative pre-copy
Stop and copy
Commitment
Compute node A Compute node B
VM APaused
VM AActive
Copying
Suspend VM and copy remaining pages and CPU state.
Stop and copy
Compute node A Compute node B
VM
A
VM
A
PAUSE
D
PAUSE
D
Pre-migration
Reservation
Iterative pre-copy
Stop and copy
Commitment
Compute node A Compute node B
VM APaused
VM APaused
Copy
Host B becomes primary host for VM A.
Commitment
Pre-migration
Reservation
Iterative pre-copy
Stop and copy
Commitment
Compute node A Compute node B
VM
A
VM
A
PAUSE
D
PAUSE
D
Compute node A Compute node B
VM AActive
Performance & reliability
o OpenStack does not allow triggering any operations on VM during LM
o VMs with intensive memory workload are hard to migrate
o LM generates heavy load on network
o Migrations between CN with different CPUs
o Memory oversubscription
Pitfalls
o OpenStack disallow any operation on ongoing LM
o You can use virsh instead to interact
Interacting With Live Migration
o Information about ongoing LM
virsh domjobinfo <domain>
Diagnosis
Time elapsed 1918595 ms
Data processed 410.137 GiB
Data remaining 4.600 GiB
Data total 16.008 GiB
Constant pages 144658
Normal pages 107307605
Normal data 409.346 GiB
Expected downtime 1023 ms
o Cancel on-going LM
virsh domjobabort <domain>
o Pause VM during LM
virsh suspend <domain>
Forcing Migration Finish
o QEMU
virsh qemu-monitor-command --hmp <domain> migrate_set_downtime <time (sec)>
o libvirtvirsh migrate-setmaxdowntime <domain> <time (sec)>
Tuning Maximum Downtime
o nova.conf setting
live_migration_flag += VIR_MIGRATE_AUTO_CONVERGE
Auto Converge
o nova.conf setting
live_migration_flag += VIR_MIGRATE_TUNNELLED
Tunneled Migration
Hypervisor
libvirt
Hypervisor
libvirt
Source Host Destination Host
o nova.conf setting
live_migration_flag -= VIR_MIGRATE_TUNNELLED
Tunneled Migration
Hypervisor
libvirt
Hypervisor
libvirt
Source Host Destination Host
o libvirt
virsh migrate-setspeed <domain> <speed (MiB/s)>
o nova.conf settings
live_migration_bandwidth = <speed (MiB/s)>
Tuning Bandwidth
o nova.conf settings
live_migration_flag += VIR_MIGRATE_COMPRESSED
XBZRLE Compression
Sent Page Cache
Updated Page
Delta Compression
Delta
Received Pages
Delta
ApplyDelta
Updated Page
Source Host Destination Host
o nova.conf
o live_migration_uri = qemu+tcp://%s/system
LM On Dedicated Network
Compute node A Compute node BManagement Network
VM AActive
VM APaused
o nova.conf
o live_migration_uri = qemu+tcp://%s-lm/system
o Set up your DNS to resolve hostnames with -lm suffix to IPs in your dedicated network.
LM On Dedicated Network
Compute node A Compute node BManagement Network
VM AActive
LM Network VM APaused
o CPU instruction set of source node needs to be a subset of CPU instruction set of destination node
Different CPUs Between Compute Nodes
Compute Node A Compute Node B
AVX
SSE2MMX
AVX
MMX
Passed
Live Migration
Failed
o This can be skipped by explicitly setting VM CPU model in nova.conf:
o cpu_mode = custom
o virt_type = kvm or virt_type = qemu
o And then you can set cpu_model
o List of supported named CPUs is in libvirt/cpu_map.xml
Different CPUs Between Compute Nodes
o LM to specific host does not use memory oversubscriptiono ram_allocation_ratio
Memory Oversubscription
Compute Node A2 GB RAM
Reported RAM = available - reserved
nova-conductor
2 GB
2 GB
2 GB
4 GB
nova-scheduler
ram_allocation_ratio = 2.0
o Skip it byo reserved_host_memory_mb=-2048
Memory Oversubscription
Compute Node A2 GB RAM
Reported RAM = available - reserved
nova-conductor
4 GB
4 GB
4 GB
4 GB
nova-scheduler
ram_allocation_ratio = 1.0
Secure Live Migration
o Everything can be sniffed!
o Migrated machines can contain sensitive data
o Legal issues with unencrypted data transfer
Why Security Matters?
o Hypervisor native encryption
o QEMU doesn’t support it
o libvirt tunneled transport
o live_migration_uri = qemu+ssh://%s/system
o live_migration_flag += VIR_MIGRATE_TUNNELLED
o Uses only one core
o IPSec tunnel between hosts
Encryption
0
0.5
1
1.5
2
2.5
3
QEMU+SSH QEMU+TCP
Tra
nsf
er
rate
[G
Bp
s]
Intel(R) Xeon(R) CPU E5-2690 v2
Intel(R) Xeon(R) CPU E5-2660 v3
Memory Access Is Critical
Future Of Live Migration
o Compress every page sent during LM
o zlib used for compression
o Configurable:
o Number of threads
o Comperession ratio
Multithreaded Compression
o Move workload immediately to destination host
Post-copy Live Migration
Compute node A Compute node B
VM AActive
VM APaused
Copying
o Cheap solution to finish live migration in a finite time
o VM needs to be rebooted in case of failure
o Heavy performance impact
Post-copy Live Migration
o Track memory transfer progress
o Detect possible problems and take actions
Active LM Monitoring In OpenStack
o Pause VM
o Abort LM
o See progress
o Change configuration on the fly:
o Maximum tolerable VM down time
o Transfer bandwith
Actions On Ongoing Live Migration
Your voice matters!o Mailing lists:
o [email protected] [email protected]
o Win The Enterprise group:o [email protected] (IRC: pkoniszewski)o [email protected] (IRC: inc0)o [email protected] (IRC: dulek)
Q&A (& disclaimers)Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com.
Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.
Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
© 2015 Intel Corporation.