1

Live Migration of Virtual Machines

Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hansen†, Eric Jul†, Christian Limpach, Ian Pratt, Andrew Warfield

University of Cambridge Computer LaboratoryDepartment of Computer Science University of Copenhagen, Denmark

Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation(2005), USENIX Association

Presented by Yu-Han Li

2

Outline

IntroductionDesign- Options- Overview- Writable Working Sets- Dynamic Rate-Limiting

EvaluationConclusion

3

Introduction

• Migrating OS instances across distinct physical hosts is a useful tool for administrators of clusters

• O.S. virtualization allows many OS instances to run concurrently on a single physical machine

• Live OS migration– Migrating an entire OS and all of its applications as a unit

• Avoid residual dependencies

4

Introduction

• Live OS migration (cont.)– In-memory state can be transferred in a consistent and

efficient fashion

• Implement migration support for Xen– A open source virtual machine monitor

• Interest– Downtime– Total migration time– Resource contention with the migrating OS

5

Design

• Options – Migrating Memory– Push phase

• Source VM continues running while pages are pushed to new destination

• Pages modified must be re-sent

– Stop and copy phase• Source VM is stopped• Pages are copied to destination VM• The new VM is started

6

Design

• Options – Migrating Memory (cont.)– Pull phase

• The new VM executed.• If it accesses a page that has not yet been copied, the

page is pulled across the network from source VM

– Pre-copy migration• A bounded iteration push phase with

– Pages to be transferred during round n – Pages to be transferred are modified during round ( n-1 )

7

Design

– Pre-copy migration (cont.)• A very short stop-and-copy phase

– Some pages update very frequently which are poor candidates for pre-copy

• Options – Network resources– A migrating VM will carry its IP address

( include protocol state)– Generate an unsolicited ARP reply from the

migrated host

8

Design• Overview

9

Design

• Writable Working Sets (WWS)– Q: how to determine when it’s time to stop the

pre-copy phase because too much time is wasted?S: dirty pages faster than the rate of copying

• Should immediately stop and copy

– A certain set of pages will seldom be modified and are good candidate for pre-copy

– The remainder pages will be transferred via stop-and-copy ( WWS )

10

Design

– Xen’s shadow page table• Track dirtying statistics on all pages used by a

particular executing OS• Determine within any time period the set of

pages written by VM– Observe the WWS under a processor

• VM has a memory allocation of 512MB• One VM started benchmark• Another VM read dirty bitmap

11

Design

• Estimating Migration Effectiveness– Trace data acquired to estimate the effectiveness

of iteration pre-copy– Know the approximate WWS behavior

• Estimate overall amount of data transferred in the final stop and copy and estimate the downtime

– For a 512MB VM using native stop-and-copy• 32, 8 (sec) downtime for 128, 512 (Mbit/sec)

12

Design• Expected downtime due to last-round memory copy on traced page

13

Design

• Managed Migration– Migration demons– It performs rounds of copying

• Perform a complete scan of the VM’s memory pages.• In Subsequent round, copy the pages dirtied during previous

round by dirty bitmap– Bitmap is cleared– Shadow page tables are destroyed and recreated

– When pre-copy phase is no longer beneficial• OS suspend itself in a state suitable for migration• Demon scanned for remaining inconsistent pages with VM’s

checkpointed CPU-register state

14

Design

• Dynamic Rate-Limiting– It is not appropriate to select network bandwidth limit for

migration traffic– Dynamically adapt the bandwidth limit during each pre-

copying round• Select a minimum and a maximum bandwidth limit• The 1st round:

– transfer at the minimum bandwidth• Each subsequent round: Bandwidth limit

– previous round’s dirtying rate + Constant– Dirty rate = #pages dirtied in previous / duration of the previous

– The final stop-and-copy phase• Minimize service downtime

15

Evaluation

• Test Setup– 2 Dell PE-2650 server-class machines– Dual Xeon 2GHz CPUs– 2GB Memory– Broadcom TG3 network interface– Gigabit Ethernet switch– XenLinux 2.4.27 OS

16

Evaluation

• Simple Web Server

17

Evaluation

• Complex Web Workload (SPECweb)

18

Evaluation

• Low latency server (Quake 3)

19

Conclusion

• Integrating live OS migration into the Xen VMM– Enables rapid movement of interactive workloads

• Dynamic network bandwidth adaption – Minimal impact on running services

• Realistic server workloads (SPECweb99)– Migrated with just 210ms downtime