Sponsored by:

& &

Real-time scheduling for virtual machines in SK Telecom

Eunkyu Byun

Cloud Computing Lab., SK Telecom

Cloud by Virtualization in SKT

•

• Provide virtualized ICT infra to customers like Amazon EC2 from SKT’s cloud resource pool exploiting server virtualization

• Resources : Servers/PC, Network, Storage, …

• Functionalities : load balancing, security solution, back-up, …

• Private cloud inside SKT – virtualized servers, virtualized I/O

• Migrate IT services on legacy servers to virtualized servers

• Provide employees with PaaS for software development

• Virtual desktop infrastructure for employees

One Common Cloud Computing Infrastructure

Cloud as Telecommunication Operator

• SK Telecom is a Telecommunication operator as well as a Cloud service provider

Legacy Network/Telecom Services on dedicated/reliable equipments

General-purpose Server farms for Cloud Hosting based on

virtualization technique

Advantage - Scale dynamically with demand - High utilization - Easy start-up of new services

Requirements 1. Guaranteeing time readiness 2. Scalability of services 3. Cost-effective secure storage

Virtual Telco

Case Study – Virtual Telco.

• IMS (IP multimedia subsystems) on Cloud

• Delivering IP multimedia services (VoIP, VOD, Instance Message, …) requiring session initiation between participants on Internet to users connected to wireless telecom networks

• Launch easily Internet services on wireless network/telecom infra

• Migrate servers for components into Cloud – require high reliability

Application Service

Session Management User Info. Database

Media Processing

IP network

SIP

Challenges

1. Guaranteeing time readiness 2. Scalability of services 3. Cost-effective secure storage

• Which virtualization technique, i.e., hypervisor, is most suitable for supporting real-time VM?

• We choose….

• Xen Hypervisor – best in responsiveness benchmark, open source

• Credit scheduler – default in Xen 4.1, known to be stable

• Second option : Credit 2 scheduler

Limitation of credit scheduler

0

10

20

30

40

50

60

CPU usage(sec) – Media Player VM

(weight)

Real-time VM can not occupy the proper amount of CPU even though with very high weight

CPU intensive VM makes use of the residual credits of non CPU-intensive (i.e. media player) VM’s credits

(sec) Contention between 6 CPU-intensive VMs(weight = 256) and 1 Media player VM

Need improvement!!

Research Goal

• Find improved soft real-time schedulers based on stable credit scheduler

• Fair CPU sharing – each VM occupies CPU (almost) exactly proportional to its weight + work-conserving

• Real-time support – fast responsiveness of real-time VMs

• Modify credit scheduler to distinguish realtime VM and non-realtime VM

• Realtime VMs are marked externally and treated specially to provide fast responsiveness

• Co-work with

Preempt based scheduling

CPU 0

Run Queue

: Under Priority

: Over Priority

: Boost Priority

VCPU 2 VCPU 7 VCPU 1 VCPU 0 VCPU 5 VCPU 4

VCPU 6 VCPU 3 VCPU 8

New Job to Schedule

: RT Priority

BOOST > RT > UNDER > OVER

Idea - Realtime VM’s VCPU is inserted to the runQ of a physical cpu at right after BOOST priority

Non-realtime VMs can run when RT VMs consume all given credits or are blocked

BOOST based scheduling (Min Lee, VEE’10 )

In the credit scheduler, VMs can get the highest priority (BOOST) when they receives events if they were blocked

However, VMs in runQ is not boosted

BOOST realtime VMs always they receives external event even they are in already in runQ

CPU 0

Run Queue

: Under Priority

: Over Priority

: Boost Priority

VCPU 2 VCPU 7 VCPU 1 VCPU 0 VCPU 5 VCPU 4

: RT Priority

External Event

Multi BOOST (by Korea Univ. at XenSummit, Aug, 2011)

Multiple BOOSTs at the same time

Driver domain and realtime VM cannot always get the highest priority

DRIVER_BOOST > RT_BOOST > BOOST > RT > UNDER > OVER

CPU 0

Run Queue

: Under Priority

: Over Priority

: Boost Priority

VCPU 2 VCPU 7 VCPU 1 VCPU 0 VCPU 5

VCPU 6 VCPU 3 VCPU 8

: RT Priority External Event !

Performance Evaluation

Physical server spec.

CPU AMD Phenom™ II X6 1055T (6 cores)

Memory 16GB

NET Gigabit Ethernet

Xen 4.1.1

VM VCPU:4, MEM:1GB

XenHypervisor

PCPU 1

PCPU 2

PCPU 3

PCPU 4

PCPU 5

PCPU 6

VM1 (micro bench)

VM2 (micro bench)

VM3 (micro bench)

VM4 (micro bench)

VM5 (micro bench)

VM6 (micro bench)

RT VM (media player)

V1 V2

V3 V4

V1 V2

V3 V4

V1 V2

V3 V4

V1 V2

V3 V4

V1 V2

V3 V4

V1 V2

V3 V4

V1

Micro bench - Not set as RT priority - repeat CPU-intensive computing during random time and sleep for random time - above 98% CPU usage

CPU, Network Usages

51.96

12.3

31.58

39.7 39.22 38.72

51.52

0

10

20

30

40

50

60

CPU usage(sec)

465.17

42.22

193.05

259.04 255.97

421.7

0

50

100

150

200

250

300

350

400

450

500

NET usage(MB)

Fairness CPU sharing according to Weight

50.81 51.33 88.91

136.43 184.54

50.97 50.58

44.82

36.65

28.47

50.64 50.64

44.71

36.55

28.38

50.76 50.85 44.64

36.6

28.46 50.78 50.42

44.63 36.56

28.51 50.8 50.84 44.73 36.67

28.52 50.77 50.64 44.71 36.67 28.44

0

50

100

150

200

250

300

350

400

unmodified weight(256)+preemption+boost+multiboost

weight(512)+preemption+boost+multiboost

weight(1024)+preemption+boost+multiboost

weight(2048)+preemption+boost+multiboost

RT VM1 VM2 VM3 VM4 VM5 VM6

RT VM runs CPU-intensive process – fully utilizing CPU

•CPU usages are proportional to weight value •Fair between RTVM and no-RTVM with equal weight

Work-conserving (sec)

51.958

12.3 31.582 39.698 39.219 51.524

57.113

53.712 52.258 52.377

49.599 57.306

53.699 52.113 52.35 49.572

57.16 53.741 52.152 52.381

49.586

56.979 53.588 52.171 52.338 49.57

57.237 53.576 52.205 52.361 49.589

57.314 53.689 52.129 52.294 49.548

0

50

100

150

200

250

300

350

400

no_contention unomdified weight weight+preemption weight+preemption+boost

weight+preemption+boost+multiboost

Media Dom0 VM1 VM2 VM3 VM4 VM5 VM6

Upper bound for media player

358.661 358.098 358.97 358.791 358.645

65.951

78.79%

15.98%

3.43%

15.93%

15.89%

15.96%

15.98%

15.94%

14.98%

8.81%

14.98%

14.95%

14.94%

14.97%

14.99%

14.53%

11.07%

14.57%

14.55%

14.56%

14.53%

14.54%

14.59%

10.93%

14.6%

14.59%

14.59%

14.57%

14.6%

13.85%

14.39%

13.86%

13.85%

13.85%

13.84%

13.85%

Weight for VM1~VM6 (non realtime VM) : 256 Weight for Media player VM( realtime VM) : 2048

Responsiveness – Test environment

Xen Hypervisor

PCPU 1

PCPU 2

PCPU 3

PCPU 4

PCPU 5

PCPU 6

VM1 (micro bench)

VM2 (micro bench)

VM3 (micro bench)

VM4 (micro bench)

VM5 (micro bench)

VM6 (micro bench)

RT VM (ping target)

V1 V2

V3 V4

V1 V2

V3 V4

V1 V2

V3 V4

V1 V2

V3 V4

V1 V2

V3 V4

V1 V2

V3 V4

V1

External server

ping request (per 0.01sec, 10000 counts)

Physical server spec.

CPU AMD Phenom™ II X6 1055T (6 cores)

RAM 16GB

NET Gigabit Ethernet

Xen 4.1.1

VM VCPU:4, MEM:1GB

Responsiveness (Ping RTT, Credit)

• The cumulative distribution of ping RTT as the number of simultaneous CPU-intensive VMs increases

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 1 2 3 4 5 6 7 8 9 10

no_contention contention_VM1 contention_VM2 contention_VM3

contention_VM4 contention_VM5 contention_VM6(cumulative distribution)

(ping RTT, ms)

+

All pings take only 0.5ms without contention

20% of ping takes longer than 10 ms when 7 VMs run simultaneously

Responsiveness (Ping RTT, modified)

(cumulative distribution)

(ping RTT, ms)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 1 2 3 4 5 6 7 8 9 10

no_contention contention_VM6 contention_VM6+preempt

contention_VM6+rtboost

contention_VM6+multiboost

contention_VM6preempt+multiboost

weight of realtime VM = 256 weight of non-realtime VM = 256

After applying our modification, all pings take only 0.5ms even with contention

What about Credit2 scheduler? CPU usage(sec) – Media Player VM

VM burn credits based on their weight

Higher weight means credits burn more slowly

VCPUs are inserted into the runQ by credit order

VM with more credits runs first

Credits are “reset” when the next vcpu in the runqueue is less than or equal to zero

(sec)

0

10

20

30

40

50

60

(weight)

Achieve both fairness and work-conserving

Responsiveness of Credit 2 scheduler

(cumulative distribution)

(ping RTT, ms)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 1 2 3 4 5 6 7 8 9 10

weight(256) weight(512) weight(768) weight(1024) weight(1536) weight(2048)

For fast responsiveness, VM needs higher weight. If we want to divide CPU cycle equally between VMs? Using special policy for realtime VM is necessary.

Ongoing Research

• What if there are several realtime VMs competing limited physical server/core ?

• Prediction based scheduling between real-time VMs

• Load balancing between physical cores

• Efficient placement policy of RTVMs between physical servers

• Load balancing between physical servers using live migration of VMs

Summary

• SK telecom is trying to operate Telco services on cloud resources

• Realtime support in hypervisor is essential

• Analyzed the performance of modifications of Credit scheduler of Xen hypervisor

• For one realtime VM per physical core, fair sharing and fast responsiveness

• Plan to improve for more complex and practical cases

Thank you

Contact: eunkyu.byun@sk.com

Comparison of hypervisers

• Evaluation environment

• Physical server : Dell R410 (Xeon 8 cores, 16GB Memory)

• Virtual Machine : 1 Core, 1 GB Memory, 20GB HDD

• Increase the number of VMs running benchmarks

• Benchmarks : PCMARK 2005, kernel compile, SPEC-CPU 2006

• A real-time application measure the delay of the timer interrupt handling in OS of VM

• Measure every 5 sec. for ten minutes

KVM 0.12.5

Vmware ESX 4.1

Xen 4.0

•Xen is the best one, but not sufficient

•Contention of non real-time VMs affects the responsiveness of real-time VM

Approach

• VM scheduler in the hypervisor is important

• Credit scheduler

• Stable (default scheduler in Xen hypervisor 4.0) , SMP support

• Need improvement for latency-sensitive VM

• Credit 2 scheduler

• Proportional sharing according to weight of each VM

• Provide responsiveness to VMs with larger weights

• Not so stable yet, need more analysis