updating xen for the client environment · 2011-02-28 · 11 12/3/07 •xen hypervisor loaded at...

Updating Xen for the ClientEnvironment

Junhong Jiang, Kevin Tian,Chris Wright, Don Dugger

12/3/072

Legal Content

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTIONWITH INTEL® PRODUCTS. EXCEPT AS PROVIDED IN INTEL'STERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTELASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMSANY EXPRESS OR IMPLIED WARRANTY RELATING TO SALEAND/OR USE OF INTEL PRODUCTS, INCLUDING LIABILITY ORWARRANTIES RELATING TO FITNESS FOR A PARTICULARPURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANYPATENT, COPYRIGHT, OR OTHER INTELLECTUAL PROPERTYRIGHT.

Intel may make changes to specifications, product descriptions,and plans at any time, without notice.

Intel is a trademark of Intel Corporation in the U.S. and othercountries.

Copyright © 2007, Intel Corporation. All rights are protected.

12/3/073

Agenda

• Xen Client Overview

• Boot Scheme/Verified Launch

• Device Virtualization

• Power Management

12/3/074

Glossary

• VA Virtual Appliance

• Xen Client is a platform for implementing VAs

• UOS User Operating System/User OS/Primary User Partition

• SOS Service OS/VA Partition

• VM Virtual Machine/Guest Partition

• VMM Virtual Machine Monitor/Hypervisor

12/3/075

Xen Client

• Xen based virtual appliance infrastructure Limited device virtualization/isolation support

• Single SOS Linux kernel (Para-virtualized) No direct disk access at run time

• Permanent storage provided by UOS• Exception during boot process

•Dom0 provides disk access

• Target platform TXT support

• Measure and authenticated boot• Trust chain from BIOS

VT-d Support

• Virtualizes NIC TPM

12/3/076

Architecture Overview

• VMM Xen hypervisor

• 64 bits Dom0

• Para-virtualized Linux.• Runs out of RamFS• Minimal components

•Control Panel, device models, default power management policy, virtualappliance specific agents

• Headless•Capable of booting messages

• UP

• Service OS Paravirtualized XenLinux

• Fully virtualized possibility for future Headless

• Boot up display for SOS in Dom0 Survive even UOS hangs Support single or multiple SOS’s (At least one)

12/3/077

Architecture Overview (Cont.)

• SOS0 (LAN filtering) LAN Filtering

• Owns physical NIC• Provide VNIF BE driver

SMP capable

• SOS1 (VoIP) Own Audio SMP Capable

• User Operating System Near native performance Own rest physical resources (CPU, memory, devices) Virtual LAN

• VNIF front-end (FE) driver SMP Windows XP SP2 (32-bit PAE w/ XD support) Vista (32 and 64-bit) Windows PE Linux

12/3/078

Xen Client Architecture OverviewV

eri

fied

boot

vP

RO

Clie

nt C

fg./

Lau

nch

Pro

vis

ion

/In

stall

Ag

en

t P

rese

nce

Dom0Dom0

Vt-x LT VT-d EM64T Channel ACPI/PM

Shadow PT PIC/PIT Models

Xen hypervisorXen hypervisor

SOSSOS

Netw

ork

Dri

vers

VA

Serv

ices

Ag

en

ts

VA

Dri

vers

VA

Pro

vis

ion

ing

Ag

en

ts

VA

N/W

Rou

tin

g S

rvc.

VA

Sto

rag

e S

erv

ices

Oth

er

VA

Dri

vers

ISV Apps.ISV Apps.

UOSUOS

User Apps.User Apps.

VA

Dri

vers

VA

Serv

ices

Ag

en

ts

Nati

ve D

rivers

VA

Pro

vis

ion

ing

Ag

en

ts

VA

Sto

rag

e S

erv

ices

Oth

er

VA

Dri

vers

TPM HW LAN/WAN SATA, USB, etc.

vTP

M

12/3/079

Disk Layout

• Reserved SOS partition Type 0x71 Pointed to by MBR Contains boot/persistent storage file system

• Boot FS in SOS partition Contains grub loaded modules

• Xen kernel image• Dom0 vmlinuz• Dom0 initrd

SOS modules are not loaded by grub• Dom0 mounts the boot FS• SOS kernel/initrd loaded by domain builder in Dom0

12/3/0710

Disk layout

Part 3

Part 2

Part 1

Part 4 (type = 71)

UOS MBR

Boot Block

UOS Partition

SOS Boot Block

PartitionTableMaster Boot

RecordBoot code

PhysicalDisk

UOSusableDisk

SOSpartition

∞

0

SOS Partition

Part 3

Part 2

Part 1

Part 4 (type = 71)

12/3/0711

• Xen hypervisor loaded at fixed memory Affects minimum memory required

• Dom0 allocated at top of memory First partition loaded

• SOS’s loaded sequentially below Dom0

• UOS loaded at low memory Allows 1-1 guest physical to machine physical mapping

• Needed to execute platform BIOS, especially SMI’s Size of UOS dynamically determined

• Max memory minus fixed size for Xen, Dom0 & SOS’s

• Implications: UOS given contiguous machine physical address

• Hole in memory for Xen Xen hypervisor memory allocation change

Memory Layout

12/3/0712

Physical RAM layout – initial boot

0

640K

1M

256M

Max

BIOS

Grub Modules

Dom0 initrd

Dom0 image

Xen VMM image

12/3/0713

Physical RAM layout - runtime

0

640K

1M

256M

Max

BIOS

Xen VMM image

Dom0 RAM

SOS RAM

UOSgpfn == mpfn

UOSgpfn == mpfn

12/3/0714

Boot sequence – 4 phases

1. Grub loads Xen & Dom0 into memory

2. Dom0 starts

3. SOS starts

4. UOS starts

12/3/0715

Persistent Storage

• SOS runs out of RAM Needs access to persistent storage on disk

• SOS run a disk Front End driver Talks to disk Back End driver in Dom0 during boot Unmounts disk from Dom0 Shuts down disk FE from Dom0 Starts disk FE, now talking to UOS Remounts disk

12/3/0716

Verified Launch – Dom0Dom0 (Linux)

Standard Linux Drivers(VGA, Chipset NV, Disk)

Device Model(PCI Config,

Serial, ICH,..)

Domain Builder(xm/libxc

equivalent)

CMFM

InterVMcommunication driver

vTPM

VM PowerManager

Back-end diskdriver


SATA, USB, etc.Shared HWUser OptimizedHW

TPMDriver

Kern

elU

ser

12/3/0717

Verified Launch – SOS


Dom0 (Linux)

Standard Linux Drivers(VGA, Chipset NV,

Disk)


Serial, ICH,..)


equivalent)

CMFMvTPM

VM PowerManager

BE diskdriver


TPMDriver

Linux NICDriver

SOS

Libxc (user levelevent channel)

Back-endNIC driver

vTPM FEDriver

CMFM agent(default)

InterVMcommunication

driver

Front-endDisk

Driver

VA services

InterVMCommunication

driverBE TPM

driver

12/3/0718

Verified Launch – UOS


Dom0 (Linux)


Serial, ICH,..)


equivalent)

CMFMvTPM

VM PowerManager


TPMDriver

Linux NICDriver

SOS (Linux)


Back-endNIC driver

CMFM agent(default)


driver

Front-endDisk

Driver

BE TPMdriver

UOS (Windows)

FE NICdriver

StandardWindows

DrivervTPM FE

Driver

VA services


driver

VAservices

PS agent: Back-endDisk Driver

TPMDriver


driver


DeviceModel

12/3/0719

I/O Device Handling

• Default is devices handled directly by UOS Near native performance DMA support

• VT-d provides protection between guests• Non-VT-d platforms will depend upon `well behaved’ guests

• Special devices paravirtualized through SOS LAN Disk

• Xen Client Currently•UOS owned

• Xen Client Future•PCI IOV w/ secure LBA allows dual ownership

Audio• Xen Client Future

•PCI IOV provides secure sharing

12/3/0720

PCI Config Space Virtualization

Dom0Dom0

Agent Integrity

Boot and Startup

ProvisioningVt-x LT VT-d EM64T Channel ACPI/PM

Shadow PT PIC/PIT Models


SOSSOS

Netw

ork

Dri

vers

VA

Serv

ices

Ag

en

ts

VA

Dri

vers

VA

Pro

vis

ion

ing

Ag

en

ts

VA

N/W

Rou

tin

g S

rvc.

VA

Sto

rag

e S

erv

ices

Oth

er

VA

Dri

vers

ISV Apps.ISV Apps.User Apps.User Apps.

VA

Dri

vers

VA

Serv

ices

Ag

en

ts

Nati

ve D

rivers

VA

Pro

vis

ion

ing

Ag

en

ts

VA

Sto

rag

e S

erv

ices

Oth

er

VA

Dri

vers

TPM HW LAN/WAN SATA, USB, etc.

TPMDriver

ICH(PowerControl)

PCI Configspace

Device ModelsDevice Models

vTPM

UOSUOS

12/3/0721

I/O Device virtualization - PCI Handling

• Dom0 owns PCI bus enumeration SOS/UOS PCI bus enumeration is virtualized in its host domain.

• SOS vPCI bus enumeration in dom0• UOS vPCI bus enumeration in Dom0

•Policy of device assignment to UOS, other SOSes setup by managementpartition

•Virtualized devices are inserted into PCI hierarchy

• PCI device physical BAR = vBAR in UOS Help for PCI hot plug support and thus docking

12/3/0722

Device assignment – PCI hierarchy

• PV (S0S) Virtual PCI front end/back end frame

• Qemu based partition (HVM SOS) Using current Piix4 in qemu as virtual PCI hierarchy Attach assigned physical device to virtual PCI bus in qemu, like current VT-

d effort• Qemu maintains vCONFIG_SPACE to pCONFIG_SPACE mapping• See PCI express, depends on virtual chipset in qemu

• Native based partition (UOS) See exactly same PCI hierarchy as physical

• FE becomes the placeholder of detached devices Need a stub to maintain vCONFIG_SPACE to pCONFIG_SPACE mapping

• HVM PCI config space interception Trapped by Xen, and emulated by vPCI Device Model

• Handlers has filter to decide which field can be written through• Handlers further issues io/mmio/interrupt assignment based on captured info

12/3/0723

Xen Client: Device Assignment

• “Hide” all PCI devices except for the Dom0 assigneddevices from Dom0 device drivers

• Assign PCI devices to guest

• Attach the assigned device to QEMU vPCI bus

• Assign corresponding VT-d context entry to guestdomain

• Intercept PCI config access in Xen and Qemuappropriately

• Turn on VMCS IOPORT bit to allow IOPORT access toassigned devices

• Install P2M entry for MMIO access of the assigneddevice

• Xen intercepts physical device interrupts and re-injectsto the target guest domain with vIRQ

12/3/0724

Platform ACPI Component

Xen Client PM Architecture Overview

DOM0 PM

OSPM Driver

DOM0

VMPM (VM Power Manager)

Semi-Virtual Platform

PlatformACPI Component

UOS PM

TPM Driver

TPM HW

Registers BIOS Tables

Virtual PM ops Real PM opsVM exit/entryXen event

Shared HWUser OptimizedHW

SegregatedHW

PlatformHardware

FEBE FE

ACPIDriver/AMLInterpreter

VA Agent

ACPI Virtualization

PM Event Coordination PCI PM Virtualization

PV SOS(No PM)

SOS Device PM

Use at boot time only

Device Driver

Event notification (via evtchn driver)

Primary UOS

12/3/0725

Power Management

• UOS OSPM policy virtualized

• Policy defined by UOS and enforced by Dom0 UOS controls native devices SOS controls everything else

• Shared devices• CPU• Platform

UOS initiates PM policy (Suspend, Hibernate, C-state or P-statechanges)

12/3/0726

PM - Suspend

• UOS owns physical PCI bridges It will suspend bridges before SOS can suspend it’s devices Implication is that UOS must see virtualized bridges

• One flat PCI space•Could be issue with drivers assuming bus limitations

• Virtual copy of actual topology•Preferable solution

• Suspend UOS suspends devices

• Including virtualized devices (as requested by device D3 request) UOS suspends PCI bridges

• Including any virtualized bridges and conforming to the policy setup inDom0 (VMPM – VM power management)

UOS completes suspend and request ACPI suspend request (ICHI/O port write)

• ICH PM suspend requests suspend of SOSes if needed•Should avoid keeping any state in SOS/DOM0 for restart of SOS/DOM0

• Wake from Suspend Restarts from Xen in real mode and execute

12/3/0727

Conclusion

• This is real, download the Alpha release at

http://eit.et.redhat.com

• Questions? Comments? Epithets?

updating xen for the client environment · 2011-02-28 · 11 12/3/07 •xen hypervisor loaded at...

Documents