libvirt: a virtualization api

libvirt: A virtualization API

MARCO GUAZZONE

Distributed Computing Systems Group (DCS)Department of Computer ScienceUniversity of Piemonte Orientale

[email protected]

September 8, 2008

Marco Guazzone (DCS) libvirt: A virtualization API September 8, 2008 1 / 45

Outline

1 Hypervisor APIsHypercall APIManagement API

2 libvirtlibvirt: Overviewlibvirt: Virtualization Support

3 ExamplesQEMUXenRemote Management

4 Conclusions

5 References


Hypervisor APIs

Two types of APIs:1 Hypercall API: used by guests for para-virtualization.2 Management API: used by management tools.

I In Xen, also known as Xen API

ImportantIn the following, we will focus our attention on the Xen hypervisor.


Hypervisor APIs: the Xen system

. . . From [6].


Hypervisor APIs: Hypercall API

APIs for letting a guest to perform privileged instructions

The hypervisor (not the kernel)has interrupt handlers installed.

When the application (in theguest) invokes a system call:

1 an interrupt is raised (trap)2 and is caught by the

hypervisor,3 which then passes the

control back to the guestOS, through anasynchronous eventnotification mechanism.

. . . From [6].


Hypervisor APIs: Hypercall APIExampleHow a Xen C hypercall might look like:

hypercall_ret = xen_op(operation, arg1, arg2, arg3,arg4);

ExampleResulting assembly-like routine:

_xen_op:mov eax, 4(esp)mov ebx, 8(esp)mov ecx, 12(esp)mov edx, 16(esp)mov esi, 20(esp)int 0x82ret



Hardware Virtual Machine (HVM), also known as hardware assistedvirtualization, has recently emerged.

Intel’s VT and AMD’s AMD-V extensions are the majors hardwaresupport technologies for virtualization.With HVM the use of additional protecting rings become lesscritical.A guest in an HVM environment can use the acceleratedtransitions to ring 0 for system calls (accelerated system calls),because it has not been moved from ring 0 to ring 1 (as happensinstead with para-virtualization).

Is this the best solution?



No! Hybrid virtualization seems to behave better.Since with HVM the guest OS is not modified for supportingpara-virtualization, it does not know that is running in a virtualenvironment and so it cannot take advantage of any of thevirtualization features.The result is that HVM might be slower than para-virtualization.Hybrid virtualization tries to take the best of each worlds:

I From HVM: makes use of accelerated system calls and exploit otherhardware assisted facilities, like nested page tables (NPTs) [3].

F With Nested Paging, a page table in the hardware takes care of thetranslation between the guest address of a virtual machine and thephysical address, reducing the overhead.

I From para-virtualization: use of light-weighted ad-hoc interfacesrather than relying on emulated hardware (e.g., for I/O).


Hypervisor APIs: (Xen) Management API

APIs used by user-space applications for management and VM’slife-cycle tasks.In Xen:

Xen API, the core API, are an XML-RPC based API.I xend listens for XML-RPC connections.I All that can be done with the xm tool is possible with Xen API.

Upon Xen API there are several bindings.I Each binding API sends an XML-RPC over the socket where an

instance of xend is listening.I xend handles the request itself or dispatch it on to the kernel’s

hypervisor interface and then on the hypervisor itself.Third party tools and libraries reside upon these bindings.

I libvirt is one of these libraries.


Hypervisor APIs: (Xen) Management APICurrently supported binding languages:

libxen, C binding.pyxen, Python binding.XenSdk.net, C# binding (in Citrix XenServer 4.1).Xen-CIM, a CIM provider for the DMTF virtualization schema.

I The Common Information Model (CIM) [9] is a family of openstandards, defined and published by the Distributed ManagementTask Force (DMTF), that defines how managed elements in an ITenvironment are represented as a common set of objects andrelationships between them. This is intended to allow consistentmanagement of these managed elements, independent of theirmanufacturer or provider.

I Is a modeling language, rather than a programming language.I Provides a uniform, generic and standard interface for accessing to

management facilities.F VMware provides a CIM layer too [16].

Java binding probably no more supported.


libvirt: What is it?

The libvirt toolkit provides a higher-level VM management interface fortools and applications, that is:

A set of command line utilities for interacting with the virtualizationcapabilities of the OS.A consistent set of API in C with the aim to provide support acrossdifferent virtualization tools.A CIM provider for the DMTF virtualization schema.

A project sponsored by the Red Hat’s Emerging Technology group [13].


libvirt: Goal

To provide all the operations needed to manage guests ordomains running on a single physical node.To supplies a stable interface that isolates upper-level softwarefrom changes in the underlying virtualization layer.

I Each virtualization layer would implement the libvirt interface onwhich the upper layer tools rely.


libvirt: Features

Virtualization support.Management of virtual machines, virtual network and storage.Remote management.

The libvirt library does not provide high-level multi-node managementfeatures such as load balancing.


libvirt: Componentslibvirt: the core C API layer.virtsh: a command line Cprogram which provides ashell environment and amanagement user interface.

I Can be used to create,pause, list, migrate andshutdown domains.

libvirtd: a C daemon formanaging guest instances andlibvirt virtual networks.

Hypervisor layer

libvirt API layer

libvirt tools layer

libvirtd virsh

libxen CIM XML

Other hypervisor drivers

API bindings

Application layer


libvirt: Virtualization support

Terminology:node: a single physical machine.hypervisor: a layer of software allowing to virtualize a node in aset of virtual machines with possibly different configurations thanthe node itself.domain: an instance of an operating system running on avirtualized machine provided by the hypervisor.


libvirt: Connections

Interaction with a virtualization technology based on connection.Use of URI to specify which driver a connection refers to.driver[+transport]://[username@][hostname][:port]/[path][?extraparameters]

I driver: the virtualization technology to interact with.I transport: the transport layer to use for connecting to the driver.I username: the credentials to use for connecting to the driver.I hostname: the (possible remote) host where the virtualization

technology resides.I port: the port where the virtualization technology listens for

connections.I path: a driver dependent path (e.g. the path to a Unix domain

socket).I extraparameters: additional optional parameters.

NULL and empty string URIs means “connect to the best availablelocal hypervisor”.

I Actually tries to connect to Xen.


libvirt: DriversThe virtualization technology to interact with:

ldom: the Sun LDoms virtualization technology [15].I Directly provided by Sun, starting from version 1.0.2 [14].

lxc: the LXC [1] Linux container system.openvz: the OpenVZ [2] Linux container system.qemu: the QEMU [5] emulator (also for the KVM [12] and Xenner[10] hypervisors).remote: a dummy driver for accessing to “remote” hypervisors.storage: storage on IDE/SCSI/USB disks, FibreChannel, LVM,iSCSI, NFS and filesystems.test: a dummy driver for testing purposes.xen: the Xen [7] hypervisor on Linux and Solaris hosts.Others undocumented/experimental/planned: uml (User ModeLinux), vserver (Linux V-Server, [11]), vmware (VMware), hyperv(Microsoft Hyper-V, [8]).

I Mostly found in libvirt-0.4.4/src/domain_conf.[hc].


libvirt: Transports

The protocol used for connecting to the virtualization technology:ext: use an external program which can make a connection to theremote machine by means outside the scope of libvirt.ssh: use an SSH connection (needs netcat and libvirtd on theremote machine).

I libvirt constructs an SSH command which looks like:command -p port [-l username] hostname netcat -Usocket

I port, username, hostname can be specified as part of the remoteURI.

I command, netcat and socket come from extra parameters (orsensible defaults).

tcp: use the TCP/IP transport protocol.tls: use a TLS connection (needs client and server certificates).unix: use a UNIX socket.


libvirt: Extra Parameters

An optional list of parameters following the syntax of RFC-2396.

Name Transports Meaningname * Explicitly force the name of the hypervisor.

Example: ...&name=qemu:///systemcommand ssh, ext The external command.

Example: ...&command=/opt/openssh/bin/sshsocket unix, ssh The path to the Unix domain socket (overrides the default).

Example: ...&socket=/opt/libvirt/run/libvirt/libvirt-socknetcat ssh The name of the netcat command on the remote machine (default

is nc).Example: ...&netcat=/opt/netcat/bin/nc

no_verify tls If set to a non-zero value, this disables client checks of theserver’s certificate.

Example: ...&no_verify=1no_tty ssh If set to a non-zero value, this stops ssh from asking for a pass-

word if it cannot log in to the remote machine automatically (eg.using ssh-agent etc.).

Example: ...&no_tty=1


libvirt: Domain ConfigurationA virtualized element (i.e. a domain, a storage or a network) is definedthrough a configuration XML file, containing, among other things:

The type of the hypervisor used for running the domain.The symbolic name of the domain.The type of boot-loader:

I The bios boot-loader, available in full virtualization, which uses theBIOS boot order priority (e.g., floppy, hard-disk, cdrom, network) forfinding and booting the boot image.

I The host boot-loader, available in para-virtualization, where thehost is responsible for kicking off the operating system boot.

I The direct kernel boot-loader, available in full/para-virtualization,which boots directly from a kernel stored in the host OS.

The boot device.The maximum resource usage for CPUs and memory.The devices provided to the guest domain.

I Include: disks (HD, floppy, CD-ROM), USB, NIC, input devices,graphical frame-buffers (e.g. for VNC), console/serial/paralleldevices.


libvirt: Local Domain Management

Hypervisor layerlibvirt API layer

xendApplication layer xenstored

Xen hypervisor

xendriver

QEMUdriver

qemu

xen:///

qemu:///


libvirt: Remote Domain Management

Hypervisor layerlibvirt API layer

xend

Application layerxenstored

Xen hypervisor

xendriver

QEMUdriver

qemu

libvirt API layer

remotedriver

xen:///

qemu:///

local host

remote host


libvirt: Bindings

Available binding languages:libvirt-python, the official Python binding.Sys::Virt: a Perl binding.ocaml-libvirt: the official OCaml binding.ruby-libvirt: the official Ruby binding.libvirt-java: the official Java binding (WIP).


Examples: QEMU

In order to use QEMU through libvirt it is necessary:1 To install QEMU and libvirt (and their dependencies)

I For QEMU is highly recommended to also install the KVM+QEMU(KQEMU) kernel module.

2 To run the proper system daemons.I On RedHat-like systems:$ /etc/init.d/kqemu start$ /etc/init.d/qemu start$ /etc/init.d/libvirtd start

3 To be able to gain root privileges.


Examples: QEMU capabilities I

Going to print QEMU capabilities.1 Gain root privileges.2 With virsh command:

$ virsh -r -c "qemu:///session" capabilities

3 With the Python APIs:1 Start the Python interpreter.

$ python2 Import the libvirt module.

>>> import libvirt3 Open a (read-only) connection to the QEMU hypervisor.

>>> con = libvirt.openReadOnly("qemu:///system")4 Print capabilities to standard output.

>>> print con.getCapabilities(), "\n"

4 An XML output should appear:


Examples: QEMU capabilities II<capabilities><host><cpu><arch>x86_64</arch></cpu></host>

<guest><os_type>hvm</os_type><arch name=’i686’><wordsize>32</wordsize><emulator>/usr/bin/qemu</emulator><machine>pc</machine><machine>isapc</machine><domain type=’qemu’></domain></arch><features>


Examples: QEMU capabilities III<pae/><nonpae/><acpi default=’on’ toggle=’yes’/><apic default=’on’ toggle=’no’/></features></guest>

<guest><os_type>hvm</os_type><arch name=’x86_64’><wordsize>64</wordsize><emulator>/usr/bin/qemu-system-x86_64</emulator><machine>pc</machine><machine>isapc</machine><domain type=’qemu’></domain><domain type=’kqemu’></domain>


Examples: QEMU capabilities IV

</arch><features><acpi default=’on’ toggle=’yes’/><apic default=’on’ toggle=’no’/></features></guest>...</capabilities>


Examples: QEMU domain list

Going to print the list of the installed QEMU domains.1 Gain root privileges.2 With virsh command:

$ virsh -r -c "qemu:///session" list --allI --all includes inactive domains.

3 With the Python APIs:1 Start the Python interpreter.

$ python2 Import the libvirt module.

>>> import libvirt3 Open a (read-only) connection to the QEMU hypervisor.

>>> con = libvirt.openReadOnly("qemu:///system")4 Print capabilities to standard output.

>>> print con.listDefinedDomains(), "\n"

4 A list of 〈id, name, execution status〉 triples should appear as output.


Examples: DSL on QEMU IGoing to run Damn Small Linux (DSL) [4] inside QEMU.

1 Download a DSL iso image$ GET http://.../current/current.iso > dsl.iso

2 Create a QEMU image file (100MB should suffice)$ qemu-img create -f qcow2 dsl.qcow2 100MB

3 Create the XML configuration file dsl-kqemu.xml for describing theDSL image.<?xml version="1.0"?>

<domain type=’kqemu’><name>KQEmu-DSL-i686</name><uuid>c7a5fdbd-cdaf-9455-926a-d65c16db1809</uuid><memory>65536</memory><currentMemory>32768</currentMemory><vcpu>1</vcpu><os><type arch=’i686’ machine=’pc’>hvm</type>


Examples: DSL on QEMU II

<boot dev=’cdrom’/></os><devices><emulator>/usr/bin/qemu-system-x86_64</emulator><disk type=’file’ device=’cdrom’><source file=’/path/to/vm/iso/dsl.iso’/><target dev=’hdc’/><readonly/></disk><disk type=’file’ device=’disk’><source file=’/path/to/vm/images/dsl.qcow2’/><target dev=’hda’/></disk><interface type=’network’><source network=’default’/></interface><graphics type=’vnc’ port=’-1’/>


Examples: DSL on QEMU III</devices></domain>

I For generating a UUID it is possible to use the uuidgen command.I Without libvirt, the QEMU command would be:$ qemu-system-x86_64 -M pc -m 64 -smp 1 -nameKQEmu-DSL-i686 -cdrom dsl.iso -hda dsl.qcow2-boot d

4 Run DSL on QEMUvirsh -c qemu:///system create dsl-kqemu.xml

5 Now it is possible to manage the newly created VM. For instance, open aVNC session$ virsh -c qemu:///system vncdisplay KQEmu-DSL-i686$ vncviewer 127.0.0.1:0

6 Shutdown DSL on QEMUvirsh -c qemu:///system destroy KQEmu-DSL-i686

I Note: it seems the shutdown command doesn’t work!


Examples: Xen

The syntax is very similar to the one used for QEMU:List of Xen capabilities:$ virsh -r -c "xen:///" capabilities

List of local domains:$ virsh -r -c "xen:///" list --all

. . .

What changes is the content of the XML configuration file and the URI.


Example: Remote Management with SSH

1 Create your local public key pair.$ ssh-keygen -t rsa

2 Copy the public key to a remote host.$ ssh-copy-id -i ~/.ssh/id_rsa.pub root@remote-host

3 Start the libvirt daemon.$ ssh root@remote-host$ /etc/init.d/libvirtd start

4 Issue a command to the libvirt daemon.$ virtsh -r -c xen+ssh://root@remote-host/ list --all

The first three steps are to be done only once.


Conclusions: Pros & Cons

Strong points:Abstraction: support different hypervisors.Isolation: isolates from hypervisor API changes.Portability: Linux, Windows and Mac OS-X clients.Security: TLS + x509, Kerberos, SSH, PolicyKit.Active community.

I Is the core of several Red-Hat virtualization softwares.


Conclusions: Pros & ConsWeak points:

Abstraction: might loose some hypervisor features.Lack of documentation.

I No tutorials.I APIs poor documented.

⇒ You have to learn-by-examples (see libvirt sources andRed-Hat virt-manager)For remote management, the libvirtd daemon must be running(with root privileges) on every remote host that needs to bemanaged.Moreover, the libvirtd daemon is needed by QEMU because ithas to do lots of privileged jobs, such as starting QEMU withpermission to use /dev/kvm, accessing disks and logicalvolumes in /dev, creating TAP devices, creating bridge devices,and much more.


Conclusions: libvirt Companions

The Red Hat’s Emerging Technology group includes other interestingprojects:

virt-manager: graphical desktop-based virtual machinemanagement.python-virtinst: guest installation manager.

I virt-viewer: secure guest console access.I virt-install: virtual machines provisioner.I virt-clone: virtual machine images cloner.I virt-image: virtual machine images creator (from XML files or

interactively).

oVirt: Web-based virtual machine management.All of them rely on libvirt.


Conclusions

Questions?


Conclusions

Grazie!


References I

[1] The Linux Containers (LXC).http://lxc.sourceforge.net/.

[2] The OpenVZ Linux Containers.http://wiki.openvz.org/.

[3] AMD.AMD-VTM Nested Paging.Technical report, Advanced Micro Devices (AMD), Inc., July 2008.

[4] John Andrews.Damn Small Linux (DSL).http://www.damnsmalllinux.org.

[5] Fabrice Bellard.The QEMU processor emulator.http://bellard.org/qemu/.


http://lxc.sourceforge.net/

http://wiki.openvz.org/

http://www.damnsmalllinux.org

http://bellard.org/qemu/

References II

[6] David Chisnall.The Definitive Guide to the Xen Hypervisor.Prentice Hall, 2007.

[7] Inc. Citrix Systems.The Xen hypervisor.http://www.xen.org/.

[8] Microsoft Corporation.Microsoft Hyper-V.http://www.microsoft.com/windowsserver2008/en/us/virtualization-consolidation.aspx?pf=true.

[9] Inc. Distributed Management Task Force (DMTF).Common Information Model (CIM).http://www.dmtf.org/standards/cim/.


http://www.xen.org/

http://www.microsoft.com/windowsserver2008/en/us/virtualization-consolidation.aspx?pf=true

http://www.microsoft.com/windowsserver2008/en/us/virtualization-consolidation.aspx?pf=true

http://www.dmtf.org/standards/cim/

References III

[10] Gerd Hoffmann.Xenner: Xen emulator for KVM.http://kraxel.fedorapeople.org/xenner/.

[11] Herbert Pötzl.Linux v-server.http://linux-vserver.org/Welcome_to_Linux-VServer.org.

[12] Inc. Qumranet.The Kernel based Virtual Machine (KVM).http://kvm.qumranet.com/.

[13] Inc. Red Hat.The Red Hat’s Emerging Technology group.http://www.dmtf.org/standards/cim/.


http://kraxel.fedorapeople.org/xenner/

http://linux-vserver.org/Welcome_to_Linux-VServer.org

http://kvm.qumranet.com/

http://www.dmtf.org/standards/cim/

References IV

[14] Inc. Sun Microsystems.Libvirt for LDoms 1.0.1 administration guide.http://docs.sun.com/app/docs/doc/820-3838-10.

[15] Inc. Sun Microsystems.Logical domains (LDoms).http://www.sun.com/servers/coolthreads/ldoms/.

[16] Inc. VMware.VMware CIM APIs.http://www.vmware.com/support/developer/cim-sdk/.


http://docs.sun.com/app/docs/doc/820-3838-10

http://www.sun.com/servers/coolthreads/ldoms/

http://www.vmware.com/support/developer/cim-sdk/

Extras: Nested Paging

Introduced by AMD [3].Virtual Machines (VMs) don’t have native direct access to the hostserver memoryAs a result, a hypervisor ends up virtualizing a “read only” layer ofmemory between physical memory and the page tables in theguest OS, which is known as shadow paging.Shadow pages requires CPU and memory, adding extraperformance overhead.With Nested Paging, a page table in the hardware takes care ofthe translation between the guest address of a VM and thephysical address, reducing the overhead.


Extras: CIM vs. libvirtCIM and libvirt both provide a hypervisor-agnostic abstractionlayer for writing tools. Nevertheless, there are a few majordifferences:

I CIM is an abstract model, with representations built on XML, andusable in a language-agnostic way.libvirt is quite closely tied to C.

I CIM is a standard defined by a working group representing multiplevendors.libvirt began life as a wrapper around xm and has grown to a moregeneral interface. It is still quite closely tied to the Xen way of doingthings, however.

I CIM is a large specification, and the virtualization parts are only asmall fraction of the whole.libvirt is designed exclusively for managing virtualization.

I CIM lets to easily and flexibly add support for managingvirtualization to a CIM-aware management tool.libvirt is extensible “only” through the bindings it provides (thoughthe set of supported languages is pretty large).


libvirt: a virtualization api

Technology

binding api

xen hypervisor

virtualization support

virtualization apiseptember

hybrid virtualization

virtualization capabilities

virtualization features

core api