citrix xenserver 5.5 troubleshooting
DESCRIPTION
Citrix XenServer 5.5 Troubleshooting Deep Dive.TRANSCRIPT
Citrix® XenServer
Webcast 26. März 2010
Thomas KrampeCitrix Technology Professional
Advanced Troubleshooting
204/19/10 (c) 2009 Thomas Krampe
Agenda
Architecture
Troubleshooting
Birth and Death of a VM
Object modell & CLI
Unattended Setup & Configuration
Question & Answers
304/19/10 (c) 2009 Thomas Krampe
Architecture
Architecture
XenServer Architecture
Xen Domain0
Guest OS Internals
XAPI
Storage Manager
CLI
GUI (XenCenter)
404/19/10 (c) 2009 Thomas Krampe
XenServer Architecture
Virtualized HardwareVirtualized Hardware
Hardware
VT/AMD-V
XAPIXAPIDrivers
Storage
CLI
SMSM
NFS iSCSI FC
504/19/10 (c) 2009 Thomas Krampe
Xen Domain0
Dom0 kernel and initrd passed on start up
Dom0 has direct access to HW
Bootloader
/boot/grub/grub.conf
/boot/extlinux.conf
[Ctrl-A] 3 times will trap into the Xen debug console
604/19/10 (c) 2009 Thomas Krampe
Guest OS internals
PV Guest
Modified Kernel (“Xenified”)
Kernel name will have “xs” somewhere in the filename
Guest agent reports VM stats
Use pygrub bootloader
HVM Guest (Windows)
Unmodified Kernel
Need PV drivers
Event Channel, Network and Block
Guest agent uses WMI
704/19/10 (c) 2009 Thomas Krampe
XAPI /opt/xensource/bin/xapi
Service, autostarted on boot
Implements the Xen API spec
Talks to Storage Manager (sm) for SR control operations
Stores configuration in sqlite DB
/var/xapi/state.db
sqlite3 /var/xapi/state.db
DB is replicated on all hosts in a pool
Sets up SR defaults on firstboot
Failing firstboot scripts might cause issues
/var/xapi/generate-* and /var/xapi/firstboot-*
Defaults from /etc/xensource-inventory
804/19/10 (c) 2009 Thomas Krampe
Storage Manager
Manages SR control operations to various backends
/opt/xensource/sm/sm.py
Plugin model
One plugin for each SR type
/opt/xensource/sm/<Type>SR.py
Error codes listed in
/opt/xensource/sm/XE_SR_ERRORCODES.xml
xe sm-list lists all supported drivers
904/19/10 (c) 2009 Thomas Krampe
CLI
Local or Remote
Talks to XAPI
Bash auto completion
xe.exe (for Windows)
xe help or xe help --all
1004/19/10 (c) 2009 Thomas Krampe
GUI
Uses Microsoft .NET framework
Uses https/xml-rpc to communicate with XAPI
VNC Client for guest frame buffer
VNC traffic is tunneled over https
RDP client for Windows guests
1104/19/10 (c) 2009 Thomas Krampe
Troubleshooting
TroubleshootingServer Install
Useful Linux Commands
Server Diagnostics & Inventory
Saving System State
XS Server Log Files
XenSource Agent
XenCenter Logging
Summery
1204/19/10 (c) 2009 Thomas Krampe
Server Install After Product CD Boots
Alt-F2 displays Linux login prompt
Login: root (no password required)
Alt-F3 displays install event log
Alt-F1 displays original install menu
Support Tarballsupport.sh (creates /tmp/support.tar.bz2)
Support tarball is created in /tmp on install error
Support tarball is saved in /root after successful install
1304/19/10 (c) 2009 Thomas Krampe
Useful Linux Commands
Storage
List Disks detected
fdisk –l
Network
Show NIC configuration
ifconfig eth0 [eth1 ..]
ifup eth0
TCP/IP Tools ping, route -n
dhclient eth0
1404/19/10 (c) 2009 Thomas Krampe
Useful Linux Commands
Process Management
ps –Af
top
xentop
1504/19/10 (c) 2009 Thomas Krampe
Server Diagnostics and Inventory
Server Diagnostics
xen-bugtool
Run on the XS Console
Records all log files
Writes compressed file for submission to L2 support
/var/opt/xen/bug-report/bug-report-xxx.tar.bz2
1604/19/10 (c) 2009 Thomas Krampe
Server Diagnostics and Inventory
Server Diagnostics
Automatic Upload for Support
xe host-bugreport-upload
Can be run from admin system
Runs xen-bugtool and uploads to XS Support
Note unique ID of report for support
1704/19/10 (c) 2009 Thomas Krampe
Server Diagnostics and Inventory
Server Inventory
system-info.sh | less
Interactive script that captures all non-log state
Hardware devices & Device drivers
Storage repository and VM configurations
1804/19/10 (c) 2009 Thomas Krampe
Saving System State
xen-bugtool archives state and configuration files
Network config
Install options
Install logs
Storage Repository metadata
VM configs
Xenstore
1904/19/10 (c) 2009 Thomas Krampe
XS Server Log Files
XenSource XAPI Agent
/var/log/messages
Contains other system logging
/var/log/xensource.log
Xen/Domain 0 Xen Boot Messages
xe host-dmesg
Domain 0 Boot Messages
dmesg
2004/19/10 (c) 2009 Thomas Krampe
Debug Logging for XAPI Agent
Enable debug logging for all subsystems
xe log-set-output level=debug output=file:/tmp/debug-all
Debug for only a certain subsystem
xe log-set-output level=debug key=storage output=file:/tmp/storage-
debug
List Available Subsystems
xe log-get-keys
Restore normal logging xe log-set-output level=debug output=nil
xe log-set-output level=debug key=storage output=nil
2104/19/10 (c) 2009 Thomas Krampe
XenSource Agent
XS Agent runs on top Xen Hypervisor API and XenStore
Can be restarted without harming VMs or other members
of the pool
service xapi restart
If the UI seems to get stuck or goes into a weird state, restart
the agent and reconnect the UI
Another simular command is xe-toolstack-restart
2204/19/10 (c) 2009 Thomas Krampe
XenCenter Logging
Location
{user home}\Application Data\XenSource\XenCenter\logs\
XenCenter.log
Rolling
Log files will get rolled
XenCenter.log.#
Increase Log Level C:\Program Files\XenSource\XenCenter directory
<level value=“DEBUG” />
2304/19/10 (c) 2009 Thomas Krampe
Summary
xen-bugtoolgenerates XE Server diagnostic file for submission to Technical Support Representative.
xe host-bugreport-upload
runs xen-bugtool and uploads the result to XenSource support.
xe host-bugtool
can be used to save Domain 0 state, to assist during emergency recovery
Use Linux commands (network, storage, modules) to troubleshoot server install.
2404/19/10 (c) 2009 Thomas Krampe
Birth and Death of a VM
Birth and Death of a VM
VM Installation Internals
Behind the Scenes
Debugging VMs
2504/19/10 (c) 2009 Thomas Krampe
VM Installation Internals
Debian Sarge/Etch
Root file system is extracted into the VDI
Debian Xen kernel
RHEL (3.x,4.x,5) / SLES (9.x, 10.x)
Uses regular repositories
RHEL / SLES Xen kernel
Windows (all) Installs from CD/ISO bits
Unmodified Windows kernel
2604/19/10 (c) 2009 Thomas Krampe
Behind the Scenes
Paravirtualized VMs use pygrub
Simple python based bootloader for Xen
Finds kernel and initrd from VM’s boot device
CPU then jumps to this kernel to start this VM
HVM VMs use hvmloader
hvmloader creates a virtual BIOS and jumps to it. This is like a
BIOS in a physical machine
qemu-dm does device emulation, provides disks, NICs, serial
ports, USB devices etc to this VM
Follows standard boot process to boot from disk
2704/19/10 (c) 2009 Thomas Krampe
Debugging Windows VMs
Have client Windows machine ready, install WinDbg
Add /DEBUGPORT=com1 to boot.ini in VM, shut down VM
xe vm-param-add uuid=<VM UUID> param-name=other-config:hvm_serial=tcp:<client IP>:7001
sockpipe my_pipe 7001 on client machine
windbg -k com:pipe,port=\\.\pipe\my_pipe,resets=0
Start VM
Refer to WinDbg manual for commands
http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx
2804/19/10 (c) 2009 Thomas Krampe
Object Model and CLI
Object Model and CLI
Object Model
CLI Command Format
Maps
Constarining Information
Interesting Commands
Looking inside a log file
2904/19/10 (c) 2009 Thomas Krampe
Object Model
3004/19/10 (c) 2009 Thomas Krampe
CLI Command Format
Object commands
xe <class>-<TAB>
Object Param commands xe <class>-param-<TAB>
Some fields are marked <expensive
Data Types Single Valued/Multi-Valued
Read Only/Read-Write
Set/Map
3104/19/10 (c) 2009 Thomas Krampe
Maps
Setting Map values[root@localhost] xe vm-param-set uuid=VM uuid other-config:foo=baa
Getting Map values
[root@localhost] xe vm-param-get uuid=VM-uuid param-name=other-config mac_seed: 6fjafhahffhadfbadfad
[root@localhost] xe vm-param-get uuid=VM-uuid param-name=other-config param-key=mac_seed 6fjafhahffhadfbadfad
3204/19/10 (c) 2009 Thomas Krampe
Constraining Information
Filtering
vm-list HVM-boot-policy="BIOS order" power-state=halted
Parameterization xe vm-list params=name-label
Minimal Output xe vm-list --minimal
Parameterization and filtering can be combined
3304/19/10 (c) 2009 Thomas Krampe
Interesting Commands
Wait for a VM to run
xe event-wait class=vm name-label=myvm power-state=running
List of CPUs on a host xe host-cpu-list
CPU Flags must match across all hosts in pool
List of subsystems xe log-get-keys
Can be used to isolate subsystem logs
3404/19/10 (c) 2009 Thomas Krampe
Looking inside a log file
Each Entry looks like
[timestamp][hostname][level] [ref-1] [(num-1) unix_rpc] text-1 text2 text3….
[timestamp] Timestamp of log entry
[hostname] Host name (in a pool)
[level] Debug level
[ref-1] ID of the task on whose behalf this is created
num-1 Thread ID, local to the current process
unix_rpc Unix Domain Socket (as opposed to inet_rpc=TCP)
text-1 Subsystem
text2/text3 more information on action performed
3504/19/10 (c) 2009 Thomas Krampe
Looking inside a log file
xensource.log[20100209 16:58:28.688|debug|localhost|3078703|SR scanner D:dfd9ef0b708d|helpers] login done[20100209 16:58:28.921|debug|localhost|3078706 unix-RPC|SR.scan R:1ac783270499|taskhelper] task destroyed[root@localhost log]#
Xenstored-access.log[20100209 16:49:05.381] D35 write data/meminfo_free 551040[20100209 16:49:05.381] D35 write data/meminfo_total 1035760[20100209 16:49:05.381] D35 write data/updated 1
messagesFeb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078916|SR scanner D:dfd9ef0b708d|dispatcher] spawning a new thread to handle the current taskFeb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078917 unix-RPC|session.slave_login D:58b671575b1e|xapi] Session.create trackid=ea252f3e59cc6f0d2d3de1345967d54b pool=true uname= is_local_superuser=true auth_user_sid=Feb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078919 unix-RPC|dispatch:SR.scan D:64e76e160ebc|taskhelper] task SR.scan R:98637b78c34f (uuid:6e8469dc-724d-0260-a584-44d36bca2953) created (trackid=ea252f3e59cc6f0d2d3de1345967d54b) by task D:dfd9ef0b708d
3604/19/10 (c) 2009 Thomas Krampe
Unattend Installation
Possible over TFTP - PXE
use a single xml configuration file
http://wiki.xenmaster.de/doku.php/xenserver/xspxeunattend
3704/19/10 (c) 2009 Thomas Krampe
Questions & Answers