october 2012 implementing & troubleshooting pvs gareth obrien

October 2012

Implementing & Troubleshooting PVS

Gareth O’Brien

© 2012 Citrix | Confidential – Do Not Distribute

Introduction

2

ᵒ Gareth O’Brien, escalation engineer

ᵒ Intermediate to advanced PVS knowledge expected


Agenda

3

• Planningᵒ Database, storage, network, farm layout

• Installing

• Configuringᵒ Files needed for booting, boot process

• Troubleshootingᵒ Stream Serviceᵒ Console, MAPI and IPC

Implementing PVS


Planning

5

• Planning the database configuration

• Hardware requirementsᵒ Client & Server considerations

• Network

• Farm layout considerations


Database

6

• Size of farm determines database server choice

• Fault tolerance and failoverᵒ Mirroring

• dbOffline – if all else fails

• SQL native client needed for mirroring

• Permissionsᵒ Dbcreator & securityadmin for Configuration Wizardᵒ Db_datareader, db_datawriter & execute permissions on stored procedures


Hardware Requirements

7

• Supported Operating systemsᵒ Windows Server 2003 R2, 2008, 2008 R2

• Minimum spec of the PVS server

• Known hardware issuesᵒ http://support.citrix.com/article/CTX131611

• NIC Teamingᵒ Make sure hardware is not on above listᵒ If streaming doesn’t work with team, try break the team

Processor: 3GHzRAM: 4GB

NIC: 1GB per 250 targets

Processor: 3GHzRAM: 4GB

NIC: 1GB per 250 targets


Network

8

• Configure UDP & TCP Portsᵒ Server to server ports must be the same on all serversᵒ Target device communication ports

• 6910-6930ᵒ SOAP Server

• 54321 & 54322ᵒ TFTP & TSB

• 69 & 6969

• Firewall must be configured to allow all these portsᵒ If in doubt, allow ANY/ANY to test

• PortFast must be enabled


Plan the farm

9

Provisioning Services Farm

Site 1

DeviceCollection

DeviceCollection

DeviceCollection

Provisioning Servers Provisioning Servers

DeviceCollection

DeviceCollection

Site 2

License Server

MSSQL Server

Database

Shared Storage

Installing PVS


Installing PVS

11

• Server installationᵒ If installing a new version of .NET, reboot when promptedᵒ Specify the mirror failover if applicable

• Console installationᵒ Needs PowerShell 2.0ᵒ If upgrading, make sure you remove the old version

• Target installationᵒ Make sure you set the target to boot from network firstᵒ If imaging fails

• Start with a vanilla install & try to image. Remove any teams• XenConvert & BNImage are other options

Configuring PVS


Configuring PVS

13

• PXE Boot

• Two Stage Boot (TSB)

• Boot Process

• Stores

• IPC Key


PXE Boot

14

• Must configure DHCPᵒ Option 66: IP of PVS Serverᵒ Option 67: Name of bootstrap (ardbp32.bin)

• Uses TFTP Protocol

• TFTP listens on one NIC onlyᵒ Configure using c:\program Files\citrix\provisioning services\tftpcpl.cpl

• TFTP Downloads bootstrap, ARDBP32.BIN

• Other TFTP servers can be used


Two Stage Boot (TSB)

15

• Used in situations where DHCP or PXE cannot be used

• Uses ISO burned from PVS Server

• Uses a boot strap called tsbbdm.bin

• Listens on NIC configured for streaming in configuration wizardᵒ Uses port 6969

• Uses Two Stage Boot service on PVS server


Boot Process

16

• Target device either uses DHCP to obtain an IP or loads static IP (for TSB)

• Once it has an IP, it will contact the 1st logon server specified in the bootstrap

• The logon server will then create a context for the device, determine which vDisk the target is assigned and the least loaded serverᵒ This calculation is based on both RIMS information and the load information in the

databaseᵒ RIMS uses notifier and inventory service to update load informationᵒ If subnet affinity is enabled, this will alter the calculation

• Once a server has been selected, a streaming IP and port will be providedᵒ IP information will be in HEX, not decimal


Boot Process (cont)

17

• Excerpt from log of a target logging on


Boot Process (cont)

18

• As streaming begins, the boot control device (BCD) is downloadedᵒ This phase is referred to as single I/O

• Once Single I/O has finished, we enter the admin phaseᵒ BNIStack handshakeᵒ Vdisk configuration (name, mode, write cache type) is requested and providedᵒ Target device requests personality string

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BNIStack\Parameters - DWORD EnableGetComputerName set to 1

ᵒ AD information requested

• Multi I/O phase entered


Stores

19

• Configure vDisk storesᵒ Centralised vs Distributed stores

• Centralisedᵒ Works well in HA environment ᵒ Low storage overhead ᵒ High cost ᵒ Potential SAN/Storage issues

• Distributedᵒ Low implementation cost ᵒ High storage overhead ᵒ Configuration

• IPC is the method for communication between servers for stores


IPC Key

20

• Defines which NIC to use for IPC communication in a multi NIC environmentᵒ HKEY_LOCAL_MACHINE\Software\Citrix\ProvisioningServices\IPC\

• Reg_sz called IPv4Address with the IP of the NIC for IPC

• Without it, stores, replication, load balancing etc won’t work

• Affects stream service

• Manager key for MAPI works the same wayᵒ HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ProvisioningServices\Manager

• RegSZ called GeneralInetAddr with the IP of the NIC and port - Eg 10.1.1.2:6909

Troubleshooting PVS

A more structured method of troubleshooting PVS


Troubleshooting PVS

23

• Stream service troubleshootingᵒ The basicsᵒ Tracingᵒ Dumpsᵒ Log Analysisᵒ Common Issues

• SOAP/Consoleᵒ Unable to validateᵒ Console timeouts


Stream Service Basics

24

• The devil is in the detail.ᵒ Network Topologyᵒ Hardware of target, server & databaseᵒ Storage type & locationᵒ Write cache typeᵒ Database location, version, HAᵒ MSInfo32

• Bluescreensᵒ Common Imageᵒ 7B

• What does it mean• Common causes


Stream Service Tracing

25

• Network tracesᵒ Wireshark trace on the serverᵒ Tcpdump on the target

• Make sure you always use –s 0 to not limit capture size

• PVS Logsᵒ Set to trace levelᵒ C:\programdata\Citrix\Provisioning Services\Logsᵒ Target side logging also

• All logs and traces should be capture simultaneously, while the issue is happening.


Stream Service Tracing (cont)

26



27



28


Crash Dumps

29

• If possible, take it in private mode

• In 2008, it’s possible to redirect to a non system drive ᵒ http://support.citrix.com/article/CTX123642ᵒ Not possible on 2003

• Configure citrix symbols

• Live debugging can be done, but difficult without symbolsᵒ Use live debugger to help set it up http://support.citrix.com/article/CTX131945ᵒ Check for BNIstack being loaded

• !devnode 0 1 ᵒ On 5.6 check for bnns

• !ndiskd.miniports


Common issues

30

• Imagingᵒ Let the wizard do the work for you

• Always revert to a clean install as a first step• If the wizard doesn’t work, pre-create the disk and use xenconvert• BNImage can work well if all else fails

• Computer Accountᵒ Is AD Management configured on the vDisk? Is Active directory correctly configured?

• GPO disabling password update, and password age• Reset the password within the PVS console• Check all relevant Microsoft hotfixes are installed – Google!• BNDevice 1015 in the event viewer – unable to negotiate a new password• Trace logs will show you the requests


Performance

31

• Follow the best practice guideᵒ http://support.citrix.com/article/CTX127549ᵒ http://support.citrix.com/article/CTX117374

• Intermediate bufferingᵒ Generally enabled if free space on target device disk is greater than vDisk, otherwise

disabledᵒ Toggling the state is always worth trying to see the performance impact

• Check interrupt safe mode is not enabled

• Check the network with a network monitor


SOAP & Console

32

• Console timeoutᵒ The console timeout can be set in the registry

• Use DWORD ConnectionTimeout in HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ProvisioningServices. Default value is 120s

• Can’t communicate with other serversᵒ Check IPC and manager keys

• HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ProvisioningServices\IPC\IPv4Address• HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ProvisioningServices\Manager\GeneralInetAddr• Mentioned above

• Load Balancing issuesᵒ Check the StreamProcess & Inventory logs to make sure there are no issues around

RIMS

Questions?

Work better. Live better.

october 2012 implementing & troubleshooting pvs gareth obrien

Documents

citrix confidential

pvs server slide

pvs slide

ipc slide

stores slide

server ports

options slide

decimal slide