1

XenDesktop Connection Leasing

Design Considerations for XenDesktop 7.6 Connection Leasing White Paper

Joe Deller

October 2014

2

Contents XenDesktop Connection Leasing – Controller scalability considerations............................................................ 3

Executive Summary ......................................................................................................................................... 3

Overview of Connection Leasing ................................................................................................................. 4

Operation..................................................................................................................................................... 4

Controller Requirements ............................................................................................................................. 5

Multiple Controllers – partial connectivity loss ........................................................................................... 5

Lease Layout on the Controller ....................................................................................................................... 5

Calculating Lease files numbers .................................................................................................................. 6

Size of leases ................................................................................................................................................ 7

Example Environments .................................................................................................................................... 8

For RDS desktops and applications ............................................................................................................. 8

For an assigned VDI desktop environment with 40,000 users, single client per user ................................. 8

Sparse Files .................................................................................................................................................. 8

Folder structure ........................................................................................................................................... 9

Lease Synchronisation ................................................................................................................................. 9

Disk systems with compression enabled ................................................................................................... 12

Forcing Lease Refresh ................................................................................................................................ 12

Disk activity when Leasing is active ............................................................................................................... 12

Cached lease file updates .............................................................................................................................. 12

Lease timeouts .......................................................................................................................................... 12

Controller CPU load ....................................................................................................................................... 13

CPU Load on Controllers when leasing is active ........................................................................................ 13

Connection Launch rates with Connection Leasing active ............................................................................ 14

Configuring Connection Leasing Parameters ................................................................................................ 15

3

XenDesktop Connection Leasing – Controller scalability considerations

Executive Summary A site database hosted on a highly available SQL server configuration is still the best and recommended

approach to ensuring users have reliable access to their resources. In situations where this is not possible,

or the where the whole SQL cluster has failed, Connection Leasing will enable users to connect to recently*

accessed resources, provided the VDAs for those connections are still accessible.

During the initial connection lease creation phase, there will be a significant number of small files written to

each controller, dependant on user count, application count and deployment type. The default

configuration is such that systems should not be impacted in terms of IOPS, CPU or session launch rates,

however as each controller in the site will need to have the lease files present on its storage, systems with

shared persistent storage for the controllers may require reassessment.

Updates to user resources, such as adding, editing or retiring resources, will also cause updates to the lease

files, with a default rate of 1000 leases being updated every 10 seconds.

Although the size of the lease files is small, for sites with large numbers of users, the sheer amount of files

each consuming the default block means that additional disk space will be required compared to previous

releases. An additional 2GB of space is a recommended starting point, but will vary from environment to

environment.

For controllers using physical hosts, a battery backed caching controller can smooth the impact in

environments where relatively large amounts of users are logging on in relatively short time windows. For

virtual hosts and those using shared storage, the amount of IOPS available to the controllers and available

disk should be examined to ensure availability of both.

In normal site operations, with the database available, Connection Leasing causes minimal impact on overall

system operations. For environments where there is a likelihood of site database failure, extra DDCs may be

required due to load balancing not being available when Connection Leasing is enabled and the VDA

reregistration loads occurring during loss of site database.

When connection leasing is active, VDA registration storms can cause significant CPU usage depending on

the total number of VDAs in the environment. Intermittent database problems can exacerbate this problem

as VDAs register and deregister as connectivity to the database is restored and lost again. In such a scenario,

it is worth considering removing database connectivity completely until the database issues are fully

resolved.

*By default users must have connected to resources within the last 14 days; this period can be configured an

administrator. A full list of configurable parameters can be found at the end of this document.

4

Overview of Connection Leasing

The connection leasing feature in XenDesktop 7.6 is designed to allow some resiliency in situations where the site

SQL database has failed due connectivity or other issues. It will allow users to continue to connect to resources that

they have previously connected to within the last two weeks (administrator configurable). It achieves this by

creating lease files for the applications and desktops and caching them on each Controller in the site. Each of these

XML files contains information about the user, available resources to the user, and the resources they have

previously connected to.

It should be noted that Connection Leasing is not intended to be a replacement for a SQL Server High Availability

configuration.

Connection leasing is only applicable to XenApp/RDS and XenDesktop-based assigned desktops – it is not applicable

to XenDesktop-based pooled desktops.

For Remote PC deployments, these are considered assigned desktops and are therefore covered by Connection

Leasing.

Connection Leasing has certain limitations when active, when the site database is inaccessible or otherwise in a

failed state:

Desktop Studio and Desktop Director operations are unavailable.

Citrix PowerShell cmdlets requiring database access will not work.

No VDA load balancing will occur.

Users can only connect to the last host they connected to when the site database was available.

There is a small window (2 minutes) during which no sessions will be brokered when the site database

becomes unavailable or is restored. This is to allow for environments with SQL HA enabled to fail over, such

that leasing does not become enabled when there is only a short window where site database connectivity is

interrupted.

Users must have logged on to the resources within the default 14 day period. This can be configured via a

registry setting, a full list of configuration settings are listed at the end of this document.

Anonymous users are not supported by Connection Leasing.

If the site database and resource hosts are not available, then users will not be redirected to alternative hosts if the

last host they connected to is unavailable. In this respect, the lease files are a snapshot of the last successful result

of a user connecting to resources.

Operation

By default, in XenDesktop 7.6 Connection Leasing is enabled.

When a user connects to a resource for the first time, the controller collects information about the connection and

sends this information to the site SQL database. At regular 10 second intervals, the information is then synchronised

down to each controller. There is therefore a small window in which a user could connect to the controller and then

the controller not being able to pull down the lease file if the database fails before the next lease synchronisation.

Each controller synchronises with the SQL database, there is no direct controller to controller communication with

regards to connection leasing.

5

Controller Requirements

The number and size of the lease files will depend on a number of factors and each controller will need some

additional space for lease file storage. The creation of the files is done in timed batches to distribute the required

disk I/O for their creation. By default each controller will synchronise at a rate of up to 1000 leases every 10 seconds

until all leases have been synchronised. This can be configured via a registry setting, listed at the end of this

document. An additional 2 GB of disk space on each controller is a good starting point for lease storage.

In previous versions of XenDesktop, a site database failure would impact all users trying to access resources and no

brokering of new sessions would occur until full connectivity was restored. With Connection Leasing, when a

database failure is detected, there is a (configurable) two minute period where no connections are brokered, then

the locally cached leases will be used to broker sessions. During this time, VDAs will start to deregister with the

controller. When database connectivity is restored, there will again be a two minute window before sessions are

brokered. As with previous versions of XenDesktop, VDAs will then start to re-register with the controllers and this

process will consume some CPU, depending on how many VDAs are attempting to re-register. For large scale VDI

deployments with many thousands of desktops, the time taken for all VDAs to re-register will depend on how many

re-registration requests each controller can handle.

Multiple Controllers – partial connectivity loss

In a multiple controller environment, if the database failure is not site wide, such that one or more controllers lose

connectivity, but one or more controllers remain connected, VDAs will attempt to register with the remaining

database connected controllers. This increase in registration demand will require CPU for processing and partial

failure scenarios should be examined in terms of how many VDA registration requests a single controller will be

required to meet if all other controllers lose database connectivity. Connection Leasing will only activate if all

controllers have lost database connectivity, normal site operation will remain whilst at least one controller can

access the database.

Lease Layout on the Controller Each lease file is an XML file that is usually relatively small, typically less than 1KB in size. However, as each user will

have an associated, application, assigned VDI desktop / RDS worker lease files, the number of files created on each

controller associated with this feature can be significant and must be planned for accordingly.

By default, leases files are stored in subdirectories in %programdata%\Citrix\Broker\Cache:

Apps

Desktops

Icons

Leases\Enumeration

Leases\Launch

Workers

The location of the leases can be changed by modifying a registry key, details of which are found at the end of this

document.

The Apps directory contains information about published applications, one file per published app per delivery group;

as such this subdirectory should remain relatively small in terms of size and number of objects.

6

The Desktops directory contains an entry per user VDA, in a VDI environment this will be one for every user assigned

VDI desktop, or in a RDS worker, one entry per published desktop. A VDI environment will therefore normally

require much more disk space than a RDS environment as there is a one to one mapping between users and their

assigned desktop, rather than the many users to one RDS host desktop.

The Icons directory will have one entry per unique published application and one for desktops. Normally desktops

share one standard icon, unless otherwise configured. If a published application shares the same base executable as

another, only one icon entry will be created. Icons will tend to be larger than lease files as they contain the raw

bitmap information on how to draw the icon for the application or desktop, but this should still only be in the

hundreds of K for a typical icon.

The Leases\Enumeration directory contains an entry about the resources available to each user, one per user. The

size of the file will depend on the number of resources (applications and desktops) available to the user.

The Leases\Launch directory contains an entry for each successful user VDA login, one for each desktop that the user

is entitled to (and has launched) and one for applications. Only a single application lease file is created no matter

how many are available to the user as session sharing will normally direct the user to the same host in normal

operations. The user can launch any app published from a delivery group from which they have previously launched

an app – even if it’s not the same one. It is possible that the enumeration lease might include details of

apps/desktops that are no longer available to the user, for example in a scenario where the controller or desktop

that hosts the resources is unavailable, there is no load balancing active during Connection Leasing, only the

previously connected host will be used.

The Workers directory contains one entry per VDA, so as per the Desktops directory, a VDI environment will

generally contain more lease files than a RDS one; each assigned desktop has data associated with the user, rather

than the many users accessing the same RDS host.

Note that published, rather than local, applications launched inside a desktop session will create their own lease

files. In some site database failure scenarios, for example a partial network failure, a user might still be able to

connect to a desktop whose host still has connectivity, but not the published applications available to that desktop

that are hosted on a separate server affected by a network outage.

Calculating Lease files numbers

The number of lease files created can be calculated as follows:

Apps + unique icons +

Users + (Users * Desktops) + (Users * Application delivery groups) + VDAs (for RDS, 1 per hosted desktop. For VDI,

one for each user desktop)

In this instance, “users” refers to the number of individual users who use the XenDesktop site.

If a resource is accessible from a LAN and a WAN via Access Gateway then the calculation is:

(Users *2) + (Users * Desktops) + (Users * Application delivery groups) + VDAs (for RDS, 1 per hosted desktop. For

VDI, one for each user desktop)

7

Size of leases

The size of the lease files will vary between environments, but the following figures may be used as a guide.

Average Application entry = 0.5K

Average Desktop entry ~ 1K

Average Icon (>1K<512K) ~256K

Enumeration Lease ~ 0.5K per available desktop + ~ 0.5K per available application

Launch Lease ~ 0.5K per desktop, 0.5K if the users have applications published.

For a normal NTFS file system with a default block size of 4K, each lease will consume a 4K block, icons being larger

will be rounded up to the next 4K block. As the number of lease files grows, there will be a disparity between the

“size” and “size on disk” properties of the cache directory, due to the leases consuming 4K even though they are

much smaller.

8

Example Environments

For RDS desktops and applications

For an environment with 1 desktop, 6 applications and 18 RDS hosts, serving 100,000 users, single client per user.

For this example, each user connected to a desktop and 10,000 users launched a single application.

The size of the icons folder will depend on the amount of unique icons for the applications published and their size.

Publishing 100 notepads for example will only need one icon lease file, but 100 unique application executables will

require 100 separate icon files.

When a user connects to a published application, normally each subsequent application launch will use the same

host and session share, so one lease will be created for applications.

Folder Files Folders Size Size on Disk (4K block size)

Apps 10 10 2K 40K

Desktops 1 1 1K 4K

Icons 6 6 792K 816K

Leases Enumeration 100,000 4096 110.4M 390M

Leases\Launch 110,000 4096 110M 0M* /429M

Workers 18 18 19K 72K

110,031 16386 115.6M 820M

* Lease launch files are sparse files, see below.

For an assigned VDI desktop environment with 40,000 users, single client per user

Folder Files Folders Size Size on Disk (4K block size)

Apps 0 0 0 0

Desktops 40,000 4096 28.9M 156M

Icons 1 1 175K 176K

Leases Enumeration 40,000 4096 26.7M 156M

Leases\Launch 40,000 4096 18.4M 0M/156.25M*

Workers 40,000 4096 42.4M 156.25M

160,001 16385 116M 312.6M /468M

* Lease launch files are sparse files, see below.

Sparse Files

The apparent lack of space consumed by the VDI Leases\Launch directory is due to the very small size of the lease

xml files created within. These files are less than 512 bytes in size, enabling NTFS to utilise a feature called sparse

files and store their data inside the MFT (Master File Table) itself. You should however plan for them as if they were

consuming a 4K block.

9

Folder structure

Due to the potential large number of files being created, Connection Leasing limits the number of files that it will

create in a single directory, creating new subdirectories as required.

Lease Synchronisation

It is important to remember that lease creation and synchronisation occurs:

When users access their resources

When application details, icons or desktops are changed or updated

When leases expire (default two weeks since last access)

When leasing is manually disabled and re-enabled

Long lived sessions will also receive updates

For a new environment, when a user logs on to a resource for the first time, the controller sends the information to

the site database, then syncs down the lease files at a rate of 1000 leases every 10 seconds. During the initial

synchronisation of leases there will be considerable I/O on the controller and SQL server, depending on the rate at

which users are connecting to resources. Consider an environment supporting 40,000 users with a single VDI

desktop:

Each assigned VDI desktop has an entry in the Desktops subdirectory

On logon, the Leases\Enumeration subdirectory will have one entry per user to determine which resources

the user can access.

An entry in the Workers subdirectory is created for the user desktop.

When the user launches their desktop, one entry per user is created in the Leases\Launch directory.

There will be a total of 160,000 xml files created once all users have completed their login. If we take the

pathological extreme, where the environment was created and 40,000 users were able to launch almost

instantaneously, for the default 1000 leases per 10s we would need:

160,000 /1000 = 160 synchronisation operations, spaced 10 seconds apart

1600 seconds = 26 2/3 minutes before all the leases were fully synchronised from the database.

Depending on your disk subsystem, the 1000 lease files may take a number of seconds for the disk to commit them,

due to IOPS restrictions.

1000 leases per 10 seconds averages to 100 per second, although systems will not necessarily handle the load

linearly, Perfmon counters should be used to observe the disk system to see how the load is distributed. The data

itself is relatively small in our 40,000 single desktop VDI - 116MB - the large number of files will be taxing the IOPS of

the disk.

10

If consider the average IOPS of a single traditional spinning disk with no caching controller, the IOPS available will be

as follows:

15k rpm: 180-210 IOPS

10k rpm: 130-150 IOPS

7200 rpm: 80-100 IOPS

5400 rpm: 50-80 IOPS

A low end blade server with single SATA 7200 rpm disk and no caching controller is going to struggle somewhat in

this extreme example, as this load is in addition to any IOPS required by normal Windows operations. The leases

will be synchronised, but there will be a backlog in the disk queue. In the unlikely event that this level of storage was

being used for a relatively large number of users, the synchronisation rate may have to be lowered, which will mean

a longer time for all of the leases to be cached on the controller.

In enterprise systems with battery backed caching controllers, even a modest amount of cache (128MB) would soak

the IOPS over the 26 minute period and a more aggressive number of leases per sync or shorter sync period could be

set.

For environments employing shared storage, whether using physical or VMs for the controllers, the shared storage

will experience bursts dependent on the number of controllers on the shared storage, but once again, battery

backed cache and tiered storage will alleviate this load.

In reality, the desktops and enumeration files would not be being created at the same time as the launch leases and

user logon times would have a wider spread.

For the following test, 40,000 users were launched at a rate of 60 logons every 1.08 seconds, with the last logon

completing at 11:32. The controllers used in this environment were physical machine with a single local SAS disk and

no RAID card.

11

During this test run, the controller is pulling down the leases at the default rate of 1000 every 10 seconds and the

leases are keeping the disk system busy until all 160,000 are synchronised from the database. As expected, the disk

queue and IOPS are varying depending as users are logged on, but all the leases are synched within a few minutes of

the last successful logon.

If your controllers are using shared storage, it is important to note that each controller will have its own copy of the

leases, so in an environment with 5 controllers and the worst case scenario of no desktops being registered, there

will be a total of 800K files created on the storage (assuming no inline de-duplication or other advanced storage

technology in operation).

In our example RDS host setup, we have ~110K files to synchronise, which will take approximately 18 minutes to

synchronise with the site database, assuming that number of users could launch at that rate. The amount of data is

larger at 820MB, but over an 18 minute interval this is still quite modest at less than 1 MB/s. A caching controller

will once again limit the impact on the system. As for VDI environments, this is for each controller, so consider the

impact for shared storage environments.

12

Disk systems with compression enabled

With the possible exception of the icon XML files, the lease files are text information that is likely to compress fairly

well on systems with compression enabled.

Forcing Lease Refresh

You can force leases to be recreated on a single or all controllers in the site. For a single controller use the following

PowerShell cmdlet

Update-BrokerLocalLeaseCache [-Workers] [-Applications] [-Icons] [-Desktops] [-Leases] [-

LoggingId <Guid>] [-AdminAddress <String>] [<CommonParameters>] Update-BrokerLocalLeaseCache

This will remove all or the specified leases from the local or target address if specified.

If you want to force all controllers to refresh their leases you can disable then re-enable connection leasing using the

following PowerShell cmdlets:

Set-BrokerSite -ConnectionLeasingEnabled $false

The leases will start to be deleted site wide from each controller; this will take some time depending on the number

of leases present.

To enable leasing again:

Set-BrokerSite -ConnectionLeasingEnabled $true

This will force synchronisation with the site database for each controller in the environment.

Disk activity when Leasing is active If connectivity to the site database is lost, or it becomes otherwise unavailable, the controllers will use the lease files

to broker connections. For each user, the controller will have to:

Read the enumeration lease to verify what resources are available to the user

Read the launch lease

Read the worker lease to direct the user to the appropriate RDS host or desktop

It most scenarios, the lease files will not have been accessed since their creation, so are not likely to be sitting in any

cache, so will require disk I/O to broker the session. If a large number of users try to connect in a short period of

time, the disk subsystems will see a large number of IOPS requests.

Cached lease file updates Each controller checks with the site database every 10 seconds to see if changes to, or updated lease files are

available, the new or updated leases are then synchronised at the normal rate of up to 1000 per 10 seconds.

Lease timeouts

By default, leases have a two week timeout, if a user has not accessed a resource for longer than this, their

associated lease files will be cleaned up and new leases created on their next login, subject to the site database

being accessible at this time.

13

Controller CPU load During the login phase the broker service will be busy servicing requests, but lease creation is handled by running on

a background thread, syncing every 10 seconds. As a result, the additional CPU load should be minimal. During

40,000 logon testing, there was minimal difference between controller CPU during initial logon seed leasing and

logon after seed leasing. However, should a site database problem occur, the brokers will have to handle load

caused by the VDAs.

CPU Load on Controllers when leasing is active

The additional CPU load caused by leases being used to broker sessions has not been significant compared to normal

brokering operations. When the site database goes down - as per previous releases of XenDesktop - VDA workers

will start to de-register with the controllers and then try and re-register, causing a spike in re-registration requests.

This is particularly relevant to the VDI case, where there will be a significant number of desktops trying to re-register

against the controllers in the site until re-registration is successful. In our 40,000 desktops scenario, peaks of 178 re-

registration requests were being processed per second, with CPU load peaking at 100%.

In previous versions of XenDesktop, the controller CPU load caused by the site database failing was a lesser issue

than restoring the database connectivity. For extended database outages, where all VDAs have become

unregistered, it may take some time for all VDAs to become registered again.

In a RDS host environment there are usually less VDAs than in the VDI environment, e.g. 100,000 users on 1000 RDS

hosts, so the registration storm is less severe and the controllers will not be as stressed.

With Connection Leasing enabled, the extra load will be dependent on the rate at which users are brokering sessions

whilst the database is down, but is relatively small compared to the load caused by the registration storm and logon

rates should approach those of normal environment operations.

For controllers deployed in a virtual environment where CPU resources have been heavily over committed, logon

rate and re-registration rates might be impacted, using Hypervisor and Windows performance counters is

recommended to characterise the load.

14

Connection Launch rates with Connection Leasing active During a test run, a target of 40,000 VDI logons was attempted at a rate of 60 logons every 1.08 seconds. During the

logon phase, the site database was deliberately disconnected and then reconnected twice during the test. The

transition period of from the site database going down to Connection Leasing becoming active and the subsequent

reconnection of the database are visible as flat lines. The launch rate takes a couple of minutes to recover to the

rate when the database is available, but does recover to a comparable pre-failure rate. The second outage is more

problematic as it occurs whilst VDAs are still in a registration phase caused by the first outage.

In situations where intermittent database issues are occurring, it is worth considering removing database

connectivity until all issues have been resolved in order to alleviate VDA registration storms and some users being

unable to logon during the transition periods.

15

Configuring Connection Leasing Parameters The default values used by Connection Leasing should work well with many environments. If you wish to change the default settings you can either:

Use Group Policy via GPOs

Manually configure the registry on each controller If you are using GPOs to configure your controllers, the GPO will store the configuration details in each controller registry under:

HKLM\Software\Policies\Citrix\DesktopServer\ConnectionLeasing

You should not change the registry values directly if using group policy to configure the settings.

If you are not using a GPO, the configuration is stored under: HKLM\Software\Citrix\DesktopServer\ConnectionLeasing

If you are not using GPOs, in a multi controller site, you will need to edit the registry on each controller. Note that Connection Leasing does not populate the registry keys by default, so you will need to create them if you wish to change the default values. As with any changes to the registry, care should be taken when modifying parameters and a roll back / recovery plan should be in place before changes are made.

Name Type Default Info Summary

DeletionCheckItemLimitPerCycle int 100 Minimum=1 Setting that controls the number of items check for possible deletion each deletion check interval, per item category. The number of items checked in a particular cycle is by whole subdirectories, so may exceed this limit by the size of the last subdirectory encountered.

EnumerationLeaseKeyMask int 9 Setting that controls the components of the enumeration lease key. Bit 0 - User Sid Bit 1 - Client Name Bit 2 - Client IP Address Bit 3 - ViaAG flag Bit 4 – AccessTags

16

LaunchRefCacheExpiryMaxMins int 3 Minutes Setting that controls the maximum time the logon tickets are cached in memory before they are discarded.

LeaseExpirationTimeInMins int 20160 Minutes Setting that specifies the time in minutes after which the lease will expire once it’s stored in the database. The default value of 20160 minutes corresponds to 14 days. The expired leases are removed every 30 minutes by the site service. Setting an expiration value smaller than 30 minutes would need a change to the site service frequency if the lease needs to be removed earlier. Expired leases will not be used by the controller even if they are still present in the lease cache.

LeaseMarkedDeletedTimeInMins int 30 Minutes Setting that specific the maximum time a lease will remain in deleted state before its purged.

MaxItemsPerSyncCycle int 1000 The maximum number of lease to sync per sync cycle. This helps to throttle and restrict the number of disk writes that would be generated every time the sync runs.

MaxRetryDuringLocalCacheDeletion int 5 Setting that controls the maximum number of time we will attempt to delete the local cache directory in case of IO exceptions.

MinLeaseLifetimeFractionBeforeRefresh

int 10 Setting that specifies the life time of an unchanged lease before its expiration time is refreshed. The value is specified as a fraction of the LeaseExpirationTimeInMins.

PendingFailureMaxSecs int 90 Seconds Setting that specific the maximum time in seconds to wait before entering leasing mode on hitting pending failure state in the DAL layer. Note that there is a default timeout of 30 seconds for SQL server queries, this timeout is in addition to that, so the total time for Connection Leasing to become active will be 120 seconds.

SyncCleanupDelaySecs int 120 Seconds Setting that controls the time between stale lease and cached data cleanup cycles.

SyncIntervalSecs int 10 Seconds Setting that specifies the intervals in which to check for any leases to sync. A sane value must be larger than SiteDynamicDataRefreshPeriodMs, as site dynamic data refreshed will tell when lease and other data last changed.

17

SyncLocation string %ProgramData%\Citrix\Broker\Cache

The location on local disk where the leases are to be cached.

SyncStartDelayMins int 1 Minutes Setting that controls the time to elapse before the first sync can run after the controller service has been started.

UploadQueueIdleMaxSecs int 10 Seconds Setting that controls the max time to wait for the upload queue to be idle. Once the queue idle time pass this limit, even if the queue item threshold is not reached the contents of the lease queue will be upload for sync.

UploadQueueMaxItems int 100 Setting that controls the max items to queue before lease upload is triggered.

18

Product Versions

Product Version

XenDesktop 7.6

Revision History

Revision Change Description Updated By Date

1.0 Document Created Joe Deller October 17th 2014

©2014 Citrix Systems, Inc. All rights reserved.

Citrix®, Access Gateway™, Branch Repeater™, Citrix Repeater™,HDX™, XenServer™, XenApp™, XenDesktop™ and Citrix Delivery Center ™ are trademarks of Citrix Systems,

Inc. and/or one or more of its subsidiaries, and may be registered in the United States Patent and Trademark Office and in other countries. All other trademarks and

registered trademarks are property of their respective owners.