set up vdi for nvidia tesla t4 and p6 in amulet hotkey products · knowledge base article113 set up...

www.amulethotkey.com KB-A000-0113 v3.2 09 June 2020 |

Knowledge Base Article Set up VDI for NVIDIA Tesla T4 and P6 in Amulet Hotkey products This Knowledge Base Article describes how to set up the NVIDIA Tesla T4 and NVIDIA Tesla P6 GPUs used in the following products:

NVIDIA Tesla T4 Products NVIDIA Tesla P6 Products

Dell MX7000 with Amulet Hotkey CoreModule(s) and CoreCartridge(s):

• Dell PowerEdge MX740c • Dell PowerEdge MX840c

DXG-P6 (Mezzanine card) for:

• Dell PowerEdge M630 • Dell PowerEdge M640 (CoreStation VM640)

Dell PE FX2s with Amulet Hotkey FX400 CoreModule(s):

• Dell PE FC640

DXF-G add in card for DXF-EXP module for:

• Dell PE FC630 • Dell PE FC640 (CoreStation VFC640)

Note: The Tesla T4, DXG-P6 and DXF-G automatically configure their mode

Always check our website for the latest versions of approved drivers.

Note: There is also a Knowledge Base Article KBA114 (KB-A000-0114) for DXG cards with an AMD FirePro S7100x MXM installed.

For setting up the DXG-M6, see earlier versions of this document or contact Technical Support.

113

http://www.amulethotkey.com/

Knowledge Base Article 113 Set up VDI for NVIDIA Tesla T4 and P6 in Amulet Hotkey products


Contents 1 Procedure summary .............................................................................................................3

2 Prepare the GPU card for installation ...................................................................................4

2.1 Prepare the DXG-P6 or DXF-G GPU card for the CoreStation VM640 or CoreStation VFC640.......... 4

2.2 Prepare the NVIDIA Tesla T4 GPU accelerator card for use with MX7000 or FX2s chassis ............... 5

3 Prepare the blade or compute sled for virtualization ............................................................9

3.1 Install ESXi ......................................................................................................................................... 9

3.2 Set PCI device availability .................................................................................................................. 9

3.3 Install the NVIDIA vGPU Manager VIB ............................................................................................. 10

3.4 Add NVIDIA GPU to the selected VM .............................................................................................. 12

3.5 Install the Horizon 7 Agent .............................................................................................................. 15

3.6 Install VMware guest driver ............................................................................................................ 16

3.7 Install the GRID license server ......................................................................................................... 17

4 Troubleshooting ................................................................................................................ 18

4.1 Hard drive not recognised ............................................................................................................... 18

5 Contact us: ........................................................................................................................ 19

6 Resources .......................................................................................................................... 19

6.1 NVIDIA links ..................................................................................................................................... 19




1 Procedure summary

Prepare the DXG card for installation:

• Prepare the DXG-P6 or DXF-G GPU card for the CoreStation VM640 or CoreStation VFC640

• Prepare the NVIDIA Tesla T4 GPU accelerator card for use with MX7000 or FX2s chassis

Prepare the blade for virtualization:

• Install ESXi

• Set PCI device availability

• Install the NVIDIA vGPU Manager VIB

• Add NVIDIA GPU to the selected VM

• Install VMware guest driver

• Install the GRID license server

See the NVIDIA GPUMODESWITCH User Guide for more detailed information. See an example copy here.


http://images.nvidia.com/content/pdf/grid/guides/GRID-gpumodeswitch-UserGuide.pdf



2 Prepare the GPU card for installation

2.1 Prepare the DXG-P6 or DXF-G GPU card for the CoreStation VM640 or CoreStation VFC640

2.1.1 Download the NVIDIA files

Step Action

1 Register to access the evaluation NVIDIA grid software and license. Register here for free. http://www.nvidia.com/object/nvidia-grid-buy.html.

2 Download the latest vSphere (NVIDIA GRID software) file.

3

Make sure the following BIOS settings are set:

Memory I/O Mapping above 4GB = Enabled

Memory Mapped I/O Base = 12TB (this setting is for the CoreStation VM640 and CoreStation VFC 640 only)

SR IOV Global Enabled = Enabled

State of Embedded Video Controller = Enabled

Note: If the Memory Mapped I/O Base is set above 12TB the DXG will not initialise correctly.

4 Proceed to Prepare the blade for virtualization.


http://www.nvidia.com/object/nvidia-grid-buy.html



2.2 Prepare the NVIDIA Tesla T4 GPU accelerator card for use with MX7000 or FX2s chassis

2.2.1 Notes for the MX7000 chassis setup

The MX7000 CoreCartridge may contain up to two NVIDIA Tesla T4 GPU accelerator cards. A CoreModule in the MX7000 may contain up to four CoreCartridges. An MX7000 may contain up to two CoreModules. In total, there may be up to eight Compute Sleds and 16 NVIDIA Tesla T4 cards. See notes below. Repeat the setup in the sections that follow for each Compute Sled and T4 card.

2.2.2 Notes for MX740c (single-width sled)

Requires two Xeon CPUs (second CPU is needed for the PCIe connection for fab B mezz card).

Two GPUs per sled requires two CoreModules, each with a CoreCartridge in the same position. Two GPUs per CoreCartridge requires two sleds (positions 1&2, 3&4, 5&6, or 7&8). One GPU for every sled requires a fully populated CoreModule and eight sleds in the chassis. Two GPUs for every sled requires two fully populated CoreModules and eight sleds.

See Table 1.

No. of sleds CoreCartridges GPU/Sled CoreModules Mezz cards 1 2 2 2 1 2 1 1 1 2

Up To … 8 4 1 1 8 8 8 2 2 8

Table 1: Supported CoreCartrigde Configurations for use with MX740c




2.2.3 Notes for MX840c (double-width sled)

Requires four Xeon CPUs (CPU2 and CPU4 are needed for PCIe connection for fab B mezz card).

Two GPUs per sled requires two CoreModules, each with a CoreCartridge in the same position.

Two GPUs per CoreCartridge requires one sled with two Mezz cards installed.

(In positions 1&2 or 3&4, 5&6 or 7&8).

Four GPUs per sled requires two CoreModules and two mezz cards installed.

Two GPUs for every sled requires a fully populated CoreModule.

Four GPUs for every sled requires two fully populated CoreModules. See Table 2.

No. of sleds CoreCartridges GPU/Sled CoreModules Mezz cards 1 2 2 2 1 1 1 2 1 2 1 2, 3, or 4 4 2 2 1 2 4 2 2

Up To … 4 4 2 1 8 4 8 4 2 8

Table 2: Supported CoreCartrigde Configurations for use with MX840c




2.2.4 Notes for the FX2s FX400 CoreModule setup

FX2s chassis with two FX400 CoreModules per sled

FX2s chassis with four FX400 CoreModules per sled

Notes: The FX400 CoreModule contains a single NVIDIA Tesla T4 GPU accelerator card.

The FX2s chassis can accept up to four sleds and eight FX400 CoreModules.

The remapping tool allows you to assign the maximum of four NVIDIA Tesla T4 GPU accelerator cards per sled.

See the FX400 CoreModule Quick Start Guide, QS-FX40-0001 for installation details.

2.2.5 Notes for the FX2s DXF-EXP Quad-Expansion Module setup

FX2s chassis with two PCIe cards per sled

FX2s chassis with four PCIe cards per sled

Notes: The DXF-EXP Quad Expansion Module can contain up to four PCIe cards.

The Quad Expansion Module supports GPU accelerator cards for implementing VDI, such as the DXG-P6 or host cards for one-to-one operation. The setup for VDI is covered in this document, the set up for one-to-one is covered in the Remote Desktop Configuration Manual, HB-CONF-0001

The FX2s chassis can support up to two Quad Expansion Modules.

See the Quad Expansion Module Quick Start Guide, QS-DXFE-0001 for installation details.


https://resources.amulethotkey.com/download/Configuration_Manual_HB-CONF-0001.pdf

https://resources.amulethotkey.com/download/DXF-EXP-V_QSG_QS-DXFE-0001.pdf



2.2.6 Download the NVIDIA files

Repeat this section for every Compute Sled that will connect to a NVIDIA Tesla T4 GPU accelerator card.

Step Action

1 Register to access the evaluation NVIDIA grid software and license. Register here for free. http://www.nvidia.com/object/nvidia-grid-buy.html.

2 Download the latest vSphere (NVIDIA GRID software) file.

3

Make sure the following BIOS settings are set:

Memory I/O Mapping above 4GB = Enabled

Memory Mapped I/O Base = 12TB (this setting is for the MX740c and MX840c only)

SR IOV Global Enabled = Enabled

State of Embedded Video Controller = Enabled

Note: If the Memory Mapped I/O Base is set above 12TB the T4 will not initialise correctly.

4 Proceed to Prepare the blade for virtualization.


http://www.nvidia.com/object/nvidia-grid-buy.html



3 Prepare the blade or compute sled for virtualization

Note: For T4 GPU card setup, repeat these sections as necessary for each Compute Sled/T4 card.

3.1 Install ESXi

Step Action

1 Navigate to the website: http://www.dell.com/support/home/us/en/19/Drivers/DriversDetails?driverId=HJFY8.

2 Download the file. For example: VMware-VMvisor-Installer-6.0.0-2494585.x86_64-Dell_Customized-A00.iso

Burn or mount the Dell VMware ESXi iso.

3.2 Set PCI device availability

Step Action

1 Make sure the NVIDIA GPU card is not passed through for DirectPath I/O.

Select PCI devices and make sure there is no entry. See Figure 1.

Figure 1: M6 card not passed through for DirectPath

2 Under ‘All PCI Devices’ make sure that the box is unchecked for the device name.


http://www.dell.com/support/home/us/en/19/Drivers/DriversDetails?driverId=HJFY8



3.3 Install the NVIDIA vGPU Manager VIB

Note: Users installing the latest version of vSphere (ESXi 6.5 or later) must put the GPU in 'Shared Direct' mode (see Figure 2). To do this:

a. select the Host Graphics tab from the Graphics side menu;

b. if the Default graphics type is not set to ‘Shared Direct’, then click on Edit and change the settings.

Figure 2: Host Graphics window

Step Action

1 Access the ESXi host via ESXi Shell or SSH and turn off ECC. Enter the command: nvidia-smi –e 0

Note: When in Graphics Mode, ECC must be disabled for virtual machines to start.

2 Reboot the machine.

3 Copy the VIB file to the ESXi host. Note the location as you will need this in step 5.

4 Put the ESXi host into Maintenance Mode.

5 Enter the install command. For example: esxcli software vib install –v /path_to_vib/NVIDIA-vGPU-VMware_ESXi_6.0_Host_Driver_361.45.09-1OEM.600.0.0.2494585.vib

Notes:

Use the full path to the vib file, even if running from the directory it is stored in.

If the vib file is located on VSAN datastore, navigate to the directory before running the vib.

6 Reboot the ESXi host and take it out of Maintenance Mode.




Step Action

7 Make sure that the correct NVIDIA VIB is installed. Enter the command: esxcli software vib list | grep –I nvidia

8 To check, enter the command: nvidia-smi

The VIB details are displayed, the version is 352.83 in this example. See Figure 3.

Figure 3: Verify the correct VIB is installed

9 After the VIB is installed correctly, the NVIDIA card will be available for the Virtual Machines (VMs) to use. See Figure 4.

Figure 4: NVIDIA GPU card available




3.4 Add NVIDIA GPU to the selected VM

Step Action

1 From the web, open the VMware vSphere management console, navigate to the desired VM. For example, Win 7 vGPU3.

2 Select Edit VM Settings.

3 Select the Virtual Hardware tab.

4 Select Shared PCI Device from the drop down menu next to ‘New device’, click Add.

NVIDIA GRID vGPU option is automatically selected.

5 To specify how much video memory (frame buffer) the VM can use, you must choose a specific Grid Profile (using Edit Settings for the VM).

• To review GPU options for a DXG-P6, see step 5a.

• To review GPU options for a T4, see step 5b

For example, in Figure 5, grid_m6-4q is selected.

Figure 5: Add Shared PCI Device




Step Action

5a Figure 6 shows example profiles for the DXG-P6.

Note: Taken from the NVIDIA user guide – always check the latest version.

GPU Type Use Case Frame Buffer (Mbytes)

Virtual Display heads

Maximum Resolution per Display Head

Maximum vGPUs per GPU

Maximum vGPUs per Board

Required License Edition

P6-16Q Designer 16384 4 4096×2160 1 1 Quadro vDWS




P6-1Q Power User, Designer

1024 2 4096×2160 16 16 Quadro vDWS

P6-1B Power User 1024 4 2560×1600 16 16 GRID Virtual PC or Quadro vDWS

P6-16A Virtual Application User

16384 1 1280×1024 1 1 GRID Virtual Application










Figure 6: GRID Virtual GPU profiles for the DXG-P6

5b Figure 7 shows example profiles for the Tesla T4.

Note: See the NVIDIA Virtual GPU Software User Guide for the latest information.

Virtual GPU Type

Intended Use Case

Frame Buffer (MB)

Virtual Display Heads


Maximum vGPUs per

GPU

Maximum vGPUs per

Board


T4-16Q

Virtual Work-stations

16384 4 4096×2160 1 1 Quadro vDWS

T4-8Q Virtual Work-stations

8192 4 4096×2160 2 2 Quadro vDWS


https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html



Step Action

Virtual GPU Type

Intended Use Case

Frame Buffer (MB)

Virtual Display Heads


Maximum vGPUs per

GPU

Maximum vGPUs per

Board



4096 4 4096×2160 4 4 Quadro vDWS


2048 4 4096×2160 8 8 Quadro vDWS

T4-1Q Virtual Desktops & Workstations

1024 2 4096×2160 16 16 Quadro vDWS

T4-16C Training Workloads

16384 1 4096×21602 1 1 vComputeServer or Quadro vDWS

T4-8C Training Workloads


T4-4C Inference Workloads


T4-2B Virtual Desktops

2048 2 4096×2160 8 8 GRID Virtual PC or Quadro vDWS

T4-2B4 Virtual Desktops


T4-1B Virtual Desktops


T4-1B4 Virtual Desktops


T4-16A Virtual Applications










Figure 7: GRID Virtual GPU profiles for the NVIDIA Tesla T4

6 Click OK.


https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#virtual-gpu-types-grid__c-series-vgpu-graphics-note



https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#virtual-gpu-types-grid__a-series-vgpu-max-res-note







3.5 Install the Horizon 7 Agent

Step Action

1 Go to the website:

https://my.vmware.com/web/vmware/details?downloadGroup=VIEW-700-ENT&productId=577&rPId=11494

2 Click the Download button to download and then install the Horizon 7 Agent in that VM. See Figure 8.

Figure 8: Download the Horizon 7 Agent

3 Click the Download button to download and then install the Horizon 7 Agent Direct-Connection in that VM. See Figure 8.

Then, you can connect to it with the VMware Horizon View Client or Zero Clients. This enables the NVIDIA vGPU remote connection support.

Note: Do not use RDP to connect.

4 Install the NVIDIA license server and enable with either the evaluation license or purchased license to unlock the full feature set of NVIDIA vGPUs.

See the NVIDIA license server install PDF for more details.

2

1






3.6 Install VMware guest driver

Step Action

1 Get the NVIDIA Operating system drivers from the NVIDIA website. For example, in the file: NVIDIA-GRID-vSphere-6.0-361.45.09-362.56.zip

There are several drivers, choose the correct driver for your operating system. See Figure 9.

Example: The files 362.56_grid_win8_win7_64bit_english.exe or 362.56_grid_win10_64bit_english.exe.

Figure 9: Example VMware drivers

2 From the Device Manager, install the correct driver for your guest OS in the VM. See Figure 10.

Figure 10: Install the correct driver with Device Manager




3.7 Install the GRID license server

Step Action

1 Follow the NVIDIA GRID™ ENTERPRISE SOFTWARE Quick Start Guide (QSG-07847-001_v02, January 2016) to set up the GRID license server.

For more detailed information, see the GRID LICENSING User Guide (DU-07757-001, April 2016).


http://images.nvidia.com/content/pdf/grid/guides/quickstartguide.pdf

http://images.nvidia.com/content/pdf/grid/guides/GRID-Licensing-Guide.pdf



4 Troubleshooting

4.1 Hard drive not recognised

1. If the Hard Disk Drive (HDD) is not detected at startup (see screenshot), it may be that the Embedded Video Controller is not set to Enabled.

2. Check the settings in the BIOS.

3. If the Embedded Video Controller is set to Disabled then you must set it to Enabled to rectify the issue.




4. Once the setting has been set, reboot the blade and the following messages appear.

5 Contact us:

Support Center Telephone Email

North America +1 (212) 269 9300 [email protected]

[email protected]

Europe +44 (0)20 7960 2400 [email protected]

Asia Pacific - [email protected]

Latin America - [email protected]

6 Resources

Visit our resources page, https://resources.amulethotkey.com/resources. See the DXG page on our website for the latest documentation including our Technote.

6.1 NVIDIA links

GRID datasheet

GRID Support

GPUMODESWITCH User Guide

Tesla T4 Datasheet


mailto:[email protected]





https://resources.amulethotkey.com/resources

http://www.amulethotkey.com/products/virtual-workstations/virtual-blade-workstation/

http://images.nvidia.com/content/grid/pdf/188270-NVIDIA-GRID-Datasheet-NV-US-FNL-Web.pdf

https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html

http://images.nvidia.com/content/pdf/grid/guides/GRID-gpumodeswitch-UserGuide.pdf

https://www.nvidia.com/en-gb/data-center/tesla-t4/

set up vdi for nvidia tesla t4 and p6 in amulet hotkey products · knowledge base article113 set up...

Documents