infiniband in the lab (vsphere 5.5) vmworld 2013 barcelona vbrownbag tech talk
DESCRIPTION
Small presentation given at VMworld 2013 EMEA Barcelona for the vBrownbag Tech Talks about how and why to use InfiniBand in the LabTRANSCRIPT
![Page 1: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/1.jpg)
InfiniBand in the Lab
Erik Bussink@ErikBussink
www.bussink.ch
www.VSAN.info
1
![Page 2: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/2.jpg)
ExecutiveSummary
FastCheap
2
![Page 3: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/3.jpg)
•EMC Isilon
•EMC xtremIO
•PureStorage
•Oracle Exadata
•Nexenta
•TeraData
•Gluster
3
Who uses InfiniBand for Storage ?
![Page 4: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/4.jpg)
What InfiniBand Offers
•Lowest Layer for scalable IO interconnect
• High-Performance
• Low Latency
• Reliable Switch Fabric
•Offers higher layers of functionality
• Application Clustering
• Fast Inter Process Communications
• Storage Area Networks
4
![Page 5: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/5.jpg)
InfiniBand physical layer
•Physical layer based on 802.3z specification operating at 2.5Gb/s same standard as 10G ethernet (3.125Gb/s)
•InfiniBand layer 2 switching uses 16 bit local address (LID), so limited to 2^16 nodes on a subnet.
5
![Page 6: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/6.jpg)
6
![Page 7: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/7.jpg)
Connectors
•For SDR (10Gb/s) and DDR (20Gb/s) use the CX4 connectors
•For QDR (40Gb/s) and FDR (56Gb/s) use QSFP connectors
7
![Page 8: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/8.jpg)
Price comparison on Adapters
8
![Page 9: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/9.jpg)
And switches too...Example... 24 ports DDR (20Gb/s) for $499
Latest firmware (4.2.5) has embedded Subnet Manager
9
![Page 10: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/10.jpg)
Subnet Manager
•The Subnet Manager assigns Local IDentifiers (LIDs) to each port in the IB fabric, and develops a routing table based off the assigned LIDs
•Hardware based Subnet Manager are located in the Switches
•Software based Subnet Manager is located on a host connected to the IB fabric
•The OpenSM from the OpenFabrics Alliance works on Windows and Linux
10
![Page 11: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/11.jpg)
OpenFabrics Enterprise Distribution (OFED)
11
![Page 12: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/12.jpg)
Installing InfiniBand on vSphere 5.1
12
![Page 13: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/13.jpg)
Configure MTU and OpenSM
•esxcli system module paramters set y-m=mlx4_core -p=mtu_4k=1
•On Old switches (No 4K support) MTU is 2044 (2048-4) for IPoIB
•vi partitions.conf
•Add single line "Default=ox7fff,ipoib,mtu=5:ALL=full;"
•copy partitions.conf /scratch/opensm/adapter_1_hca/
•copy partitions.conf /scratch/opensm/adapter_2_hca/
13
![Page 14: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/14.jpg)
14
![Page 15: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/15.jpg)
Installing InfiniBand on vSphere 5.5
•Mellanox drivers are in vSphere 5.5, and they support 40GbE inbox when using ConnectX-3and SwitchX products.
•If not same HCA you need to uninstall the Mellanox drivers
• esxcli software vib remove -n=net-mlx4-en -n=net-mlx4-core
•And reboot
•Then install the Mellanox OFED 1.8.2
• esxcli software vib install -d /tmp/MLX
•And reboot
15
![Page 16: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/16.jpg)
Installing InfiniBand on vSphere 5.5
16
![Page 17: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/17.jpg)
Copy partitions.conf
•Set MTU for mlx4
•Copy partitions.conf to /scratch/opensm/<HCA_Port>/
17
![Page 18: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/18.jpg)
InfiniBand IPoIB backbone for VSAN
18
![Page 19: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/19.jpg)
Configure VMkernel MTU
![Page 20: InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk](https://reader036.vdocuments.site/reader036/viewer/2022062513/55515168b4c905e1708b459a/html5/thumbnails/20.jpg)
InfiniBand in the Lab
Thanks to Raphael Schitz,William Lam, Vladan Seget, Gregory Roche
20
Fast & Cheap
Erik Bussink@ErikBussink
www.bussink.ch
www.VSAN.info