![Page 1: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/1.jpg)
VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus
10GE Networking
BRAD HEDLUND .com
Brad Hedlund, CCIE #5530 Data Center Architect Cisco Systems [email protected]
v 2.5
![Page 2: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/2.jpg)
- About the Author -
Brad Hedlund Technical Solutions Architect, Data Center Cisco Systems, Inc. CCIE #5530
Blog: http://bradhedlund.com Twitter: http://twitter.com/bradhedlund E-mail: [email protected] Comments welcome.
BRAD HEDLUND .com
http://bradhedlund.com/about/
![Page 3: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/3.jpg)
Cisco UCS Networking Overview
BRAD HEDLUND .com
6100 Fabric A
6100 Fabric B
B200 B250
CNA
FEX
A CNA CNA
FEX
B
FEX
A
FEX
B
SAN A SAN B ETH 1 ETH 2
MGMT MGMT
Chassis 1 Chassis 2
![Page 4: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/4.jpg)
Emulex/Qlogic CNA
10GE
vNIC vNIC
Fabric A
Fabric B
vmnic0
10GE
vHBA vHBA
vhba0 vmnic1 vhba1
CNA
vSwitch / Nexus 1000V
FEX FEX
BRAD HEDLUND .com
![Page 5: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/5.jpg)
Emulex/Qlogic CNA – Logical View
CNA
vSwitch / Nexus 1000V
FEX FEX
BRAD HEDLUND .com
vmnic0 vhba0 vmnic1 vhba1
vfc 1
vEth1
vEth2
vfc 2
Fabric Interconnect A Fabric Interconnect B
![Page 6: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/6.jpg)
Service Profile / Fabric Selection
BRAD HEDLUND .com
![Page 7: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/7.jpg)
Cisco VIC “Palo”
vNIC
Fabric A
Fabric B
vhba0
Cisco VIC
vHBA vNIC vNIC vNIC vNIC vNIC vNIC vNIC vHBA 58
vhba1
10GE 10GE
vSwitch / vDS / Nexus 1000V
vmnic0
vmnic1
vmnic2
vmnic3
vmnic4
vmnic5
vmnic6
vmnic7
FEX FEX
BRAD HEDLUND .com
![Page 8: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/8.jpg)
Cisco VIC QoS: Example
vNIC 2
Fabric A
Fabric B
Cisco VIC
vHBA 1
vNIC 1
vNIC 3
vNIC 4
vNIC 5
vNIC 6
vNIC 8
vNIC 7
vHBA 2
58
FEX FEX
BRAD HEDLUND .com
0 2 4 5 6
7 7 10GE
0 1 2 4 5 6
10GE
40% 40% 10%
Strict Priority
10% 40% 40% 10% 10%
Strict Priority
1 3
RR RR
VIC Mgmt
VIC Mgmt
Bandwidth Scheduler Bandwidth Scheduler
COS 1 COS 3 COS 4 COS 5 COS 1 COS 3 COS 4 COS 5
Limit TX 1GE Limit TX 1GE
TX TX
No Drop
(8) COS Queues
No Drop
3
![Page 9: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/9.jpg)
Cisco VIC – Logical View
vhba0
Cisco VIC
vhba1
vSwitch / Nexus 1000V
vmnic0
vmnic1
vmnic2
vmnic3
vmnic4
vmnic5
vmnic6
vmnic7
vfc 1
vEth1
vEth2
vEth3
vEth4
vfc 2
vEth6
vEth7
vEth8
vEth5
FEX A FEX B
Fabric Interconnect A Fabric Interconnect B
BRAD HEDLUND .com
![Page 10: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/10.jpg)
I/O Consolidation with 10GE
BRAD HEDLUND .com
vmnic0
vmnic1
vmnic2
vmnic3
10GE Network Switch
VM Data
VMotion
MGMT
IP Storage
CONCERN: Bandwidth Availability
SOLUTIONS: 1. MAX bw per NIC
Rate Limits 2. MIN bw per NIC
QoS 3. Do Nothing
Let it rip
![Page 11: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/11.jpg)
I/O Consolidation with 10GE
BRAD HEDLUND .com
vmnic1
10GE Network Switch
VM Data
VMotion
MGMT
IP Storage
CONCERN: Bandwidth Availability
SOLUTIONS: 1. MAX bw per Class
Rate Limits 2. MIN bw per Class
QoS 3. Do Nothing
Let it rip
![Page 12: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/12.jpg)
MAX Bandwidth – Rate Limits
BRAD HEDLUND .com
vmnic0
vmnic1
vmnic2
vmnic3
10GE
Network Switch
VM Data
VMotion
MGMT
IP Storage
2.5G
2.5G
2.5G
2.5G
Sum of Limits must not exceed physical link speed Inefficient use of unused BW
This slide animated @ http://bradhedlund.com/2010/08/16/cisco-ucs-qos-vs-hp-virtual-connect-rate-limiting/
![Page 13: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/13.jpg)
MIN Bandwidth – QoS
BRAD HEDLUND .com
vmnic0
vmnic1
vmnic2
vmnic3
10GE
Network Switch
VM Data
VMotion
MGMT
IP Storage
Minimum BW guarantee with no Maximum Efficient: unused BW available to other NICs
This slide animated @ http://bradhedlund.com/2010/08/16/cisco-ucs-qos-vs-hp-virtual-connect-rate-limiting/
![Page 14: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/14.jpg)
MIN Bandwidth – QoS
BRAD HEDLUND .com
10GE
Network Switch
VM Data
VMotion
MGMT
IP Storage
vmnic1
Nexus 1000V Classifies Traffic
UCS 6100 Nexus 5000 Enforces Policy
Cisco VIC Enforces Policy
![Page 15: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/15.jpg)
Traffic Considerations
Required
• Guest VM Data
• Management – VMKernel
– Service Console
• vMotion
• Central Storage Access
Optional
• Fault Tolerance (FT)
• NFS
• iSCSI
• FCoE
• Realtime – Video/Voice, Market Data
BRAD HEDLUND .com
![Page 16: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/16.jpg)
Cisco UCS + VIC Design (1): Required Traffic
vhba0
vhba1
vSwitch or vDS
vmnic0
vmnic2
vmnic4
vmnic1
vmnic3
vmnic5
vfc 1
vEth1
vEth2
vEth3
vfc 2
vEth6
vEth7
vEth8
Fabric Interconnect A Fabric Interconnect B
VM DATA vMotion MGMT FC
Primary vmnic1 vmnic2 vmnic5 MP
Backup vmnic0 vmnic3 vmnic4 MP
Min BW 10% 1GE Max 40% 10% 40%
Class Bronze COS 1 Best Effort COS 0 Silver COS 2 COS 3 BRAD HEDLUND .com
![Page 17: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/17.jpg)
UCS Manager QoS System Class Design #1
BRAD HEDLUND .com
• Assign each Class of Service a minimum bandwidth weighting • System wide policy
![Page 18: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/18.jpg)
Apply QoS Policy to vNIC
BRAD HEDLUND .com
• Create QoS Policy • Assign Policy to vNIC
![Page 19: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/19.jpg)
Cisco UCS + VIC Design (2): Req + 1 Opt
vhba0
vhba1
vSwitch or vDS
vmnic0
vmnic2
vmnic4
vmnic6
vmnic1
vmnic3
vmnic5
vmnic7
vfc 1
vEth1
vEth2
vEth3
vEth4
vfc 2
vEth6
vEth7
vEth8
vEth5
Fabric Interconnect A Fabric Interconnect B
FC NFS VM DATA vMotion MGMT
Primary vmnic1 vmnic2 vmnic5 vmnic7 MP
Backup vmnic0 vmnic3 vmnic4 vmnic6 MP
Min BW 10% 1GE Max 30% 10% 10% 40%
Class Bronze COS 1 Best Effort COS 0 Silver COS 2 Gold COS 4 COS 3 BRAD HEDLUND .com
![Page 20: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/20.jpg)
UCS Manager QoS System Class Design #2
BRAD HEDLUND .com
• Assign each Class of Service a minimum bandwidth weighting • System wide policy
![Page 21: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/21.jpg)
Cisco UCS + VIC Design (3): Req + 2 Opt + Realtime
vhba0
vhba1
vSwitch or vDS
vmnic2
vmnic4
vmnic6
vmnic8
vmnic1
vmnic3
vmnic5
vmnic7
vfc 1
vEth2
vEth3
vEth4
vEth5
vfc 2
vEth7
vEth8
vEth9
vEth6
Fabric Interconnect A Fabric Interconnect B
VM DATA vMotion FT MGMT NFS
vmnic9
vmnic0
vEth10
vEth1
Primary vmnic0 vmnic1 vmnic2 vmnic5 vmnic6 vmnic9 MP
Backup vmnic1 vmnic0 vmnic3 vmnic4 vmnic7 vmnic8 MP
Min BW 1GE Max 10% 10% 20% 10% 10% 40%
Class Bronze COS 1 Bronze COS 1 Plat COS 5 Gold COS 4 Best Effort COS 0 Silver COS 2 COS 3
FC
BRAD HEDLUND .com
Realtime
![Page 22: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/22.jpg)
UCS Manager QoS System Class Design #3
BRAD HEDLUND .com
• “Platinum” Class for Realtime traffic • Lossless Ethernet and Multicast optimized • “Silver” and “Bronze” for vMotion & NFS with Jumbo Frames
![Page 23: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/23.jpg)
Design #4: (2) 10GE NICs with Nexus 1000V Cisco UCS or Nexus 5000
vhba0
vhba1
vmnic1
vmnic0
vfc 1
vEth4
vfc 2
vEth7
Fabric Interconnect A / Nexus 5000 #1 Fabric Interconnect B / Nexus 5000 #2
NEXUS 1000V QoS Classification
Primary vmnic0 vmnic1 vmnic0 vmnic1 vmnic0 vmnic1 MP
Backup vmnic1 vmnic0 vmnic1 vmnic0 vmnic1 vmnic0 MP
Min BW 1GE Max 10% 10% 20% 10% 10% 40%
Class Bronze COS 1 Bronze COS 1 Plat COS 5 Gold COS 4 Best Effort COS 0 Silver COS 2 COS 3
VM DATA vMotion FT MGMT NFS FC Realtime
QoS Enforcement (if Cisco VIC)
QoS Enforcement
Cisco VIC or Other adapter Cisco UCS server or non-Cisco
BRAD HEDLUND .com
QoS Enforcement (if N1K v1.4)
![Page 24: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/24.jpg)
vNIC QoS Policy for Design #4
BRAD HEDLUND .com
• Preserve and use COS markings from Nexus 1000V • Same QoS System Class settings as Design #3
![Page 25: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/25.jpg)
Design #5: (2) 10GE NICs, no Cisco VIC, no N1K Nexus 5000
vhba0
vhba1
vmnic1
vmnic0
vfc 1
Eth 1/1
vfc 2
Eth 1/1
Nexus 5000 #1 Nexus 5000 #2
vSwitch or vDS
QoS Classification
QoS Enforcement
Primary vmnic0 vmnic1 vmnic0 vmnic1 vmnic0 vmnic1 MP
Backup vmnic1 vmnic0 vmnic1 vmnic0 vmnic1 vmnic0 MP
Min BW 1GE Max 10% 10% 20% 10% 10% 40%
Class Bronze COS 1 Bronze COS 1 Plat COS 5 Gold COS 4 Best Effort COS 0 Silver COS 2 COS 3
VM DATA vMotion FT MGMT NFS FC Realtime
Cisco VIC or Other adapter Cisco C-Series or non-Cisco
BRAD HEDLUND .com
VMware NetIOC ? FCoE?
![Page 26: VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexusbradhedlund.s3.amazonaws.com/.../VMware-10ge-QoS-with-Cisco-UC… · VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus](https://reader031.vdocuments.site/reader031/viewer/2022021501/5a9ee9d27f8b9a8e178c11e1/html5/thumbnails/26.jpg)
Narrative posted at:
BRAD HEDLUND .com
http://bradhedlund.com/?p=1650