1. vSphere Networking and vCloud Networking Suite Best Practices and Troubleshooting Richard Cockett, VMware Umesh Goyal, VMware Software India Pvt ltd VSVC5103 #VSVC5103

2. 22 Agenda vSphere Networking Anatomy of Virtual Network Basics of Virtual Networking Teaming - Redundancy and Load Balancing VLAN Implementation Distributed Virtual Network Network IO Control Configuration Best Practices 3. 33 Anatomy of Virtual Networking Service Console Physical Network VM0 VM1 VM2 VM3 ESX/ESXi Host vmkernel Port Group Virtual NIC (vnic) Physical NIC (vmnic or pnic) Physical switch Service Console (vswif) Vmkernel (vmknic) Uplinks vSwitch NIC Teams 4. 44 vNetwork Concepts Virtual Network Adapters vNic VMs interface to the network vmknic vSphere hypervisors interface to network (nfs, iSCSI, vMotion, FT, Management) vswif Interface for Service Console (not present on ESXi) Physical Network Adapter pNic for communicating with entities outside ESX/ESXi host Virtual Switch vSwitch forwards packets between vNics, vmknics, and pNics Port Group Group of ports sharing the same configuration (e.g. vlan) Uplinks: connections to physical switches NIC Team: a group of pNics connected to the same physical network 5. 55 Three Types of Virtual Switches vNetwork Standard Switch (vSS) Created and managed on a per-host basis Support basic features such as VLAN, NIC teaming, port security vNetwork Distributed Switch (vDS) Created and managed at vSphere vCenter Supports all vSS features and more (PVLAN, traffic management, etc.) NOTE: vSS/vDS share same etherswitch module, only control path differ Cisco Nexus 1000v (N1K) Created and managed by VSM (either VM or hardware/Nexus 1010) Supports features typically available in Cisco hardware switches 6. 66 ESX/ESXi Network Traffic - Classification Virtual Machine Traffic Traffic sourced and received from virtual machine(s) Isolated from each other VMotion Traffic Traffic sent when moving a virtual machine from one ESX/ESXi host to another Must be dedicated and isolated Management Traffic Should be isolated from VM traffic If VMware HA is enabled, includes heartbeats IP Storage TrafficNFS, iSCSI If using the software iSCSI initiator FT Traffic Should be isolated completely Generally heavy I/Os and low latency (< 1 ms) 7. 77 NIC Teaming 8. 88 Load Balancing - Originating Virtual Port ID Based Default mode, distributes load on a per vnic basis Physical switches not aware/involved Virtual NICs VM ports uplink ports Teamed physical NICs 9. 99 Load Balancing - MAC Based Teaming Distributes load on a source MAC hash basis Physical switches not aware/involved VM ports uplink ports Virtual NICs Teamed physical NICs 10. 1010 Load Balancing - IP Hash Based Distributes load on a per SRC IP/DST IP basis (hash) Requires Portchannel/Etherchannel on physical switches VM ports uplink ports Virtual NICs Teamed physical NICs PM0 PM2PM1 SRC IP A DST IP D DST IP E DST IP F SRC IP B SRC IP C 11. 1111 Load Based Teaming Introduced in vSphere 4.1 Only traffic-load-aware teaming policy Supported only with the vNetwork Distributed Switch (vDS) Reshuffles the port binding dynamically Only move a flow when the mean send or receive utilization on an uplink exceeds 75% of capacity Default Change over time is 30 Seconds In combination with VMware Network IO Control (NetIOC), LBT offers a powerful solution Refer: http://blogs.vmware.com/performance/2010/12/vmware-load-based- teaming-lbt-performance.html 11 12. 1212 VLAN Implementation 13. 1313 VLAN Tagging Options vnic vnic vnic vSwitch Physical Switch vnic vnic vnic vSwitch Physical Switch vnic vnic vnic vSwitch Physical Switch VST Virtual Switch Tagging VGT Virtual Guest Tagging EST External Switch Tagging VLAN Tags applied in vSwitch VLAN Tags applied in Guest PortGroup set to VLAN 4095 External Physical switch applies VLAN tagsVST is the preferred and most common method Port Groups assigned to a VLAN 14. 1414 vNetwork Distributed Switch 15. 1515 Distributed Virtual Network (vNetwork) vCentervCenter Standard vSwitch vNetwork & dvSwitch 16. 1616 vDistributed Switch Architecture Control Plane (CP) and Data Plane, or IP Plane are separated. CP, responsible for configuring dvSwitches,dvPortgroups, dvPorts, Uplinks, NICTeaming and so on, and for coordinating the migration of the ports, runs on vCenter DP, responsible for performing the forwarding, runs inside the VMKernel of the ESX/ESXi (vSwitch). vCenter ESX ESX ESX Distributed vSwitch vSwitch vSwitch vSwitch Distributed vSwitch vSwitch Control Plane I/O Plane 17. 1717 vSwitch vs DVSwitch Vs Cisco N1K 17 Capabilities vSwitch dvSwitch Cisco N1K L2 Switch Yes Yes Yes VLAN Segmentation Yes Yes Yes 802.1Q Tagging Yes Yes Yes Link Aggregation Static Static & LACP Static & LACP TX Rate Limiting Yes Yes Yes RX Rate Limiting No Yes Yes Unified Management Interface vSphere Client @Host vSphere Client @Vcenter Cisco CLI PVLAN No Yes Yes Network I/O Control No Yes Yes Port Mirroring No Yes Yes SNMP, Netflow, etc. No Yes Yes Load Based Teaming No Yes No 18. 1818 Network IO Control 19. 1919 Introduction vSphere Network IO Control prioritize network access by continuously monitoring I/O load over the network and dynamically allocating available I/O resources according to specific business needs 20. 2020 NIOC at a Glance Improve and meet service levels for business-critical applications Reduces the amount of active performance management required Bridge virtual and physical infrastructure quality of service with per resource 802.1 tagging Set, view and monitor network resource shares and limits Optimize your workloads Virtualize more types of workloads, including I/O-intensive business-critical applications Ensure that each cloud tenant gets their assigned share of I/O resources Set and enforce network priorities (per VM) across a cluster Increase flexibility and agility of your infrastructure Reduce your need for network interfaces dedicated to a single virtual machine or application Enable multi-tenancy deployments 21. 2121 Features Isolation Shares Limits Load-Based Teaming IEEE 802.1p tagging 22. 2222 Network Traffic Classifications vMotion iSCSI FT logging Management NFS (Network File System) Virtual machine traffic vSphere Replication traffic User Defined 23. 2323 Configuration Best Practices 24. 2424 Choosing the Type of Switch Size of your deployment If you have a small deployment and need basic network connectivity, vSS should be sufficient If you have a large deployment, consider vDS/N1K Organizational If you have a group which controls both VM deployment and network provisioning, then choose vSS/vDS (integrated control via vSphere Client UI) If you have a separate network admin group, trained on Cisco IOS CLI, and wishes to maintain control over virtual and physical networking, then choose N1K Other factors Budget vDS/N1K requires Enterprise+ License Features vSS features are frozen, vDS features are evolving (ask Cisco about N1K) 25. 2525 Configuration Best Practices: #1 Enable on Physical Switch Ports Spanning Tree Protocol- Loop avoidance mechanism PortFast- Fast convergence after failure Link State tracking-Detection of upstream ports(on Cisco switches) Enable BPDU Guard Validate Duplex settings NIC Hardware status Link status Switch Port status Switch Port Configuration Jumbo Frames Configuration Ensure adequate CPU resources are available Heavy gigabit networking loads are CPU-intensive Both native and virtualized 26. 2626 Enabling Jumbo Frame Physical Switches Set MTU to desired value on all switches in the network Virtual Switch For vDS set MTU on UI For vSS, run esxcfg-vswitch m Physical Adapter MTU set automatically as part of vSwitch setting. Check for errors! Virtual Adapter Change vNic MTU inside the guest Run esxcfg-vmknic m to set MTU of vmknic Ping Test Make sure you specify dont fragment 27. 2727 Configuration Best Practices: #2 Use separate Networks to avoid contention For Console OS (host management traffic), VMKernel (VMotion, iSCSI, NFS traffic), and VM For VMs running heavy networking workloads Enable BPDU Guard? With explicit failover, Set Failback = No to avoid the flapping of traffic between two network adapters Tune VM-to-VM networking on same host Use same virtual switch to connect communicating VMs Avoid buffer overflow in guest driver: Tune receive/transmit buffers (Refer KB: 1428) Use vmxnet3 virtual device in guest Default 32-bit guest vNIC is vlance, but vmxnet3 performs better For vmxnet3 driver install tools e1000 is the default for 64-bit guests Enhanced vmxnet3 is available for several guest OSes 28. 2828 Configuration Best Practices: #3 Converge Network and Storage I/O onto 10GE Reduce cabling requirements Simplify management and reduce cost Tools for Traffic Management 1. Traffic Shaping Limit the amount of traffic a vNic may send / receive 2. Network I/O Control (vDS + vSphere 4.1) Isolate different traffic class from each other Each type of traffic is guaranteed a shared of the pNic bandwidth Unused bandwidth are automatically distributed to other traffic types 29. 2929 vCloud Networking and Security Best practices and Troubleshooting Global Support Services 30. 3030 Agenda Best Practices for vCloud Networking and Security ( vCNS ) VXLAN When to use App vs Edge or both Troubleshooting vCNS 31. 3131 Best Practices for vCNS Manager . . . Install on dedicated management cluster Run on ESX host unaffected by downtime Network interfaces placed in common network Backup regularly Ensure NTP is setup and working 31 32. 3232 . . . Best Practices for vCNS Manager Change Admin password after install Create new admin account for CLI Prior to upgrade backup DB and clone / snapshot manager 32 33. 3333 Best Practice for vCNS App FW Deployments Migrate vCenter server / database VMs to alternate ESX server Set unique IP for the management port of each vShield App Install VMware Tools on each VM Use System Status screen to monitor health of a App FW 33 34. 3434 App FW Policy Management . . . Use vCenter containers and security groups for enforcement Use service groups to reduce rules Know when to use General / Ethernet rules Set the Fail Safe Mode to Block Utilize Flow Monitoring 34 35. 3535 . . . App FW Policy Management Create firewall rules allowing access to default services Use different syslog servers for different log levels Use the comments fields Use the Load History option to revert configuration Exclude machines when necessary 35 36. 3636 Virtual eXtensible LAN 37. 3737 VXLAN Setup - Physical Requirements DHCP available on VXLAN transport VLANs Increased MTU needed to accommodate VXLAN encapsulation overhead Leverage 5-tuple hash distribution for uplink and interswitch LACP Multicast routing enabled if traffic is traversing a router 38. 3838 VXLAN Setup - Virtual Requirements vSphere 5 .1 vShield Manager 5 .1 vSphere Distributed Switch 5 .1 .0 Virtual Tunnel End Point (VTEP) 38 39. 3939 VXLAN Implementation 39 40. 4040 When to use vShield Edge or vShield App or both? 40 DMZ Development Finance 41. 4141 vShield App 42. 4242 vShield Edge 43. 4343 vShield App and Edge 44. 4444 Use Case - Securing Business Critical Applications DMZ FinanceDevelopment FinanceDevelopment Solution - vShield App + Edge Protect data and applications with hypervisor level firewall Create and enforce security policies with virtual machine migration Facilitate compliance by monitoring all application traffic Improve performance and scalability with load balancer and software based solution Requirements Deploy production and development applications in a shared infrastructure with: Traffic segmentation between applications Authorized access to applications Strict monitoring and enforcement of rules on inter- VM communications Ability to maintain security policies with VM movement Compliance to various audit requirements VMware vShield App 45. 4545 Multiple sizes (Compact, Large, X-Large) Up to 10 interfaces per vShield Edge DHCP, NAT, and DNS relay Firewall support Load Balancing IPsec and SSL VPN-Plus VXLAN Gateway Routing (static routes) High Availability Flexible IP address management Intuitive deployment workflow CLI vShield Edge Secure the Edge of the Virtual Data Center Tenant A Tenant X Highlights Load balancer firewall VPN 46. 4646 Edge Scalability 46 47. 4747 vShield App Application Protection for Network Based Threats DMZ PCI HIPAA Features Hypervisor-level firewall Inbound, outbound connection control applied at vNIC level Elastic security groups - stretch as virtual machines migrate to new hosts Robust flow monitoring Policy Management Simple and business-relevant policies Managed through UI or REST APIs Logging and auditing based on industry standard syslog format 48. 4848 Troubleshooting vCloud Networking and Security 49. 4949 VXLAN Issue #1 Not Ready Shown in vCNS UI 49 BypassVUMenabled is not set to false in EAM Managed IP is not set in vCenter 50. 5050 Verify VXLAN Agency Settings . . . Access the EAM managed object browser Verify the VXLAN agency has the bypassVumEnabled set to FALSE 50 51. 5151 . . . Verify VXLAN Agency . . . Access vCenter EAM Managed Object Browser http://vcenter51.vmware.local/eam/mob/ 52. 5252 . . . Verify VXLAN Agency Settings . . . 53. 5353 . . . Verify VXLAN Agency Settings . . . 54. 5454 . . . Verify VXLAN Agency Settings 55. 5555 Alter the bypassVumEnabled Setting . . . 55 56. 5656 . . . Alter the bypassVumEnabled Setting . . . Visit the following URL: https:///eam/mob/?moid=agency- 0&method=Update The value will be set to the desired setting true or false Once the XML data is filled in, click the Invoke Method link. 56 57. 5757 . . . Alter the bypassVumEnabled Setting 57 58. 5858 Managed IP not set in vCenter . . . In eam.log you see a smilar error: ('http://vCenter1.vmware.com:80/eam/vib?id=8e840536-1855- 4c7e-81bd-8814b43f8ee0-0', '/tmp/tmpjzGgUU', '[Errno 4] IOError: ') vCenter FQDN is being used which does not work to install VXLAN agent on ESX host 58 59. 5959 . . . Managed IP Not Set in vCenter . . . 60. 6060 . . . Managed IP Not Set in vCenter . . . 61. 6161 . . . Managed IP Not Set in vCenter . . . 62. 6262 . . . Managed IP Not Set in vCenter 62 63. 6363 VXLAN issue #2 class domain-cX already has been configured with mapping Download the curl command if needed from the internet Run the following command on command line curl -i -k -H "Content-type: application/xml" -u admin:default -X DELETE https:///api/2.0/vdn/map/cluster//switches/dvs 63 64. 6464 Edge/App debug packet Enable debug packet mode though the App/Edge CLI debug packet display interface (interface) [Expression] Example vShield# debug packet display interface mgmt host_10.10.11.11_and_port_80 64 65. 6565 Questions ?? 66. 6666 Other VMware Activities Related to This Session HOL: HOL-SDC-1302 vSphere Distributed Switch from A to Z Group Discussions: VSVC1004-GD Top 10 Customer Support Issues with Josh Gray 67. THANK YOU 68. vSphere Networking and vCloud Networking Suite Best Practices and Troubleshooting Richard Cockett, VMware Umesh Goyal, VMware Software India Pvt ltd VSVC5103 #VSVC5103