ucs nirvana - cisco.com · iang’s journey from legacy blade servers to cisco unified computing...
TRANSCRIPT
![Page 1: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/1.jpg)
I A N G ’ S J O U R N E Y F R O M L E G A C Y B L A D E S E R V E R S T O C I S C O U N I F I E D C O M P U T I N G S Y S T E M , W I T H N O ( N O T I C E A B L E ) V M W A R E
D O W N T I M E
UCS Nirvana
Lee BurlesonNetwork and Systems EngineerIowa National Guard
Cisco ARNG Training EventMarch 2011
![Page 2: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/2.jpg)
Disclaimer
This presentation contains information on Iowa National
Guard migration from a legacy blade server environment
to a Cisco UCS system. The content was created for the
audience attending the 5th Annual Training Event and
not for distribution beyond the attendees without the
express permission of the Iowa National Guard .
![Page 3: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/3.jpg)
Introduction
whoami
My credentials
Outline
Can’t do a UCS AO class in 45m
![Page 4: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/4.jpg)
Legacy Environment
2 Dell 1855 chassis
18 ea 1955 Blade servers
Great for IANG when purchased
![Page 5: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/5.jpg)
Legacy Environment, cont.
The aura wears off…
Less-than-ideal chassis management interface
Cabling: power, FC, Ethernet, management, KVM
Design limitations, especially when virtualizing
Module management
Eth, FC
Points of management
![Page 6: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/6.jpg)
Photos
![Page 7: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/7.jpg)
Photos, cont.
![Page 8: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/8.jpg)
Overview and Background
Project goals
Modernize
Improve management
Boot from SAN
Reduce complexity
Increase performance
Blade Chassis Research
Read both vendor-produced AND independent literature
Drank Cisco Kool-Aid @ Minneapolis
Wrote specs (anyone need those?)
Received 2 ea chassis & 8 ea B200M2 blades
![Page 9: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/9.jpg)
UCS Basics
Fabric Interconnects are heart of system (2)
40c x 8b = 320b*
Not just for virtualization, but does it very well
Single and well-done mgmt UI
All config kept in FI – hardware independence
Built-in, transparent fabric redundancy
![Page 10: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/10.jpg)
UCS PhotoFront view
![Page 11: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/11.jpg)
UCS PhotoRear view
![Page 12: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/12.jpg)
![Page 13: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/13.jpg)
Hardware Installation
Chassis are heavy
Very clean cabling
Power cable challenges with our PDUs• Ordered C19-C14 cables
Per chassis:
2 ea 1M cords
2 ea 2M cords
![Page 14: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/14.jpg)
Hardware Installation, cont.
Ordered reduced-length IOM interconnects (SFP-H10GB-CU1M)
Remember: only 1,2,or 4 IOM connections are allowed!
![Page 15: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/15.jpg)
Management
Baby steps
Connected only mgmt ports of both Fabric Interconnects
Must reserve IP space in same subnet for KVM access to blades
Used management interface (Java) to explore and design system without external impact
![Page 16: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/16.jpg)
Management, cont.
Able to create entire system without upstream connectivity or even the blades themselves
Resource pools Server pools
MAC address pools
WWNN pools
KVM IP address pools
vNIC templates
Etc…
Service profile templates
Service profiles (7b+1ub)
![Page 17: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/17.jpg)
Storage
Configure switches based on UCS config
WWNN Aliases + Zoning = Port independence
Enable NPIV (globally + UCS ports)
Connected FIs to FC fabric; verified config
May want to fully alias and zone in both fabrics; depends on your design
No production VMFS presentation yet!
Built initial test ESXi server as SAN-boot POC
![Page 18: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/18.jpg)
Network
Single 1GbE link from FI pair connected upstream -
helped prove failover and identify issues
Spent a while in this state
4 x 10Gb transition was later seamless
Ease of adding VLANs
![Page 19: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/19.jpg)
VMware Integration
Wanted to improve VM I/O, even past basic host
improvements gained with upgrade
Hypervisor bypass/PCI pass-through/VMDirectPath
I thought this was what I wanted, until I tried it
Various painful consequences
Recommendation: don’t do it! (until….)
![Page 20: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/20.jpg)
VMware Integration, cont.
Our host design: 4 network interfaces (vNICs) + 2 FC vHBAs
vNIC0 - vswitch0 – “mgmt” - fabric A/B
vNIC1 - vswitch1 – “vmotion” - fabric B/A
vNIC2 - FI dvs – “vm_Uplink01” - fabric A only
vNIC3 - FI dvs – “vm_Uplink02” - fabric B only
No trunking!
Pass-through switching - VN-Link
vMotion works great
Reduces host CPU, reduces latency, improves I/O
Creates dynamic vNIC## – vm int xx - FI dvs – balanced fabric connections!
![Page 21: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/21.jpg)
VMware Integration, cont.VMware host interfaces, UCS screenshot
![Page 22: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/22.jpg)
VMware Integration, cont.VMware host interfaces, VMware screenshot
![Page 23: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/23.jpg)
VMware Integration, cont.
Created addl cluster for isolation
Joining the datacenter
DVS connection
Different DVSs available: VMware, Nexus 1000V,
Nexus 1010, 6100 FI
![Page 24: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/24.jpg)
VM Migration
The culmination of all our work
Recommend temporarily setting DRS to Partial so you don't fight it
Ran into first problem
![Page 25: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/25.jpg)
VM Migration, cont.Changing the VM Network Connection
Second problem: destination host didn’t have a Port Group for the VM to connect to
![Page 26: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/26.jpg)
VM Migration, cont.Changing the VM Network Connection
![Page 27: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/27.jpg)
VM Migration, cont.Moving vCenter itself
The last challenge, so I thought
Catch 22?
Procedure used: [Bad memory block] Did something bad; I lost my vCenter
connection
Direct-connect to vCenter-owning host
Power down vCenter VM
Remove from Inventory
Direct-connect to UCS crossover host
Import VM, verify network, and power it on
Connect to vCenter and change adapter over to dvs (as in standard procedure)
![Page 28: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/28.jpg)
VM Migration, cont.Moving vCenter itself (also cont.)
Procedure that I should have used:
Migrate vCenter VM to crossover host
Connect to vCenter and change adapter over to dvs (as in
standard procedure)
Still, be careful
![Page 29: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/29.jpg)
VMware Integration, cont.Balanced fabric screenshot
![Page 30: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/30.jpg)
Goals Met?
Definitely modernized
Vastly improved management
100% boot from SAN (even the MSSQL blade)
Reduced complexity
Improved performance:
vMotion SharePoint VM with 16GB RAM
Before: ~3 minutes, After: 40 seconds
Can take down a host running a dozen VMs within 5 minutes
![Page 31: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/31.jpg)
Future
Nexus 5548
Topology considerations - are the 5548s the new de
facto datacenter core with L3?
HP iSCSI gateway for EVA – 10Gb iSCSI for VMs
The Nexus 1000V is no longer needed with
VMWare on UCS (or is it?)
![Page 32: UCS Nirvana - cisco.com · IANG’S JOURNEY FROM LEGACY BLADE SERVERS TO CISCO UNIFIED COMPUTING SYSTEM, WITH NO (NOTICEABLE) VMWARE DOWNTIME UCS Nirvana Lee Burleson Network and](https://reader030.vdocuments.site/reader030/viewer/2022020316/5b3bda677f8b9a26728cf423/html5/thumbnails/32.jpg)
Acknowledgements/Resources
www.bradhedlund.com
blog.scottlowe.com
www.cisco.com/go/ucs
bladesmadesimple.com