cisco hyperflex systems upgrade guide, release 2...cisco hyperflex systems upgrade guide, release...

Cisco HyperFlex Systems Upgrade Guide, Release 2.5First Published: 2017-07-31

Last Modified: 2017-09-08

Americas HeadquartersCisco Systems, Inc.170 West Tasman DriveSan Jose, CA 95134-1706USAhttp://www.cisco.comTel: 408 526-4000

800 553-NETS (6387)Fax: 408 527-0883

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS,INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND,EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITHTHE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY,CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version ofthe UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHERWARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS.CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OFMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUTLIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERSHAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, networktopology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentionaland coincidental.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL:http://www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationshipbetween Cisco and any other company. (1110R)

© 2017 Cisco Systems, Inc. All rights reserved.

http://www.cisco.com/go/trademarks

C O N T E N T S

New and Changed Information 1C H A P T E R 1

New and Changed Information 1

Overview 3C H A P T E R 2

About This Guide 3

Upgrade Modes 3

Upgrade Guidelines for 2.5(1c) 4

Cisco HyperFlex HX-Series System Compatibility Matrix 6

Supported Upgrade Paths 7

Upgrading a 1.7.x HyperFlex Cluster 10

Upgrading SED Ready Systems 10

Prerequisites and Guidelines 11C H A P T E R 3

Overview 11

Prerequisites 11

Cautions and Guidelines 12

Pre-Upgrade Validation Checks 13

HyperFlex Node Upgrade Validations 13

Viewing HyperFlex Cluster Health 14

Verifying If DRS Is Enabled 15

Viewing ESX Agent Manager 15

Verify Health of HyperFlex Cluster In Cisco UCS Manager 15

Verify UCS Server Firmware (C-Bundle) Version 15

Configuring vMotion Interfaces 16

Configure Lenient Mode 16

Cisco HyperFlex Systems Upgrade Guide, Release 2.5iii

Detailed Pre-Upgrade Procedures 19C H A P T E R 4

Overview 19

Downloading Software 19

Test Upstream Network Connectivity 20

Graceful Shutdown of a HX Cluster 21

Modifying Host Firmware Package Using Cisco UCS Manager 21

Bootstrap Process 22

Entering Cisco HyperFlex Maintenance Mode 23

Exiting Cisco HyperFlex Maintenance Mode 24

Upgrade Procedures 27C H A P T E R 5

Online Upgrade Process Workflow 27

Split Upgrade 28

Combined Upgrade 29

Offline Upgrade Process Workflow 29

Start Cluster and Power On VMs 30

Upgrading Cisco HyperFlex Software Components 33C H A P T E R 6

Upgrading Cisco UCS Firmware Using Cisco UCS Manager 33

Upgrading Cisco HX Data Platform 35

Upgrading Cisco UCS Firmware 37

Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware 39

Upgrading ESXi 40

Compute-only Node Upgrade Considerations 42

Upgrade of Compute-only Nodes 42

Upgrading Cisco UCS Firmware on Compute-only Node from Version 2.2(6f) to 3.1(2b) or Later 42

Post Upgrade Tasks 45C H A P T E R 7

Confirm That Upgrade Is Complete 45

Check Firmware Versions in UCSM 46

Verify If Cleaner Is Running 46

Known Issues 49C H A P T E R 8

Cisco HyperFlex Systems Upgrade Guide, Release 2.5iv

Contents

Overview 49

Upgrade Button Missing in Cisco HX Data Platform 49

Cluster Start Command Error: Node Not Available 49

A Node Fails to Upgrade due to vCenter Issues 50

Cisco HyperFlex Systems Upgrade Guide, Release 2.5v

Contents

Cisco HyperFlex Systems Upgrade Guide, Release 2.5vi

Contents

C H A P T E R 1New and Changed Information

• New and Changed Information, on page 1

New and Changed InformationThe following table provides an overview of the new features and changes made to this guide for this currentrelease.

Table 1: New Features in HX Release 2.5

Where DocumentedDate AddedDescriptionUpdate

Upgrade Guidelines for2.5(1c), on page 4

09/08/2017Upgrade guidelines for 1.7.x andSelf-Encryption Drives (SED).

Upgrade Guidelines

Entering CiscoHyperFlexMaintenance Mode, onpage 23

Exiting Cisco HyperFlexMaintenance Mode, onpage 24

07/24/2017Enter and Exit Cisco HyperFlexmaintenance mode through HX Connect.

Entering andExiting HyperFlexMaintenance Mode

Cisco HyperFlex Systems Upgrade Guide, Release 2.51


New and Changed InformationNew and Changed Information

C H A P T E R 2Overview

• About This Guide, on page 3• Upgrade Modes, on page 3• Upgrade Guidelines for 2.5(1c), on page 4• Cisco HyperFlex HX-Series System Compatibility Matrix, on page 6• Supported Upgrade Paths, on page 7• Upgrading a 1.7.x HyperFlex Cluster, on page 10• Upgrading SED Ready Systems, on page 10

About This GuideThis document describes how to upgrade an existing installation of Cisco HX Data Platform .

Cisco HyperFlex Systems has several components that may be upgraded depending on your environment.HyperFlex Systems core components are:

• Cisco UCS server firmware

• Cisco HX Data Platform software

• VMware ESXi software

Upgrade ModesCisco HyperFlex Systems upgrade deployment can be performed in either online or offline mode.

Online Upgrade Mode

Use the online upgrade mode if you cannot tolerate downtime and the maintenance window is permissible.

During an online upgrade, all virtual machines remain in a running state without service disruption. Userintervention is not required after the upgrade starts, so long as DRS is licensed, enabled, and set to automaticmode. Cisco recommends performing the upgrade during a maintenance window whenever possible. Referto the Online Upgrade Process Workflow , on page 27 for more details.


To upgrade a nested VMware vCenter (a vCenter on an HX datastore) use the online upgrade mode only.Note

Offline Upgrade Mode

Use the offline upgrade mode if you desire a shorter time window and workload downtime is acceptable.

Offline upgrade mode means the Cisco HX Data Platform is offline. However, storage controller VMs areonline and HX Data Platform management is viewable through both the stcli command-line access and theVMware vSphere Web Client. The stcli cluster command returns that overall storage cluster status isoffline. Refer to the Offline Upgrade Process Workflow, on page 29 for more details.

Upgrade Guidelines for 2.5(1c)The 2.5(1c) release supports upgrading from older HX versions. Cisco recommends upgrading to 2.5(1c) from2.5(1a) or 2.5(1b) releases.

The list below is a highlight of critical criteria for performing an upgrade of your HyperFlex system.

• vDS Support—Upgrades with vDS environments are supported 2.5(1c).

• Initiating Upgrade―Use either the CLI stcli commands or the HX Data Platform Plug-in to thevSphereWeb Client. Do not use HXConnect or the Tech PreviewUI to upgrade to from a pre-2.5 versionto a 2.5 version.

• Cluster Readiness—Ensure that the cluster is properly bootstrapped and the updated plug-in loadedbefore proceeding. See Bootstrap Process, on page 22 for more details.

• HXData Platform 1.7.x Clusters—Users upgrading from 1.7.x must step through an intermediate versionbefore upgrading to 2.5. See the Upgrading a 1.7.x HyperFlex Cluster, on page 10 for more details.

• HX Data Platform 2.1(1b) with SED—Upgrading SED-ready systems running 2.1 require UCSinfrastructure and server firmware upgrades. See the Upgrading SED Ready Systems, on page 10 formore details.

• vSphere 5.5 Upgrades—Users on vSphere 5.5 must upgrade to 6.0/6.5 before starting HXData Platformupgrade. vSphere 5.5 support is deprecated with HXData Platform 2.5(1a) and upgrade fails if attempted.

• For HX220 users running 5.5, contact TAC for upgrade assistance.

• For HX240 users running 5.5, upgrade components in the following order.

1. Upgrade vCenter to 6.0 or 6.5. If upgrading to 6.5, you must upgrade your vCenter in place.Using a new vCenter 6.5 is not supported for users migrating from 5.5.

2. Upgrade ESXi to 6.0/6.5 using the offline zip bundle. See Upgrading ESXi, on page 40 formore details.

During upgrade it might be necessary to reconnect ESXi host manually in vCenterafter ESXi upgrade and host reboot.

Note


OverviewUpgrade Guidelines for 2.5(1c)

3. Upgrade HX Data Platform to 2.5 (and optionally the UCS firmware). See Upgrading CiscoHX Data Platform, on page 35 and Upgrading Cisco UCS Firmware, on page 37 for moredetails.

• If Upgrading to vSphere 6.5:

• Certain cluster functions such as native and scheduled snapshots, ReadyClones, and Enter/ExitHXMaintenance Mode will not operate from the time the upgrade is started until the HX DataPlatform upgrade to 2.5 is complete.

• After upgrading ESXi using the offline zip bundle, use the ESX Exit MaintenanceMode option.The Exit HX Maintenance Mode option does not operate in the vSphere Web Client until theHX Data Platform upgrade to 2.5 is complete.

• vSphere 6.0 Upgrades—Users on vSphere 6.0 migrating to 6.5, upgrade components in the followingorder:

1. HXData Platform upgrade to 2.5 (and optionally the UCS firmware). See Upgrading Cisco HXDataPlatform, on page 35 and Upgrading Cisco UCS Firmware, on page 37 for more details.

2. Upgrade vCenter Server following VMware documentation and best practices. Optionally, deploya new vCenter server and perform stcli cluster reregister.

3. Upgrade ESXi to 6.5 using the offline zip bundle. See Upgrading ESXi, on page 40 for more details.

• Server Firmware Upgrades—Server firmware should be upgraded to ensure smooth operation and tocorrect known issues. Specifically, newer SAS HBA firmware is available in this release and isrecommended for long-term stability.

• Users are encouraged to upgrade to 3.1(3c) C-bundle or later wheneverpossible. To determine the current C-bundle in use on the cluster, see VerifyUCS Server Firmware (C-Bundle) Version, on page 15 for more details.

• Users running C-bundle versions prior to 3.1(2f) must upgrade serverfirmware by performing a combined upgrade of UCS server firmware(C-bundle) to 3.1(3c) or later and HX Data Platform to 2.5. Do not split theupgrade into two separate operations.

• If the cluster is already on 3.1(2f) C-bundle or later, you may perform anHX Data Platform only or combined upgrade, as required.

Note

• Maintenance Window—If upgrading both HX Data Platform and UCS firmware, either a combined orsplit upgrade can be selected through the vSphere HX Data Platform Plug-in depending on the length ofthe maintenance window. Direct firmware upgrade using server firmware auto install through Cisco UCSManager should not be attempted. Instead, use the UCS server upgrade orchestration framework providedby the HX Data Platform Plug-in.


OverviewUpgrade Guidelines for 2.5(1c)

Cisco HyperFlex HX-Series System Compatibility MatrixSee resolved and open caveats before upgrading. Review the new features, refer to the Cisco HXData PlatformRelease Notes.

For a complete list of hardware and software interdependencies, refer to respective UCSM release version ofHardware and Software Interoperability for Cisco HyperFlex HX-Series.

Supported HyperFlex Software Versions

Recommended UCS FIFirmware

HX Data PlatformHX Data Platform InstallerHyperFlex Release

3.1(3c) required forSED systems andrecommended fornon-SED systems.

2.5(1c)2.5(1c)2.5(1c)


2.5(1b)2.5(1b)2.5(1b)


2.5(1a)2.5(1a)2.5(1a)

3.1(2g)2.1(1c)2.1(1c)2.1(1c)

3.1(2g)2.1(1b)2.1(1b)2.1(1b)

3.1(2f)2.0(1a)2.0(1a)2.0(1a)

3.1(2b)1.8(1f)1.8(1f)1.8(1f)

3.1(2b)1.8(1e)1.8(1e)1.8(1e)

3.1(2b)1.8(1c)1.8(1c)1.8(1c)

3.1(2b)1.8(1b)1.8(1b)1.8(1b)

3.1(2b)1.8(1a)1.8(1a)1.8(1a)

Supported VMware vSphere Versions and Editions

vSphere EditionsvSphere VersionsHyperFlex Version

Enterprise, Enterprise Plus, Standard,Essentials Plus, ROBO

6.0 U1b, 6.0 U2, 6.0 U2 Patch 3, 6.0U2 Patch 4, 6.0 U3, 6.5 Patch 1a

2.5(1c)


OverviewCisco HyperFlex HX-Series System Compatibility Matrix

http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-release-notes-list.html


https://ucshcltool.cloudapps.cisco.com/public/

vSphere EditionsvSphere VersionsHyperFlex Version



2.5(1b)



2.5(1a)


6.0 U1b, 6.0 U2, 6.0 U2 Patch 3, 6.0U2 Patch 4, 6.0 U3.

2.1(1c)

Enterprise, Enterprise Plus5.5 U3


6.0 U1b, 6.0 U2, 6.0 U2 Patch 3, 6.0U2 Patch 4, 6.0 U3

2.1(1b)



6.0 U1b, 6.0 U2, 6.0 U2 Patch 3, 6.0U2 Patch 4

2.0(1a)



6.0 U1b, 6.0 U2, 6.0 U2 Patch 31.8(1f)



6.0 U1b, 6.0 U2, 6.0 U2 Patch 31.8(1e)



6.0 U1b, 6.0 U2, 6.0 U2 Patch 31.8(1c)



6.0 U1b, 6.0 U2, 6.0 U2 Patch 31.8(1b)


6.0 U1b, 6.0 U2, 6.0 U2 Patch 31.8(1a)

Supported Upgrade PathsThe following four table matrices map the supported upgrade paths for Cisco HyperFlex Systems.


OverviewSupported Upgrade Paths

Cisco HX Data Platform Matrix

Supported Upgrade Paths

Desired version

Cisco HX DataPlatform

Current Version

1.8(1f)2.0(1a)2.1(1b)2.5(1c)

———Yes2.5(1a)/2.5(1b)

———Yes2.1(1b)

—YesYesYes1.8.(1f)

YesYesYesYes1.8(1e)

YesYesYesYes1.8(1c)

YesYesYesYes1.8(1b)

YesYesYesYes1.8(1a)

YesYesYes—1.7.x1

1 Direct upgrade from 1.7 to 2.5 is not supported. See Upgrading a 1.7.x HyperFlex Cluster, on page 10for more details.

Cisco UCS Manager Matrix


Desired Version

Cisco UCSManager

CurrentVersion

2.2(7c)3.1(2b)3.1(2c)3.1(2e)3.1(2f)3.1(2g)3.1(3c)

——————Yes3.1(2g)

—————YesYes3.1(2f)

————YesYesYes3.1(2e)

———YesYesYesYes3.1(2c)

——YesYesYesYesYes3.1(2b)

—YesYesYesYesYesYes2.2(7c)

YesYesYesYesYesYesYes2.2(6f)



VMware ESXi Matrix

• ESXi 5.5 support is deprecated with HXDP 2.5.

• If running ESXi 5.5 U3 on HX220, contact TAC for upgrade guidance.

• If running ESXi 5.5 U3 on HX240, see Upgrade Guidelines for 2.5(1c), on page 4 for furtherdetails.

• If you have the ESXi 6.0 U1 version, we recommend an ESXi upgrade. There is a known VMware issuewhere the node becomes unresponsive due to a PSOD and OS crash. See VMware KB article, VMwareESXi 6.0, Patch ESXi600-201608401-BG: Updates esx-base, vsanhealth, vsan VIBs (2145664).


Desired Version

VMware ESXi

Current Version

6.0 U1b6.0 U2 (All PatchLevels)

6.0 U36.5 Patch 1a

———Yes6.0 U3

——YesYes6.0 U2 (All PatchLevels)

—YesYesYes6.0 U1b

YesYesYesYes5.5 U3b (Only onHX240c)

Cisco HX Data Platform Versions Supported by Cisco UCS Manager

Cisco HX Data PlatformCisco UCSManagerVersion

1.7.x1.8.x2.0(1a)2.1(1b)2.5(1c)

—YesYes

Supports hybridand All Flash

Yes


Yes


3.1(3c)

—YesYes


Yes


Yes


3.1(2g)

—YesYes


——3.1(2f)

—YesYes

Supports hybrid

——3.1(2e)



https://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2145664&sliceId=1&docTypeID=DT_KB_1_1&dialogID=308884749&stateId=1%200%20308902416

https://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2145664&sliceId=1&docTypeID=DT_KB_1_1&dialogID=308884749&stateId=1%200%20308902416

Cisco HX Data PlatformCisco UCSManagerVersion

—YesYes

Supports hybrid

——3.1(2c)

—YesYes

Supports hybrid

——3.1(2b)

Yes————2.2(7c)

Yes————2.2(6f)

Upgrading a 1.7.x HyperFlex ClusterClusters running HXDP 1.7.x must first be upgraded to an intermediate release before upgrading to 2.5. Ciscorecommends first upgrading from 1.7 to 2.1(1b) using normal upgrade procedures. After the cluster issuccessfully upgraded and healthy, the cluster may then be upgraded from 2.1(1b) to 2.5(1x).

Upgrading SED Ready SystemsSED-ready systems are HyperFlex clusters running HXDP 2.1(1b) with self encrypting drives (SEDs) installed.UCS Manager must be upgraded to 3.1(3c) or later. In addition, UCS server firmware (C-bundle) must beupgraded to 3.1(3c) or later. Either combined or split upgrade may be used, so long as all the cluster nodesare upgraded prior to enabling key management in HX Connect.


OverviewUpgrading a 1.7.x HyperFlex Cluster

C H A P T E R 3Prerequisites and Guidelines

• Overview, on page 11• Prerequisites, on page 11• Cautions and Guidelines, on page 12• Pre-Upgrade Validation Checks, on page 13

OverviewBefore you upgrade the Cisco HXData Platform and the Cisco UCS server firmware in your Cisco HyperFlexSystem, consider the guidelines, best practices, and recommendations listed in this chapter.

Prerequisites1. See Resolved Caveats andOpen Caveats before upgrading and review theNew Features for this release.

Refer to the latest Cisco HX Data Platform Release Notes.

2. Review supported versions and system requirements. See Hardware and Software Interoperability forCisco HyperFlex HX-Series for more details.

• Verify that you have the latest software bundle versions, review the software versions. Refer to the latestCisco HX Data Platform Release Notes.

• Ensure that the operating systems on all servers have the right driver levels for the release of Cisco UCSto which you plan to upgrade. See Cisco UCSDriver InstallationGuide for identifying the server hardware.

Important

3. Back up the configuration into an All Configuration backup file. See Cisco UCS Manager BackingUp and Restoring the Configuration Guide for the detailed steps.

4. Before you perform firmware updates, use the Cisco UCS Manager image management interface todownload relevant images to the fabric interconnect. Images are stored in bootflash partitions in thefabric interconnect. See Downloading Software, on page 19 for more details.

5. An ESXi upgrade is required from ESX 5.5 U3b and ESXi 6.0 U1. Only VMware vSphere versionsESXi 6.0 U1b, 6.0U2, 6.0 U3, and 6.5 Patch 1a are supported. For ESXi upgrade, refer to UpgradingESXi, on page 40.



http://www.cisco.com/c/en/us/support/servers-unified-computing/unified-computing-system/products-technical-reference-list.html

http://www.cisco.com/c/en/us/support/servers-unified-computing/unified-computing-system/products-technical-reference-list.html


http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-manager/116349-technote-product-00.html#anc12

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/GUI-User-Guides/Admin-Management/3-1/b_Cisco_UCS_Admin_Mgmt_Guide_3_1/b_Cisco_UCS_Admin_Mgmt_Guide_3_1_chapter_01001.html

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/GUI-User-Guides/Admin-Management/3-1/b_Cisco_UCS_Admin_Mgmt_Guide_3_1/b_Cisco_UCS_Admin_Mgmt_Guide_3_1_chapter_01001.html

6. Keep SSH enabled on all ESXi Hosts.

7. Enable vMotion so that the VMs can be moved automatically during the upgrade and MTUs are set asrequired in the environment. See Configuring vMotion Interfaces, on page 16 for details on addingVMkernel interface.

8. If you have reregistered a cluster which has compute nodes, ensure that the compute nodes were re-added.Use the stcli node add command, before you attempt to upgrade the cluster.Example:# stcli node add --node-ips <computeNodeIP> --controller-root-password <ctlvm-pwd>--esx-username <esx-user> --esx-password <esx-pwd>

9. Verify the HyperFlex cluster is healthy. See HyperFlex Node Upgrade Validations, on page 13 formore details.

10. Verify that the cluster is in lenient mode. If not, set the cluster to lenient mode, refer Configure LenientMode, on page 16.

Cautions and GuidelinesBefore you begin upgrade of a Cisco HyperFlex System, consider the following cautions, guidelines, andlimitations.

• Optimizations in Capacity Tier—Backend access is optimized to significantly reduce the magnitude andfrequency of high latency spikes.

This optimization is included automatically for all clusters upgraded to 2.5(1c).

• Upgrade VMware ESXi before starting the upgrade process.

• If you have to upgrade from VMware ESXi version 5.5 U3 on HX220,contact Cisco TAC for assistance.

• For HX240, see Upgrade Guidelines for 2.5(1c), on page 4.

Important

• If you have replication configured, put replication in pause mode prior to performing upgrade or expansionor cluster maintenance. After the upgrade or expansion orcluster maintenance is completed, resumereplication. Perform the pause and resume on any cluster that has replication configured to or from thislocal cluster.

• Cisco recommends using GUI upgrade over CLI for ease of use and better reporting.

• When Upgrade is complete, for each browser interface you use, empty the cache and reload the browserto page to refresh the HX content.

• Ensure that all nodes (including compute nodes) are up and running and the cluster is healthy beforestarting an upgrade or other maintenance activities.

• The Cisco HXData Platform and Cisco UCS firmware bundles must be compatible. Refer UCSHardwareand Software Compatibility Matrix for more details.


Prerequisites and GuidelinesCautions and Guidelines



• For a split upgrade, Cisco HX Data Platform should be updated first before updating the Cisco UCSfirmware.

• During online upgrade, as one node is being upgraded (put into maintenance mode), the number oftolerated node failures is reduced based on the Data Replication Factor and Access Policy settings.

• All endpoints in a Cisco HyperFlex domain must be fully functional and all processes must be completebefore you begin a firmware upgrade on those endpoints. For example, the firmware on a server that hasnot been discovered cannot be upgraded or downgraded. Each endpoint is a component in the CiscoHyperFlex domain that requires firmware to function.

• In a three node cluster, if you shut down one node or put into maintenance mode it makes the clusterunhealthy, but the cluster is still online. If you need to perform manual maintenance, put the hosts inmaintenance mode one at a time and move to the next host only after the cluster is healthy. For HXDPand UCS server firmware upgrades, this process is automatic.

You cannot remove a node from 3 node cluster by doing stcli node removeoperation. To replace a node on a 3 node cluster, please contact Cisco TAC forassistance with the node replacement procedure.

Note

• Firefox browser is not supported due to an outdated version of flash that is bundled with the browser.Manual update of flash within Firefox is possible, but the recommendation is to use either Chrome orInternet Explorer with a modern version of flash.

Pre-Upgrade Validation Checks

HyperFlex Node Upgrade ValidationsPerform the following validations on each HyperFlex node before moving on to upgrade the next node in thecluster.

• Verify that the HyperFlex cluster is healthy and online. Verify all HyperFlex cluster nodes are connectedto the vCenter and are online.

• SSH into each controller VM in the HX cluster and make sure that there is space in /var/stv for theupgrade using the command df -h /var/stv.Output example/dev/sdb1 63G 2.7G 58G 5% /var/stv

If there is not enough space (usage is over 50%), contact Cisco TAC for assistance.Important

• Verify that no major Alarms are reported for the HyperFlex cluster in the vSphere Web Client.

• Verify that DRS is enabled and set to fully automated.

• Verify that vSphere services are running and ESXi Agent Manager (EAM) health is normal.


Prerequisites and GuidelinesPre-Upgrade Validation Checks

• Verify the health of the cluster in Cisco UCS Manager.

Viewing HyperFlex Cluster Health

Using GUI

• From the vSphereWeb Client Navigator, select vCenter Inventory Lists >Cisco HyperFlex Systems >Cisco HX Data Platform > cluster > Summary. View the cluster widget to verify if the HyperFlexcluster is healthy and online.

• From the vSphere Web Client Navigator, select vCenter Inventory Lists > Clusters > cluster >Summary. Verify if all HX Cluster nodes are connected to the vCenter and they are online.

Using CLI

Log in to any controller VM in the storage cluster. Run the command stcli cluster storage-summary

–-detail.

Sample response that indicates the HyperFlex storage cluster is online and healthy.address: 192.168.100.82name: HX-Cluster01state: onlineuptime: 0 days 12 hours 16 minutes 44 secondsactiveNodes: 5 of 5compressionSavings: 78.1228617455deduplicationSavings: 0.0freeCapacity: 38.1ThealingInfo:

inProgress: FalseresiliencyDetails:

current ensemble size:5# of ssd failures before cluster shuts down:3minimum cache copies remaining:3minimum data copies available for some user data:3minimum metadata copies available for cluster metadata:3# of unavailable nodes:0# of nodes failure tolerable for cluster to be available:2health state reason:storage cluster is healthy.# of node failures before cluster shuts down:3# of node failures before cluster goes into readonly:3# of hdd failures tolerable for cluster to be available:2# of node failures before cluster goes to enospace warn trying to move the existing

data:na# of hdd failures before cluster shuts down:3# of hdd failures before cluster goes into readonly:3# of ssd failures before cluster goes into readonly:na# of ssd failures tolerable for cluster to be available:2

resiliencyInfo:messages:

Storage cluster is healthy.state: healthyhddFailuresTolerable: 2nodeFailuresTolerable: 1ssdFailuresTolerable: 2

spaceStatus: normaltotalCapacity: 38.5TtotalSavings: 78.1228617455usedCapacity: 373.3GclusterAccessPolicy: lenient


Prerequisites and GuidelinesViewing HyperFlex Cluster Health

dataReplicationCompliance: compliantdataReplicationFactor: 3

Verifying If DRS Is Enabled

Step 1 From the vSphere Web Client Navigator, select vCenter Inventory Lists > Clusters > cluster > Summary.a)

Verify that DRS is Enabled.

Step 2 Click the vSphere DRS tab.

Check ifMigration Automation Level is set to Fully Automated.

Viewing ESX Agent Manager

From the vSphere Web Client Navigator, select Administration > vCenter Server Extensions > vSphere ESX AgentManager > Summary.

Verify that vSphere services are running and ESX Agent Manager (EAM) health is normal.

Verify Health of HyperFlex Cluster In Cisco UCS Manager

Step 1 Verify if the high availability status of the fabric interconnects shows that both the fabric interconnects are up and running.Step 2 Verify that the data path is up and running. See the CiscoUCSManager FirmwareManagement Guide for more information.Step 3 Verify that the HyperFlex servers have no faults.Step 4 Verify that vNIC faults are cleared to ensure VMware ESXi vSwitch uplinks are up and operational.Step 5 Verify if all servers have been discovered.

Verify UCS Server Firmware (C-Bundle) Version

Using UCS Manager

1. Log in to UCS Manager.

2. Select the Server tab.

3. Select the Host Firmware Package policy by navigating to, Policies > Root > Sub-Organizations ><hx-cluster> > Host Firmware Packages > HyperFlex.

Ensure that you select the desired cluster under the sub-org list.Note


Prerequisites and GuidelinesVerifying If DRS Is Enabled

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/firmware-mgmt/gui/2-5/b_GUI_Firmware_Management_25/b_GUI_Firmware_Management_25_chapter_01011.html

4. Under properties, note the current Rack Package version. It is listed as X.Y(Z)C. For example, 3.1(2g)C.

Using the HX Data Platform Plug-in

1. Navigate to the HX Data Platform Plug-in in the vSphere Web Client.

2. On the summary tab, select Upgrade Cluster.

3. Select the UCS Firmware checkbox only, and click Next.

4. Provide UCS Manager FQDN/IP, username, and password and click Discover.

5. Note the current C-bundle version displayed.

6. Close the upgrade window.

Configuring vMotion InterfacesThe following steps add the VMkernel interface necessary for vMotion to work:

Before you begin

vMotion networking is already pre-defined with a vSwitch created, and the vNICs and VLANs already definedin UCS Manager.

Step 1 Click on the Host in, Inventory >Manage > Networking > VMkernel adapters.Step 2 Click Add Host Networking.Step 3 Select VMkernel Network Adapter.Step 4 Select the existing vmotion vSwitch by selecting browse.Step 5 Provide a name, and refer to table below to enter the appropriate VLAN ID.

VLAN IDCluster Installation Version

0 (default)1.7.x

same as vMotion network1.8.x and later

Step 6 Provide a Static IP Address and complete the wizard.Step 7 (Optional) To use jumbo frames, edit the vmk2 and set theMTU to 9000. Your upstream switch must be configured to

pass jumbo frames on the vMotion VLAN.Step 8 Repeat steps 1 to 5 for all hosts in the cluster.

Configure Lenient ModeCluster access policy is set by default to lenient mode. To manually set the cluster access policy to lenient,use the following procedure.


Prerequisites and GuidelinesConfiguring vMotion Interfaces

Procedure

PurposeCommand or Action

SSH to any one of the controller VMs and login as root.Step 1

Check if lenient mode is already configured.#stcli cluster get-cluster-access-policyStep 2

If set to strict, change to lenient.~/#stcli cluster set-cluster-access-policy --name

lenient

Step 3

If already set to lenient, no further action is required.

Confirm the change.stcli cluster info | grep -i policyStep 4

Example

The following example checks how lenient mode is currently configured. If the lenient mode is setto strict, it sets the mode to lenient, and confirms the change made to lenient mode.

~/#stcli cluster get-cluster-access-policystrict~/#stcli cluster set-cluster-access-policy --name lenientstcli cluster info | grep -i policy


Prerequisites and GuidelinesConfigure Lenient Mode


Prerequisites and GuidelinesConfigure Lenient Mode

C H A P T E R 4Detailed Pre-Upgrade Procedures

• Overview, on page 19• Downloading Software, on page 19• Test Upstream Network Connectivity, on page 20• Graceful Shutdown of a HX Cluster, on page 21• Modifying Host Firmware Package Using Cisco UCS Manager, on page 21• Bootstrap Process, on page 22• Entering Cisco HyperFlex Maintenance Mode, on page 23• Exiting Cisco HyperFlex Maintenance Mode, on page 24

OverviewThis chapter contains a list of procedures required by various upgrade workflows. Follow only the proceduresthat are required for the specific upgrade workflow you intend to use. For step-by-step guidance on onlineand offline upgrades, refer to the Upgrade Procedures, on page 27 chapter.

Downloading SoftwareFor a successful HyperFlex upgrade, the Cisco HyperFlex System component bundles can be downloadedfrom the Cisco HyperFlex Download website:

1. HX Data Platform upgrade bundle (.tgz file)

2. VMware ESXi Offline Zip bundle

3. Cisco UCS infrastructure bundle, blade firmware bundle, and rack firmware bundle.

After the Cisco UCS bundles and firmware are downloaded, they need to be copied to Cisco UCS Manager.

To copy UCS software bundles to Cisco UCS Manager

Step 1 Log in to the Cisco UCS Manager GUI.Step 2 Enter the username and password.Step 3 In the Navigation pane, click the Equipment tab.Step 4 On the Equipment tab, click the Equipment node.


https://software.cisco.com/download/release.html?mdfid=286305544&flowid=79522&softwareid=286305994

Step 5 In the Work pane, click Firmware Management > Installed Firmware > Download Firmware.Step 6 In the Download Firmware dialog box, click the Local File System radio button in the Location of the Image File

field and fill in the required fields.Step 7 In the Filename field, enter the full path and name of the image file.

If you do not know the exact path to the folder where the firmware image file is located, click Browse and navigate tothe file.

Step 8 Click OK. The Cisco UCS Manager GUI begins downloading the firmware bundle to the fabric interconnect.Step 9 Monitor the status of the download on the Download Tasks tab.

If Cisco UCS Manager reports that the bootflash is out of space, delete obsolete bundles on the Packagestab to free up space. To view the available space in bootflash, navigate toEquipment > Fabric Interconnect >Local Storage Information and look in the Work pane area under the General tab.

Note

Step 10 Repeat this task until all the required firmware and bundles are downloaded to the fabric interconnect.

Test Upstream Network ConnectivityEnsure that the hx-storage-data and vMotion upstream switches are configured for Jumbo Frames. Skippingthis step could lead to input/output interruption during Cisco UCS infrastructure upgrade.

Step 1 Put a node in Cisco HX Maintenance mode, refer Entering Cisco HyperFlex Maintenance Mode, on page 23.Step 2 SSH to the ESXi host in step 1.Step 3 Verify that ping is working.

If using Jumbo Frames

vmkping -I vmk1 -d -s 8972 <data IP of address of another host>

If not using Jumbo Frames


Step 4 Swap active interfaces in vswitch-hx-storage-data to force traffic upstream.

esxcli network vswitch standard policy failover set -a vmnic2 -s vminc3 -v vswitch-hx-storage-data

Step 5 Verify that ping is working.

If using Jumbo Frames


If the ping fails, try again with


If not using Jumbo Frames



Detailed Pre-Upgrade ProceduresTest Upstream Network Connectivity

If the ping fails, do not proceed to upgrade the Cisco UCS firmware. Investigate the network configurationincluding upstream switch to identify cause of failure. If you are unable to identify and correct the issue, pleasecontact Cisco TAC for assistance.

Attention

Step 6 Swap interface back to defaults even if ping fails.

esxcli network vswitch standard policy failover set -a vmnic3 -s vminc2 -v vswitch-hx-storage-data

Step 7 Exit the node from Cisco HX Maintenance mode, refer Exiting Cisco HyperFlex Maintenance Mode, on page 24.

Graceful Shutdown of a HX Cluster

Step 1 SSH to any controller VM in the cluster.Step 2 Check cluster health ~#stcli cluster info | grep health.Step 3 If healthy, shutdown the cluster ~#stcli cluster shutdown.Step 4 Shutdown takes a few minutes, wait for the prompt to return.

Modifying Host Firmware Package Using Cisco UCS ManagerHost Firmware Package is set automatically, during Cisco UCS infrastructure upgrade. To manually stagethe correct firmware version before starting the upgrade process use the following procedure.

Step 1 Sign in to Cisco UCS Manager.Step 2 In the Navigation pane, click Servers.Step 3 Expand Servers > Policies > Sub-Organizations > hx-cluster.Step 4 Expand Host Firmware Packages and choose the policy you want to update.Step 5 In the Work pane, click the General tab.Step 6 To modify the components in the host firmware package, clickModify Package Versions.Step 7 Modify value for Blade Package and Rack Package, to the latest firmware version. See HyperFlex Software Versions

for the list of recommended UCS FI firmware.Step 8 In the Excluded Components area, check the boxes corresponding to the components that you want to exclude from

this host firmware package.Step 9 Click OK. Click yes for all the warnings.

What to do next

Verify pending activities.

User acknowledgement of each server is performed automatically during upgrade. Optionally, you canmanuallyacknowledge pending activities on the HyperFlex node.


Detailed Pre-Upgrade ProceduresGraceful Shutdown of a HX Cluster

https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/Cisco_HXDataPlatform_RN_2_0.html#id_13130

On the Cisco UCS Manager toolbar, click Pending Activities.

The User Acknowledged Activities tab lists the HyperFlex nodes that require user acknowledgment withinthe selected sub-org. They are in Pending Reboot status.

Do not acknowledge service profiles.Note

Bootstrap ProcessBootstrap process enables you to upgrade the Cisco HX Data Platform Plug-in.

Perform this procedure on the node that has the Cluster Management IP address.Note

Step 1 From the vSphere Web Client Navigator, select vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HXData Platform > cluster.

Step 2 Navigate to Actions > Summary and note the Cluster Management IP address.Step 3 SSH to the cluster management IP address with root privileges.Step 4 Transfer the latest HX Data Platform upgrade bundle to the controller VM's /tmp directory.

Depending on your operating system, use you can either use SCP directly or download third-party tools, such asWinSCPor MobaXterm.

Step 5 From the controller VM shell, change to the /tmp directory.

Do not use any folder other than /tmp and do not create any subfolders.Warning

Step 6 Un-compress the package using tar –zxvf <storfs package name>.tgz.

Example:tar –zxvf storfs-packages-2.5.1a-26263.tgz

This un-compresses and extracts all files to the root of the /tmp folder.

Step 7 Invoke the cluster-bootstrap.sh script to bootstrap packages for upgrade. Execute the command~# ./cluster-bootstrap.sh

Enter the vCenter FQDN or IP address and administrator level username and password.

Example:root@ucs-stctlvm-101-1:/tmp# ./cluster-bootstrap.shPreparing...Enter vCenter Host: 1.1.1.1Enter vCenter User: [email protected] vCenter password:*******Cluster upgrade bootstrap is required. Updating now...Installing HyperFlex Software Package--------------------------------------Target OS: 12.04


Detailed Pre-Upgrade ProceduresBootstrap Process

BUILD_ID: 26263Release: 2.5.1aTag:Date: 20170719Branch: 2.5Install date: Sat Jul 22 13:32:13 UTC 2017------------------------------------------Updating vCenter plugin ...Updating plugin version for Springpath Storage Manager from 1.8.1.19828 to 2.5.1.26263Updated plugin version to 2.5.1.26263

Bootstrap completed. Please use HX vCenter Plugin UI or CLI to complete the upgrade to "2.5(1c)"releaseHX Connect UI wil not be supported until cluster is fully upgraded to "2.5(1c)" release

Wait for the systemmanagement service to restart and the bootstrap process to complete. Verify if the HXData PlatformPlug-in is now updated.

Step 8 Log out from the cluster management IP controller VM. Make sure to log out of vSphere Web Client.

Do not merely close the browser.Note

Step 9 Log in to vSphere Web Client again to refresh the HX Data Platform Plug-in.Step 10 Verify the plugin version in vCenter by navigating to Administration > Client Plug-Ins > Springpath Plugin in the

vSphere Web Client. Confirm the current version matches the new verison you are upgrading to.

Entering Cisco HyperFlex Maintenance Mode

If you have replication configured, put replication in pause mode prior to performing upgrade or expansionor cluster maintenance. After the upgrade or expansion or cluster maintenance is completed, resume replication.Perform the pause and resume on any cluster that has replication configured to or from this local cluster.

If the HX Storage Cluster is offline, use the VMware Maintenance Mode.

Note

Using the HX Connect User Interface

Only supported on release 2.5(1a)/2.5(1b) and later releases.Note

1. Log in to HX Connect: https://<cluster management ip>.

2. In the menu, click System Information.

3. Click Nodes, and then click the row of the node you want to put in to maintenance mode.

4. Click Enter HX Maintenance Mode.

5. In the Confirm HX Maintenance Mode dialog box, click Enter HX Maintenance Mode.


Detailed Pre-Upgrade ProceduresEntering Cisco HyperFlex Maintenance Mode

After you complete any maintenance tasks, you must manually exit HX maintenance mode.Note

Using the vSphere Web Client

1. Log in to the vSphere web client.

2. Go to Home > Hosts and Clusters.

3. Expand the Datacenter that contains the HX Cluster.

4. Expand the HX Cluster and select the node.

5. Right-click the node and select Cisco HX Maintenance Mode > Enter HX Maintenance Mode.

Using the Command-Line Interface

1. Log in to the storage controller cluster command line as a user with root privileges.

2. Move the node into HX Maintenance Mode.

1. Identify the node ID and IP address.

# stcli node list --summary

2. Enter the node into HX Maintenance Mode.

# stcli node maintenanceMode (--id ID | --ip IP Address) --mode enter

(see alsostcli node maintenanceMode --help)

3. Log in to the ESXi command line of this node as a user with root privileges.

4. Verify that the node has entered HX Maintenance Mode.

# esxcli system maintenanceMode get

You can monitor the progress of the Enter Maintenance Mode task in vSphere Web Client, under theMonitor > Tasks tab.

If the operation fails, an error message displays. Try to fix the underlying problem and attempt to entermaintenance mode again. If you are unable to resolve the problem, please contact Cisco TAC.

Exiting Cisco HyperFlex Maintenance Mode

If you have replication configured, put replication in pause mode prior to performing upgrade or expansionor cluster maintenance. After the upgrade or expansion orcluster maintenance is completed, resume replication.Perform the pause and resume on any cluster that has replication configured to or from this local cluster.

Note


Detailed Pre-Upgrade ProceduresExiting Cisco HyperFlex Maintenance Mode

Using the HX Connect User Interface

Only supported on release 2.5(1a)/2.5(1b) and later releases.Note

1. Log in to HX Connect: https://<cluster management ip>.

2. In the menu, click System Information.

3. Click Nodes, and then click the row of the node you want to remove from maintenance mode.

4. Click Exit HX Maintenance Mode.

Using the vSphere Web Client

1. Log in to the vSphere web client.

2. Go to Home > Hosts and Clusters.

3. Expand the Datacenter that contains the HX Cluster.

4. Expand the HX Cluster and select the node.

5. Right-click the node and select Cisco HX Maintenance Mode > Exit HX Maintenance Mode.

Using the Command-Line Interface

1. Log in to the storage controller cluster command line as a user with root privileges.

2. Exit the node out of HX Maintenance Mode.

1. Identify the node ID and IP address.

# stcli node list --summary

2. Exit the node out of HX Maintenance Mode.

# stcli node maintenanceMode (--id ID | --ip IP Address) --mode exit

(see alsostcli node maintenanceMode --help)

3. Log in to the ESXi command line of this node as a user with root privileges.

4. Verify that the node has exited HX Maintenance Mode.

# esxcli system maintenanceMode get

You canmonitor the progress of theExitMaintenanceMode task in vSphereWeb Client, under theMonitor >Tasks tab.

If the operation fails, an error message displays. Try to fix the underlying problem and attempt to exitmaintenance mode again. If you are unable to resolve the problem, please contact Cisco TAC.



C H A P T E R 5Upgrade Procedures

• Online Upgrade Process Workflow , on page 27• Offline Upgrade Process Workflow, on page 29

Online Upgrade Process WorkflowFirst upgrade Cisco UCS infrastructure to the latest version and then use the automated upgrade workflowfor a combined upgrade of Cisco UCS firmware and Cisco HX Data Platform. Online upgrade uses hostfirmware packages to upgrade all server endpoints.

During online upgrade, as one node is being upgraded (placed into maintenancemode), the number of toleratednode failures is reduced based on the Data Replication Factor and Access Policy settings.

If upgrading both HXDP and UCS firmware, either a combined or split upgrade can be selected through thevSphere Web Client plugin depending on the length of the maintenance window. Direct firmware upgradeusing server firmware auto install through Cisco UCSManager should not be attempted. Instead, use the UCSserver upgrade orchestration framework provided by the HyperFlex Plugin.

Do not use Firefox browser. It is not supported due to an outdated version of flash that is bundled with thebrowser.

Caution

1. If UCSM (A-bundle) or UCS Server Firmware (C-bundle) upgrade is required, download Cisco UCSInfrastructure A, blade bundle B, and rack bundle C. See Downloading Software, on page 19.

2. Ensure that the hx-storage-data and vMotion upstream switches are configured for full network failovercapability before proceeding forward. Otherwise the HyperFlex Cluster becomes offline and all datastoresunmount from the ESXi hosts. See Test Upstream Network Connectivity, on page 20 for more details.

3. Upgrade Cisco UCS Infrastructure as required. See Upgrading Cisco UCS Firmware Using Cisco UCSManager, on page 33.

4. Bootstrap to upgrade Cisco HX Data Platform Plug-in. See Bootstrap Process, on page 22.

• Be sure to copy the bootstrap file to the controller VM /tmp directory.

• Ensure you confirm the version of the plugin in the vCenter Administration > Client Plug-Ins page.

Important


5. To disable snapshot schedule, on the bootstrapped storage controller VM, run the command stcli

snapshot-schedule --disable.

It is enough to run this script on one of the controller nodes.

6. Log in to the vSphere Web Client Plug-in with administrator credentials.

7. Start combined upgrade of HX Data Platform and Cisco UCS firmware in the vSphere Web Client. SeeCombined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware, on page 39.

To upgrade HXData Platform only or Cisco UCS firmware only. See Upgrading Cisco HXData Platform,on page 35 and Upgrading Cisco UCS Firmware, on page 37 for more details.

When only Cisco UCS firmware is being upgraded, you may see the upgrade process pause at the validationscreen, after the fabric interconnect discovery. Most likely this occurs due to network connectivity failure.Please contact Cisco TAC for help.

Note

8. Confirm that upgrade is complete. See Post Upgrade Tasks, on page 45.

9. On the same controller VM, to enable snapshot schedule, run the command stcli snapshot-schedule

-–enable.

Split Upgrade

The recommended upgrade method is to upgrade Cisco HX Data Platform first and then upgrade Cisco UCSfirmware.

Attention

If you prefer to upgrade Cisco UCS firmware first, before upgrading the HX Data Platform do the following:

1. Bootstrap CIP-M using the command ~# ./cluster-bootstrap.sh.

2. Bootstrap all other nodes using ./cluster-bootstrap.sh -n.

3. Begin Cisco UCS firmware only upgrade.

4. After UCS firmware upgrade is complete, begin Cisco HX Data Platform upgrade.

Upgrade Cisco HX Data Platform Only Using GUI

Start upgrade of HX Data Platform only. See Upgrading Cisco HX Data Platform, on page 35.

Upgrade Cisco UCS Firmware Only Using GUI

Start upgrade of UCS firmware only. See Upgrading Cisco UCS Firmware, on page 37.

Upgrade Cisco HX Data Platform Only Using CLI

stcli cluster upgrade --components hxdp --location/tmp/<storfs package name> --vcenter-user <vcuser>

Example:


Upgrade ProceduresSplit Upgrade

stcli cluster upgrade --components hxdp --location/tmp/storfs-packages-1.8.1c-19694.tgz --vcenter-user [email protected]

Upgrade Cisco UCS Firmware Only Using CLI

From the same controller VM, use the command:stcli cluster upgrade --components ucs-fw --ucsm-host eng-fi1--ucsm-user <UCSM User> --ucsm-pwd <UCSM Password> --ucsfw-version <UCSM Firmware Version>

Example:stcli cluster upgrade --components ucs-fw --ucsm-host eng-fi1--ucsm-user admin --ucsm-pwd admin --ucsfw-version '3.1(2b)'

Combined Upgrade

Combined Upgrade Using GUI

Start combined upgrade of HX Data Platform and Cisco UCS firmware. See Combined Upgrade of Cisco HXData Platform and Cisco UCS Firmware, on page 39.

Combined Upgrade Using CLI

From the same controller VM, use the command:# stcli cluster upgrade --components hxdp,ucs-fw --location/tmp/<storfs package name> --vcenter-user <vcuser> --ucsm-host <IP/FQDN of UCSM>--ucsm-user <UCSM User> --ucsfw-version <UCSM Firmware Version>

Example:root@ucs-stctlvm-357-1:~# stcli cluster upgrade --components hxdp,ucs-fw --location/tmp/storfs-packages-1.8.1c-19712.tgz --vcenter-user [email protected] eng-fi16.eng.storvisor.com --ucsm-user admin --ucsfw-version '3.1(2b)'

Offline Upgrade Process Workflow1. If UCSM (A-bundle) or UCS Server Firmware (C-bundle) upgrade is required, download Cisco UCS

Infrastructure A, blade bundle B, and rack bundle C. See Downloading Software, on page 19.

2. Ensure that the hx-storage-data and vMotion upstream switches are configured for full network failovercapability before proceeding forward. Otherwise the HyperFlex Cluster becomes offline and all datastoresunmount from the ESXi hosts. See Test Upstream Network Connectivity, on page 20 for more details.

3. Upgrade Cisco UCS Infrastructure as required. See Upgrading Cisco UCS Firmware Using Cisco UCSManager, on page 33.

4. Launch the vSphereWeb Client and power down all user VMs residing on HX servers and all user VMsrunning on HX datastores. This includes VMs running on compute-only nodes. After the VMs havebeen shut down, verify the health state of the cluster and perform a graceful shutdown. See GracefulShutdown of a HX Cluster, on page 21 for more details.

HyperFlex controller VMs (stCtlVMs) must remain powered on.Important


Upgrade ProceduresCombined Upgrade

5. To manually stage the correct firmware version before starting the upgrade process, refer ModifyingHost Firmware Package Using Cisco UCS Manager, on page 21.

6. Shutdown the HyperFlex Controller VM's (stCtlVMs). In vCenter, right-click on each HX ControllerVM (stCtlVM) and select Power > Shut Down Guest OS.

Once the Controller VM's are shutdown, place the ESXi hosts into Maintenance Mode. In vCenter,right-click on each ESXi host selectMaintenance Mode > Enter Maintenance Mode.

7. Acknowledge the pending reboot on the servers that comprise your HX cluster nodes, including bothconverged nodes and compute-only nodes connected to the cluster. Wait until all nodes are upgraded.Confirm that correct firmware packages have been installed before proceeding.

8. Once the ESXi hosts have booted, take them out of Maintenance Mode. Now the controller VM shouldcome back online. In vCenter, right-click on each ESXi host select Maintenance Mode > ExitMaintenance Mode.

9. Bootstrap to upgrade the Cisco HX Data Platform Plug-in. See Bootstrap Process, on page 22.

• Be sure to copy the bootstrap file to the controller VM /tmp directory.

• Ensure you confirm the version of the plugin in the vCenter Administration > Client Plug-Ins page.

Important

10. To disable snapshot schedule, on the bootstrapped storage controller VM, run the command stcli

snapshot-schedule --disable.

It is enough to run this script on one of the controller nodes.

11. From the same controller VM, begin the upgrade. See Upgrading Cisco HX Data Platform, on page 35for more details.

12. Confirm that the upgrade is complete. See Post Upgrade Tasks, on page 45.

13. After the upgrade is complete, start the cluster and power on VMs. See Start Cluster and Power OnVMs , on page 30.

14. On the same controller VM, to enable snapshot schedule, run the command stcli snapshot-schedule

-–enable.

Start Cluster and Power On VMsAfter the upgrade is complete and the cluster has been upgraded, log out and log back in to vCenter to seeupgrade changes.

Step 1 After the upgrade is complete, start your cluster.Step 2 Login to any controller VM through SSH: .

# stcli cluster start

Example:HyperFlex StorageController 1.8(1c)Last login: Wed Sept 21 23:54:23 2016 from pguo-dev.eng.storvisor.com


Upgrade ProceduresStart Cluster and Power On VMs

root@ucs-stclivm - 384 -1;~# stcli cluster upgrade-statusCluster upgrade succeeded. Cluster version: 1.8(1c)root@ucs-stctlvm-384;~# stcli cluster startwaiting for Cluster to start on nodes: [ucs-383, ucs-384, ucs-385, ucs-386]

This will start the cluster and mount the HX datastores. Wait for cluster to come online. You will see the prompt:Started cluster on nodes; [ucs-383, ucs-384, ucs-385, ucs-386]Cluster is onlineroot@ucs-stctlvm-384-1;~#

Step 3 Wait for cluster to become healthy before starting the VMs. Run command:

~# stcli clustr info|grep health

Example:

root@SpringpathControllerZRVF040451;~# stcli cluster info | grep healthhealthState: healthystate: healthystorage cluster is healthy

Step 4 After the cluster is healthy, launch vSphere Web Client or Thick Client, navigate to Hosts and Cluster > Datacenter >Cluster > . Right click, select Power> Power On to start the VMs.



C H A P T E R 6Upgrading Cisco HyperFlex SoftwareComponents

• Upgrading Cisco UCS Firmware Using Cisco UCS Manager, on page 33• Upgrading Cisco HX Data Platform, on page 35• Upgrading Cisco UCS Firmware, on page 37• Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware, on page 39• Upgrading ESXi, on page 40• Compute-only Node Upgrade Considerations, on page 42

Upgrading Cisco UCS Firmware Using Cisco UCS ManagerTheUCS infrastructure firmware includes UCSManager, switch firmware, blade chassis IOmodule firmwareand rack server FEX firmware. The UCS infrastructure firmware can be upgraded with no disruption to theHyperFlex workloads. This is achieved by doing a rolling upgrade of the UCS fabric interconnects. The HXnodes are configured to failover all Ethernet traffic when a UCS fabric interconnect is rebooted to upgradethe firmware.

Ensure that the hx-storage-data and vMotion upstream switches are configured for Jumbo Frames beforeproceeding forward, otherwise the HyperFlex Cluster will become offline and all datastore will unmount fromthe ESXi hosts.

Note

Step 1 Open the UCS Manager GUI.Step 2 Select Equipment > Firmware Management > Firmware auto-install.Step 3 Click Install Infrastructure Firmware.Step 4 Select the desired UCS infrastructure version. Refer the compatibility matrix to identify the version desired for your

use case. Click Next.

A pre-requisite dialog box will pop-up. You will see a list of warnings. Fix them first before proceeding further.

1. Cisco UCS Manager upgrade validation fails.

In that case, stop the upgrade process. Determine the reason for upgrade validation failure. Take corrective actionand restart the upgrade process.


2. Cisco UCS Manager GUI is disconnected.

This is expected during UCSManager upgrade as UCSManager is stopped, then restarted at the new version. Waituntil UCS Manager goes back online. Log back in to UCS Manager to complete the next steps.

You may check Ignore All box if the warnings are not critical to user environment.

Step 5 Check Upgrade Now box.Step 6 Click Finish.Step 7 Wait for IOM to be upgraded (if blade chassis are present)

1. Select Equipment > Installed Firmware, expand each chassis and check the Update Status of the IO Module.

2. During upgrade, the Update Status of the IO Modules will be Upgrading.

3. When IOM upgrade has completed, the Update Status of the IO Modules is set to Ready.

Step 8 Wait for Subordinate FI to be activated.

1. Select Equipment > Installed Firmware > Fabric Interconnects.

2. Check theActivate Status of the kernel and switch images. During upgrade, the Activate Status is set toActivating.

Step 9 During FI reboot, all HX traffic will be forwarded to primary FI (based on ESXi vSwitch failover policy). This willcause a brief traffic interruption. This will not cause storage IO failures.

Step 10 Verify subordinate FI has rebooted and joined the UCS cluster.


2. After activation, the Activate Status of the FI is set to Ready.

3. Check the Overall Status of the FI is operable.

4. Check the kernel and switch versions of the FI match the desired and updated version.

5. Check the FI has no fault.

6. Check the FI cluster membership is Subordinate.

Step 11 Wait for IOM activation to complete.

1. Select Equipment > Blade Chassis > IO Module.

2. Wait for the Activate Status of IO module to change to Ready.

Step 12 Wait until HX traffic is re-pinned to both FIs.

Wait for UCS Manager vNIC faults to be cleared. The fault clearing indicates ESXi has loaded the ENIC driver andthe interface is up. The traffic is not re-pinned immediately when the network interface goes up because ESXi has afail back timer. But the Net.teampolicyupdelay timer is very low by default (100ms).

Step 13 Verify the HX Cluster is online, and healthy before rebooting the primary fabric interconnect.

1. Access summary tab from the vSphere Web Client Navigator. Select Home > vCenter Inventory Lists > CiscoHyperFlex Systems > Cisco HX Data Platform > cluster > Summary .


Upgrading Cisco HyperFlex Software ComponentsUpgrading Cisco UCS Firmware Using Cisco UCS Manager

Step 14 In the UCS manager GUI, on the toolbar, click Pending Activities. Click on Fabric Interconnects tab that displaythe tasks requiring user acknowledgment before they can complete.

1. Check the box in the Reboot Now for each pending activity that you want to deploy immediately.

2. ClickOK. Cisco UCS Manager immediately reboots the primary FI. This will cause the subordinate FI to becomeprimary (FI failover).

Step 15 During FI reboot, all HX traffic will be forwarded to the new primary FI. This will cause a brief traffic interruption.However, this will not cause storage IO failures.

Step 16 Wait for UCS Manager to be disconnected, then reconnected on the other FI. This is because a UCS Manager fail overoccurs due to primary FI reboot.

Check subordinate FI has become primary.

Step 17 Check subordinate FI has become primary.

Check that the FI cluster membership is Primary.

Step 18 Wait for FI to be activated.


2. Wait for the Activate Status of the FI to be Ready.

3. Check the Overall Status of FI is operable.

4. Check the FI has no fault.

Step 19 Verify FI has rebooted and joined the UCS cluster as subordinate.

Check that the FI cluster membership is Subordinate.

Step 20 Wait for IOM activation to completea) Select Equipment > Blade Chassis > IO Module.b) Wait for the Activate Status of the IP module to be Ready.c) You can monitor the status on the FSM tab.

You will lose connectivity to UCS Manager throughout the entire upgrade. This is a normal behavior.Note

Step 21 Wait until HX traffic is re-pinned to both FIs.

In the UCS manager GUI, wait until all server vNIC faults have been cleared.

Step 22 Verify the HX Cluster is online, and healthy after rebooting the FI.

Access summary tab from the vSphere Web Client Navigator. Select Home > vCenter Inventory Lists > CiscoHyperFlex Systems > Cisco HX Data Platform > cluster > Summary .

Upgrading Cisco HX Data PlatformUpgrading the HX Data Platform does not cause any disruption to the workloads running on the HyperFlexCluster.


Upgrading Cisco HyperFlex Software ComponentsUpgrading Cisco HX Data Platform

The upgrade process proceeds as follows:

• HyperFlex cluster readiness for upgrade is checked.

• One by one, the HX nodes enter maintenance mode.

• HyperFlex vSphere Installation Bundles on Hypervisor are upgraded.

• Cisco HX Data Platform is upgraded.

• HX node exits maintenance mode.

• Cluster will begin to rebuild back to full health.

• Once the cluster is healthy, upgrade moves on to the next node in the HyperFlex cluster.

Before you begin

• Complete pre-upgrade validation checks. Refer to Upgrade Prerequisites in Chapter 3.

• Download the latest Cisco HX Data Platform upgrade bundle for upgrading existing clusters, fromDownload Software.

• Complete steps 1 to 6 in theOnline Upgrade Process Work flow. See Online Upgrade Process Workflowfor more details.

• Upgrade Cisco UCS Infrastructure.

• Bootstrap to upgrade Cisco HX Data Platform plug-in.

• Disable snapshot schedule, on the bootstrapped storage controller VM.

• Log in to the Cisco HX Data Platform plug-in with administrator credentials.

• If DRS is Enabled, the VMs are automatically vMotioned to other hosts.

If DRS isDisabled, vMotion the VMs manually to continue the upgrade process.For more information, see VMware Documentation for Migration with vMotion.

Note

Step 1 From the vSphere Web Client Navigator, select vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HXData Platform > HX-Cluster > Summary.

Step 2 Select Upgrade Cluster.Step 3 Select only, HX Data Platform.Step 4 Navigate to the .tgz package file on your local PC.

Upload the Cisco HX Data Platform upgrade bundle. This is the same .tgz package file that was used to upgrade the HXData Platform Plug-in.

Step 5 Enter administrator level vCenter credentials.

(Optional) Enter the MD5 Checksum # information under Advanced Options. The file checksum can be found on theCisco.com download page by clicking on the download title to reveal the md5 checksum.


Upgrading Cisco HyperFlex Software ComponentsUpgrading Cisco HX Data Platform

https://software.cisco.com/download/release.html?mdfid=286305544&softwareid=286305994&release=1.8(1a)&relind=AVAILABLE&rellifecycle=&reltype=latest&i=rm

https://software.cisco.com/download/release.html?mdfid=286305544&softwareid=286305994&release=2.1(1b)&relind=AVAILABLE&rellifecycle=&reltype=latest

This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle.

Step 6 Click Upgrade.The Validation screen shows the progress of checks performed. Fix validation errors, if any.

What to do next

Confirm that the upgrade is complete. See Post Upgrade Tasks for more details.

Upgrading Cisco UCS FirmwareCombined upgrade does not cause any disruption to the workloads running on the HyperFlex Cluster.

Do not manually acknowledge the pending activities in Cisco UCS Manager.Important




• HyperFlex vSphere Installation Bundles (VIBs) on Hypervisor are upgraded.

• The HX Data Platform requests Cisco UCS Manager to begin firmware upgrade. This process can takeup to 1 hour.

You can monitor the progress in the Cisco UCS Manager GUI, under the FSMtab for the service profile.

Note




Make sure that the HX Data Platform is already upgraded. In case of a split upgrade, Cisco UCS Firmwareshould be upgraded around the same time as the HX Data Platform upgrade.

Important

Before you begin

• Complete pre-upgrade validation checks. See Prerequisites for more details.



Upgrading Cisco HyperFlex Software ComponentsUpgrading Cisco UCS Firmware









Note


Step 2 Select Upgrade Cluster.Step 3 Select only, UCS Firmware. Click Next.Step 4 Enter administrator level UCS Manager credentials.

DataField

For example: eng-fi12.eng.storvisor.comUCS Manager Host Name

<admin> usernameUser Name

<root> passwordPassword

Step 5 Click Discover to view the current firmware package version.Step 6 Type in the exact, latest version of Cisco UCS firmware in the Target version field.Step 7 Click Upgrade.

The Cisco UCS servers are now upgraded with the desired firmware packages. The pending activities will be automaticallyacknowledged in a rolling fashion.

You can monitor the progress in the Cisco UCS Manager GUI, under the FSM tab for the service profile.Note

The Validation screen shows the progress of checks performed. Fix validation errors, if any.

What to do next



Upgrading Cisco HyperFlex Software ComponentsUpgrading Cisco UCS Firmware

Combined Upgrade of Cisco HX Data Platform and Cisco UCSFirmware

Combined upgrade does not cause any disruption to the workloads running on the HyperFlex Cluster.

Do not acknowledge the pending activities in Cisco UCS Manager.Important




• HyperFlex vSphere Installation Bundles (VIBs) on Hypervisor are upgraded.

• Cisco HX Data Platform is upgraded followed by a reboot of the host server.

• The HX Data Platform requests Cisco UCS Manager to begin firmware upgrade. This process can takeup to 1 hour.




Before you begin

• Complete pre-upgrade validation checks. See Prerequisites for more details.









Note


Upgrading Cisco HyperFlex Software ComponentsCombined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware



Step 2 Select Upgrade Cluster.Step 3 Select both, HX Data Platform and UCS Firmware. Click Next.Step 4 Navigate to the .tgz package file on your local PC.

Upload the Cisco HX Data Platform upgrade bundle. This is the same .tgz package file that was used to upgrade the HXData Platform Plug-in.

Step 5 Enter administrator level vCenter credentials.

(Optional) Enter the MD5 Checksum # information under Advanced Options. The file checksum can be found on theCisco.com download page by clicking on the download title to reveal the md5 checksum.

This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle.

Step 6 Enter administrator level UCS Manager credentials.Step 7 Click Discover to view the current firmware package version.Step 8 Type in the exact, latest version of Cisco UCS firmware in the Target version field.Step 9 Click Upgrade.

The Cisco UCS servers are now upgraded with the desired firmware packages. The pending activities will be automaticallyacknowledged in a rolling fashion.

You can monitor the progress in the Cisco UCS Manager GUI, under the FSM tab for the service profile.Note

The Validation screen shows the progress of checks performed. Fix validation errors, if any.

What to do next


Upgrading ESXi

Using VMware Update Manager (VUM) to upgrade ESXi is discouraged. If you are using VUM to upgradeESXi, use VUM one host at a time. Make sure that the cluster is in healthy state before moving on to the nextnode. Do not use VUM to upgrade ESXi across a cluster, as there is no guarantee that the cluster will behealthy by the time VUM moves on to the next node.

Caution

The ESXi hypervisor version can be upgraded with no disruption to the HyperFlex cluster workload. This isachieved by performing an online rolling upgrade of each node in the HX cluster.


Upgrading Cisco HyperFlex Software ComponentsUpgrading ESXi

https://software.cisco.com/download/release.html?mdfid=286305544&softwareid=286305994&release=2.1(1b)&relind=AVAILABLE&rellifecycle=&reltype=latest

• ESXi upgrade requires a manual online upgrade.

• When upgrading VMware ESXi from 5.5 U3b through any version up to 6.0 U2, please contact CiscoTAC.

• Use the ESXi command line interface esxcli for upgrading or updating ESXi.

• Replace the build numbers provided in the examples below with the latest version.

Important

Before you begin

• Complete pre-upgrade validation checks. See upgrade prerequisites in Chapter 3.

• Ensure that you upgrade vCenter to a compatible version before beginning ESXi upgrades on the hosts.

Step 1 Download ESXi upgrade package. When upgrading ESXi from 6.0 Ux to any newer version, use the offline zip filefrom Download Software.

Example filename: HX-Vmware-ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip

Do not use the HX ISO file or any other VMware ISO to attempt an ESXi upgrade.Attention

Step 2 Select one of the hosts and put it in HXmaintenancemode using the vSphereWeb Client, refer Entering Cisco HyperFlexMaintenance Mode, on page 23. After the host enters maintenance mode, complete the following steps.

Step 3 Remote secure copy the ESXi upgrade bundle to an appropriate folder with sufficient space.

To copy files using SCP, start the SSH service in the destination ESXi hosts as well.

• On HX240, you can use the local SpringpathDS datastore or a mounted HX datastore.

• On HX220, you can use either a mounted HX datastore or create a temporary RAM disk.

Note

scp local_filename user@server:/path/where/file/should/go

Step 4 Log in to ESXi, and execute the following command to query the list of available image profiles and for profile nameverification.

esxcli software sources profile list -d <location_of_the_esxi_zip_bundle_on_the_datastore>

Full path must be used when running the esxcli software command.Attention

Example:[root@HX-ESXi-01:/vmfs/volumes/1a234567-89bc1234] esxcli software profile update -d/vmfs/volumes/1a234567-89bc1234/HX-Vmware-ESXi-650-5224529-Cisco-Custom-6.5.0.3-Bundle.zip -pVmware-ESXi-6.5.0-HX-5224529-Custom-Cisco-6.5.0.3

Name Vendor Acceptance Level---------------------------------------- --------- -------------------Vmware-ESXi-60U2-4192238-Cisco-Custom-HX Cisco PartnerSupported

Step 5 Run the following command to perform the upgrade.esxcli software profile update -d <path_to_profile_ZIP_file> -p < profile name>


Upgrading Cisco HyperFlex Software ComponentsUpgrading ESXi

https://software.cisco.com/download/release.html?mdfid=286305544&softwareid=286305994&release=2.1(1c)&relind=AVAILABLE&rellifecycle=&reltype=latest

Example:[root@HX-ESXi-01:/vmfs/volumes/1a234567-89bc1234] esxcli software profile update -d/vmfs/volumes/1a234567-89bc1234/HX-Vmware-ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip -pVmware-ESXi-60U2-4192238-Cisco-Custom-HX

Step 6 Once the upgrade completes, restart the ESXi host by issuing the command reboot.Step 7 After the ESXi host comes up, verify that the host has booted up with the correct version.

vmware -vl

Step 8 Wait for the ESXi host to auto reconnect to vCenter. In some upgrade scenarios it may be necessary to force ESXi toreconnect from vCenter. Right-click on the host and select Connection > Connect.

Step 9 Exit maintenance mode using the vSphere Web Client, refer Exiting Cisco HyperFlex Maintenance Mode, on page 24.Step 10 Ensure that the cluster becomes healthy between each ESXi upgrade.

stcli cluster storage-summary --detail

Step 11 Repeat this process for all hosts in the cluster in a sequence.

Make sure that the cluster becomes healthy between each ESXi upgrade.Note

Compute-only Node Upgrade Considerations

Upgrade of Compute-only NodesThe upgrade deployment can be done either in online of offline mode. For a successful upgrade, you mustunderstand the upgrade options and the sequence of tasks that impact the upgrade process.

Compute-only Node Upgrade from version 2.2(7c) to 3.1(2b)

If you are upgrading your compute-only node from 2.2(7c)/1.7x to 3.1(2b)/1.8(1c), follow the non-disruptiveonline rolling upgrade process as described in the Online Upgrade Process Workflow , on page 27.

The new HXData Platform Upgrade workflow will automatically upgrade both compute-only node firmwareand HX Data Platform components as part of the full upgrade.

For offline upgrade, refer Offline Upgrade Process Workflow, on page 29.

Upgrading Cisco UCS Firmware on Compute-only Node from Version 2.2(6f) to3.1(2b) or Later

If you are upgrading your compute-only node from 2.2(6f)/1.7x to 3.1(2b)/1.8(1c), the newHXData PlatformUpgrade workflow will automatically upgrade the HX Data Platform components.

However, after the HX Data Platform upgrade completes, you need to manually complete the upgrade of thecompute-only node server firmware.

Step 1 Ensure Distributed Resource Scheduler (DRS) is enabled and set to automatic mode.


Upgrading Cisco HyperFlex Software ComponentsCompute-only Node Upgrade Considerations

This allows the HX node to be vacated automatically. Otherwise, use vMotion to manually migrate all VMs off the host.Migration with vMotion allows you to move a virtual machine to a new host without any interruption in the availabilityof the virtual machine. For detailed steps, see VMware KB article.

Step 2 Place each compute-only node into maintenance mode.Step 3 In the UCS manager GUI, Servers > Service Profiles, click Pending Activities to display the tasks requiring user

acknowledgment before they can complete. Manually Acknowledge the pending reboot for each of the compute-onlyserver. This will trigger a reboot and subsequent installation of new version of UCS firmware.

Step 4 Wait for the compute-only node to upgrade and fully reboot. After the compute-only node is up, verify the correct UCSfirmware is present in UCS Manager. See Check Firmware Versions in UCSM, on page 46.

Step 5 Repeat this process for all compute-only nodes in the cluster.


Upgrading Cisco HyperFlex Software ComponentsUpgrading Cisco UCS Firmware on Compute-only Node from Version 2.2(6f) to 3.1(2b) or Later

https://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.vsphere.vcenterhost.doc/GUID-0542FCAA-209A-4296-8AA4-0A51C13A6233.html


Upgrading Cisco HyperFlex Software ComponentsUpgrading Cisco UCS Firmware on Compute-only Node from Version 2.2(6f) to 3.1(2b) or Later

C H A P T E R 7Post Upgrade Tasks

• Confirm That Upgrade Is Complete, on page 45• Check Firmware Versions in UCSM, on page 46• Verify If Cleaner Is Running, on page 46

Confirm That Upgrade Is CompleteAfter the upgrade is complete and the HyperFlex cluster has been upgraded, log out and log back in to vCenterto see the upgrade changes.

Step 1 Log in to Cisco UCS Manager to ensure that the HX nodes have no pending server activities.

From Server > Pending Activities tab check for all server activities.

Step 2 Confirm that the HX nodes match the expected firmware version.

In Cisco UCS Manager, from Equipment > Firmware Management > Installed Firmware tab, verify for the correctfirmware version.

Step 3 Log in to any controller VM through SSH.

# ssh root@controller_vm_ip

Step 4 Confirm the HyperFlex Data Platform version.

# stcli cluster version

Cluster version: 2.5(1c)Node HX02 version: 2.5(1c)Node HX05 version: 2.5(1c)Node HX01 version: 2.5(1c)Node HX03 version: 2.5(1c)Node HX04 version: 2.5(1c)

Step 5 Verify that the HX storage cluster is online and healthy.

# stcli cluster info|grep -i health

Sample output:healthstate : healthystate: healthystorage cluster is healthy


Step 6 Verify that the datastores are up and are mounted properly on the ESXi host.

From HX controller VMs run the command:

# stcli datastore list

From the ESXi host run:

# esxcfg-nas -l

Step 7 For each browser interface you use, empty the cache and reload the browser page to refresh the HX Connect content.

Check Firmware Versions in UCSMIn Cisco UCS Manager, from Equipment > Firmware Management > Installed Firmware tab, verify forthe correct firmware version.

For a complete list of hardware and software inter-dependencies, refer to respective UCSM release versionusing the UCS Hardware and Software Compatibility tool.

Verify If Cleaner Is RunningIf Upgrade Fails

If upgrade fails, run cleaner. This is required even if you do not want to continue with an upgrade.

To run cleaner manually, restart the storage cluster cleaner using the following command.

stcli cleaner start [-h] [--id ID | --ip NAME]

Syntax Description DescriptionRequired or OptionalOption

ID of storage cluster node. The ID is listed in thestcli cluster info command.

Optional.--id ID

IP address of storage cluster node. The IP is listedin the stcli cluster info command.

Optional.--ip NAME

If Upgrade Completes

If upgrade completes, verify if cleaner is running. To obtain information about the storage cluster cleaner forthe specified node, use the following command.

stcli cleaner info [-h] [--id ID | --ip NAME]

Syntax Description DescriptionRequired or OptionalOption

ID of storage cluster node. The ID is listed in thestcli cluster info command.

Optional.--id ID


Post Upgrade TasksCheck Firmware Versions in UCSM


DescriptionRequired or OptionalOption

IP address of storage cluster node. The IP is listedin the stcli cluster info command.

Optional.--ip NAME


Post Upgrade TasksVerify If Cleaner Is Running


Post Upgrade TasksVerify If Cleaner Is Running

C H A P T E R 8Known Issues

• Overview, on page 49• Upgrade Button Missing in Cisco HX Data Platform, on page 49• Cluster Start Command Error: Node Not Available, on page 49• A Node Fails to Upgrade due to vCenter Issues, on page 50

OverviewThis chapter provides information to help you troubleshoot common problems that may occur during theCisco HyperFlexupgrade process.

Upgrade Button Missing in Cisco HX Data PlatformIf the upgrade button is missing in the Cisco HX Data Platform Objects tab, under the Actions drop-downlist:

• Try using a different browser with the latest version of flash.

Firefox browser is not supported due to an outdated version of flash that is bundled with the browser.Manual update of flash within Firefox is possible, but the recommendation is to use either Chrome orInternet Explorer with a modern version of flash.

• Try restarting VMware vCenter.

Cluster Start Command Error: Node Not AvailableDescription

After an offline upgrade, due to a VMware EAM issue, sometimes all the controller VMs do not restart. Thestcli start cluster command returns an error: Node not available.

Action: Manually power on the controller VMs, then start the storage cluster.

Step 1 Manually power on the controller VMs.a) Login to the vSphere Web Client.


b) Locate the controller VMs that are not powered on.

From the Navigator select, vCenter Inventory Lists > Virtual Machines > vm.

Storage controller VMs, have the prefix, stCtlVM.

c) From the right-click or Actions menu select, Power > Power On.d) Repeat until all the storage controller VMs are powered on.

Step 2 Restart the storage cluster.a) Login to the command line of any controller VM.b) Run the command.

# stcli cluster start

A Node Fails to Upgrade due to vCenter IssuesDescription

Sometimes during an online upgrade the vCenter daemon crashes on a node. When this happens, the nodecannot enter HX maintenance mode. Without entering HX maintenance mode, the node cannot complete theupgrade. All other nodes, with properly functioning vCenter, complete the upgrade.

Action: Re-run the Upgrade on the Affected Node

1. Correct the vCenter issue.

2. Re-run the upgrade on the affected node.


Known IssuesA Node Fails to Upgrade due to vCenter Issues

cisco hyperflex systems upgrade guide, release 2...cisco hyperflex systems upgrade guide, release...

Documents