troubleshooting the cisco unified computing system...–does the cimc have an ip? is the bios...

BRKCOM-3001

Troubleshooting the Cisco Unified Computing System

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKCOM-3001 2

Agenda

UCSM & Fabric Interconnects

Blade Servers

IOM & Chassis

FEX & Rack server

Network

SAN


UCS System Components

UCS manager

UCS 6100 Fabric Interconnect

UCS 2100 Fabric Extenders

UCS 5100 Blade Chassis

UCS B-series servers

Nexus 2248 switch

UCS C-series servers

UCS Network adapters


UCS 6100 Fabric Interconnect (FI)

Standalone or Clustered

Primary / Subordinate

Data Management Engine (DME)

FI-B#FI-A#

Virtual IP

IP #BIP #A

Management Network

Cluster links

DBDB


UCS Fabric Interconnect ports

LAN & SAN northbound connections


UCS 2100 Fabric Extender

Chassis Discovery

Chassis SEEPROM stores UCS cluster state

FI-A FI-B

UCS 2100 Series Fabric ExtendersUCS 5100 Series Blade

Server Chassis

Chassis

CMC CMC

SE

EP

RO

M


UCSM

UCSM GUI

CLI

UCS-A# scope server x/y

NXOS

UCS-A# connect nxos a

UCS-A(nxos)# show…

XML API


UCSM – Faults & Events

Trigger

Fault Rules

Process

Fault or Event

Explicit Changes

Applied to MOs

Stimulus

Post-Processing

Transaction

Begin

Transaction

EndAG’s report

fault or event


Events lifecycleFaults Lifecycle

UCSM – Faults & Events lifecycle

Clear Condition

Retention Interval

or Acknowledged

Clear

Action

Retain

Delete

Raise Fault Clear Fault

Delete Fault

Create Event

Delete EventLog Full


Finite State Machine (FSM)

Workflow with many stages

Data Management Engine (DME)

… Application Gateway (AG)

… End Point (EP)

Notation

<Object><Workflow><Operation><Where-is-it-executed>


Sample FSM flow


UCSM – Common issues Firmware upgrade issues


UCSM – Common issues

Is the other FI up and operational?

Are clustering links up?

Is there at least 1 chassis successfully discovered on both FIs?

UCS-A# show cluster extended-state

UCS-A# show pmon state

UCS-A(local-mgmt)# cluster lead a

UCS-A(local-mgmt)# cluster force primary

DME Clustering problems


UCSM – Top 5 commands

UCS-A# show cluster extended-state

UCS-A /fabric-interconnect # show fsm status

UCS-A (local-mgmt) # show pmon state

UCS-A /monitoring/sysdebug # show cores

UCS-A (nxos) # show mgmt-ip-debug


Agenda

UCSM & Fabric Interconnects

Blade Servers

IOM & Chassis

FEX & Rack server

Network

SAN


Blade ServersMeet the family!

B200 M22 Socket Intel 5600, 2 SFF Disk, 12 DIMM



Bla

des

B230 M12 Socket Intel 6500/7500, 2 SSD (7MM), 32 DIMM


Blade serversBlade overview – Hardware & Software Components

CPU

& Heatsink

Memory DIMMS

Mezzanine

Adapter

CIMC

HDD


Blade servers

DIMM characteristics

Memory Population Rules

DIMM speed dependencies

Blade overview - Memory


Blade servers

CIMC

– Monitors Temperature and Power readings

– KVM & vMedia

– Blade control

BIOS

– Can be configured via F2 or via BIOS Policy

Blade overview – CIMC and BIOS


UCS B-Series Mezzanine CardsModel Chip Part No Type vNIC,vHBA Failover

Broadcom Everest M51KR-B NIC 2,0 OS

Cisco Palo M81KR-C CNA Up to 56 Fabric/OS

Emulex Menlo M71KR-E CNA 2,2 Fabric/OS

Emulex Tigershark M72KR-E CNA 2,2 OS

Intel Niantic M61KR-I NIC 2,0 OS

Qlogic Menlo M71KR-Q CNA 2,2 Fabric/OS

Qlogic Schultz M72KR-Q CNA 2,2 OS


Blade servers – Common issues

Server discovery failed

– Check minimum software version

– Reseat blade

– Minimum hardware satisfied?

No KVM Video

– Does the CIMC have an IP?

Is the BIOS corrupt?

– Recover BIOS

– Reset CMOS

UCS-A# show version

UCS-A /system # show capability

UCS-A /chassis/server/cimc # show mgmt-if

UCS-A /chassis/server # show post

UCS-A /chassis/server # reset-kvm

UCS-A /chassis/server # recover-bios <file>

UCS-A /chassis/server # reset-cmos

CIMC issues



Blade won’t boot

– Did POST complete?

Types of DIMM errors

– Mapped out

– Disabled

– Inoperable

– Degraded

UCS-A# connect cimc x/y

[ help ] # post

[ post ] # obfl

[ obfl ] # sel

UCS-A /chassis/server # show memory [detail]

UCS-A /chassis/server/memory-array/dimm #

show stats memory-error-stats detail

Hardware issues



Insufficient resources

Uplink connectivity issues

Service profile association failed



Service profile modifications

– Firmware updates

– Configuration changes

OS initiated

Hardware issue

IOM / FI issues

Use Maintenance policies to defer changes

Check OS

Unexpected reboot

UCS-A /chassis/server# show fsm status

UCS-A# connect cimc x/y

[ help ] # post

[ post ] # obfl

[ obfl ] # sel


Blade servers – Top 5 commands

UCS-A /chassis/server # show inventory expand detail

UCS-A /chassis/server # show status detail

UCS-A /chassis/server # show post

UCS-A /chassis/server # show sel

UCS-A /chassis/server# show fsm status


Agenda

UCSM & Fabric Interconnect

Blade Servers

IOM & Chassis

FEX & Rack server

Network

SAN


IOM & Chassis

CMC responsibilities

– Chassis Discovery

– Local cluster management

– Power & Thermal Management

Overview

ChassisManagement

Controller

FLASH

EEPROM

DRAM

Control

IO

Chassis

Signals

Switch

1 - 4

Fabric links

To

Interconnect

To Blades

Redwood ASIC


IOM & Chassis – Common issues

Check chassis discovery policy

Server ports defined correctly

FI to IOM 1:1 relationship only

UCS-A(nxos)# show run interface

ethernet x/y

UCS-A(nxos)# show interface fex-fabric

UCS-A(nxos)# show fex <chassis#> detail

Chassis not discovering



Spinning at 100%

– Temperature

– Any fans missing?

– CMC access to thermal sensors

– Component discovery

UCS-A# connect iom 1

fex-1# show platform software cmcctrl thermal status

fex-1# show platform software cmcctrl fancontrol all

fex-1# show platform software cmcctrl ohms all

Fan issues



Power Policy

– Grid, N+1 or non redundant

Power cap issues


fex-1# show platform software cmcctrl power manager

Power issues


IOM & Chassis – Top 5 commands

UCS-A(nxos)# show fex detail


fex-1# show platform software cmcctrl cmc manager

fex-1# show platform software cmcctrl thermal status

fex-1# show platform software cmcctrl obfl logs

fex-1# show platform software cmcctrl pstate


Agenda


Blade Servers

IOM & Chassis

FEX & Rack server

Network

SAN


FEX & Rack Server

Meet the Family!

C200 M22 Socket Intel 5600, 4 Disks, 12 DIMM, 2 PCIe 1U



Ra

ck M

ou

nt

Nexus 2248

48 1000BASE-T host interfacesFE

X


FEX & Rack ServerOverview


FEX & Rack ServerPort states

Unconfigured

UntrustedAllow native vlan 4044

Trusted Port Flap

(No Change)

Config

Server port

Admin down


FEX & Rack Server – Common issues

Server port configured?

FI to IOM 1:1 relationship

UCS-A(nxos)# show run interface eth x/y



FEX discovery issues


FEX & Rack Server – Common issues

Minimum hardware satisfied?

Minimum software version?

Are the ports in untrustedstate?

Are adapter connected ports on FI configured as server ports?

UCS-A(nxos)# show run int eth 2/1/8

interface Ethernet2/1/8

pinning server

switchport mode trunk

switchport trunk native vlan 4044

switchport trunk allowed vlan 4044,4047

no shutdown

Rack server discovery issues


FEX & Rack Server – Top 5 commands

UCS-A(local-mgmt)# show tech-support server # detail


UCS-A /server # show post

UCS-A /server # show fault

UCS-A /server # show fsm status


Agenda


Blade Servers

IOM & Chassis

FEX & Rack server

Network

SAN

LAN SAN BSAN A

UCS Fabric

Interconnects


UCS Networking

Each adapter has a connection to A and B

Half width blades have 1 adapter

Full width blades have 2 adapters

Fabric Failover

LAN SAN BSAN A

Half Width Blade Half Width Blade

Fabric Extender Fabric Extender

vN

IC

vN

IC

vN

IC

vN

IC

Adapter

CIMC

Adapter

CIMC

Chassis

UCS Fabric

Interconnects


UCS Networking – Blade to Blade traffic

Blade 1 Blade 2

vNIC0

VLAN 10

vNIC1

VLAN 10

HA link not for data traffic

Fabric InterConnect AFabric InterConnect B

External LAN

Chassis 1


UCS Networking – Blade perspective

Blade 1

veth0 veth1

vhba0 vhba1

0

OS

1

Eth X/Y/Z interface

External mezz card 10GE port

Virtual interface tagto associate frames to a VIF

IOM 1

Fabric A

IOM 2

Fabric B

‘Southbound’ or OS-side interfaces

Vif 1 Vif 2 Vif 3 Vif 4

IOM-to-FI link


UCS Networking – IOM to FI links

IOM

switch1 link

IOM

switch2 links

IOM

switch4 links

Server slots pinned to uplink

Uplink: slots 1,2,3,4,5,6,7,8

Uplink 1: slots 1,3,5,7Uplink 2: slots 2,4,6,8

Uplink 1: slots 1,5Uplink 2: slots 2,6Uplink 3: slots 3,7Uplink 4: slots 4,8

slot 1

slot 2

slot 3

slot 4

slot 5

slot 6

slot 7

slot 8

slot 1

slot 2

slot 3

slot 4

slot 5

slot 6

slot 7

slot 8

slot 1

slot 2

slot 3

slot 4

slot 5

slot 6

slot 7

slot 8


Blade to IOM to FI pinning with 4 links

UCS Networking – Blade to IOM to FI


UCS Networking - Modes

FI-A

LAN

FI-B

Server Ports

Primary Root Secondary Root

Active/ActiveBorder Ports

Server Ports

Primary Root Secondary Root

Border Ports

FI-BFI-A

LAN

End Host Mode (default) Switch Mode

Blocking


MulticastUnicast

UCS Networking – End host mode

Uplink

Ports

Server

Ports

UCS 6100

Deja-Vu Check

Blade 1 Blade 2 Blade 7

Uplink

Ports

Server

Ports

UCS 6100

RPF Check

Deja-Vu Check

Blade 1 Blade 2 Blade 7

G-pinned

Blade 4


UCS Networking - Trace the packet

M81KR

UCS-A# connect adapter 1/3/1

adapter 1/3/1 # connect

adapter 1/3/1 (top):1# show-macstats 0

adapter 1/3/1 (top):2# show-log

adapter 1/3/1 (top):3# attach-mcp

adapter 1/3/1 (mcp):1# vnic <id>

adapter 1/3/1 (mcp):2# lifstats -a <lif>.<uif>

M71KR


adapter 1/5/1 # show-port-stats [0-5]

Adapters





UCS-A(nxos)# show platform software fex info satport eth

1/1/3


fex-1# show platform software redwood sts

fex-1# show platform software redwood rate

fex-1# show platform software redwood oper

fex-1# show platform software redwood rmon 0 ni2

IOM



UCS-A# show service-profile circuit server 1/3

UCS-A(nxos)# show interface brief

UCS-A(nxos)# show vifs interface eth 1/1/3

UCS-A(nxos)# show run interface vethernet 1195

UCS-A(nxos)# show mac-address-table

UCS-A(nxos)# show platform software enm internal info global

UCS-A(nxos)# show platform software sifmgr info int eth 1/1/6

UCS-A(nxos)# show platform software sifmgr info int veth 1195

EHM mode Switch mode

UCS FI server ports



UCS-A(nxos)# show pinning border-interfaces

UCS-A(nxos)# show pinning server-interfaces

UCS-A(nxos)# show port-channel summary

UCS-A(nxos)# show interface port-channel 1

counters

UCS-A(nxos)# show hardware internal gatos port

eth 1/19

UCS-A(nxos)# show spanning-tree vlan <vlanid>

EHM mode Switch mode

UCS FI uplinks



switch# show run interface ethernet x/y

switch# show port-channel summary

switch# show mac-address-table

switch# show spanning vlan 300

Upstream switch


UCS Networking – Common issues

Is vethernet & uplink interface up?

Is vlan tagging correct?

Is the mac address being learnt on the FI and upstream switches?

Can it reach other devices in the same L2?

Can it reach its default gateway?



UCS-A(nxos)# show run interface vethernet 1195

UCS-A(nxos)# show mac-address-table

Blade cannot talk to outside network


UCS Networking – Common issues

Only occurs in EHM mode

Disjoint Layer 2 networks


broadcast-if 0x88c836c(port-channel1)

Network 1

VLAN 10-20

AR

P

Network 2

VLAN 50-90


UCS Networking – Top 5 commands


UCS-A# show fabric-interconnect mode

UCS-A(nxos)# show pinning border-interfaces




Agenda


Blade Servers

IOM & Chassis

FEX & Rack server

Network

SAN

LAN SAN BSAN A

UCS Fabric

Interconnects


SAN

Prior to UCS version 1.4

– FC end host mode only

– Direct attached NAS required ethernet switch mode

UCS version 1.4+

– FC switch mode or EHM

– Direct attached NAS in Ethernet EHM

SAN overview

UCS B-Series

UCS 6100 UCS 6100

FCoE Storage FC Storage


SAN – Hybrid topology

Merge zoning from upstream FC switches

UCS 6100

FCoE StorageFC Storage

Fibre Channel

FCoE

Core

Fabric A Fabric B

SAN Fabric Storage Arrays

SAN Edge A SAN Edge B

Direct Attach

UCS 6100


SAN - Trace the packet


adapter 2/2/1 # connect

adapter 2/2/1 (top):1# attach-fls

adapter 2/2/1 (fls):1# vnic

adapter 2/2/1 (fls):1# login <vnic>

adapter 2/2/1 (fls):1# lunmap <vnic>

Adapter M81KR





UCS-A(nxos)# show npv status

UCS-A(nxos)# show npv flogi-table

UCS-A(nxos)# show npv traffic-usage

FC EHM mode

FC Switch mode

UCS FI FC EHM





UCS-A(nxos)# show flogi database

UCS-A(nxos)# show fcns database vsan 300

UCS-A(nxos)# show zone status vsan 300

UCS-A(nxos)# show zoneset active vsan 300

UCS-A(nxos)# show fcdomain domain-list vsan 300

FCoE StorageFC Storage

Direct Attach

UCS FI FC Switch mode

FC EHM mode

FC Switch mode


SAN- Trace the packet

MDS# show npiv status

MDS# show flogi database

MDS# show fcns database vsan 300

MDS# show zoneset active vsan 300

MDS# fcping pwwn <pwwn>

Upstream FC switch


SAN – Common issues

Did it login to the correct VSAN?

Is zoning configured on upstream MDS for this blade?

Is LUN masking configured for this host?

Does OS have correct HBA driver installed?


UCS-A(nxos)# show npv status


MDS# show zoneset active vsan 300

Blade can flogi but cannot see LUN

FC EHM mode

FC Switch mode



Is the service profile boot order configured to boot from vHBA?

Is the storage processor (SP) to vHBA mapping correct?

Is the local LUN ID correct?

Blade can see LUN during install, but can’t boot from it?

FC EHM mode

FC Switch mode

MDS# show fcns database

UCS-A /chassis/server # show boot-order detail

VHBA: fc0

SAN Image Path:

Type: Primary

LUN: 0

WWN: 50:06:01:62:44:60:44:FA



Are the storage ports up?

Are they logged in?

Is zoning configured?

If no zoning is configured, is default zoning enabled?

Is LUN masking configured?



UCS-A(nxos)# show zoneset active vsan 300

UCS-A(nxos)# show zone status vsan 300

UCS-A(nxos)# show platform software fcoe_mgr info

interface ethernet 1/9

Direct attach FC/FCoE storage not visible to blade

FC EHM mode

FC Switch mode


SAN – Top 5 commands

UCS-A# show fabric-interconnect mode


UCS-A(nxos)# show tech-support details

UCS-A(nxos)# show tech-support npv

MDS# show flogi database

FC EHM mode

FC Switch mode


Conclusion


What to collect in a jiffy

General UCS issues

UCS-A(local-mgmt)# show tech-support ucsm detail

UCS-A(local-mgmt)# show tech-support chassis # all detail

Networking Issues

Upstream_Switch# show tech-support details

SAN Issues

UCS-A(nxos)# show tech-support npv

MDS# show tech-support details


Where to find more information

Hardware Installation & Service Guides Information http://www.cisco.com/en/US/docs/unified_computing/ucs/overview/guide/UCS_roadmap.html#wp38892

Release Notes http://www.cisco.com/en/US/products/ps10281/prod_release_notes_list.html

Software Upgrade & Installation Information http://www.cisco.com/en/US/products/ps10281/prod_installation_guides_list.html

UCS Troubleshooting Guide http://www.cisco.com/en/US/docs/unified_computing/ucs/ts/guide/UCSTroubleshooting.html

UCS Faults Reference http://www.cisco.com/en/US/docs/unified_computing/ucs/ts/faults/reference/ErrMess.html

Cisco Support Community https://supportforums.cisco.com/community/netpro/data-center/unified-computing

http://www.cisco.com/en/US/docs/unified_computing/ucs/overview/guide/UCS_roadmap.html

http://www.cisco.com/en/US/docs/unified_computing/ucs/overview/guide/UCS_roadmap.html

http://www.cisco.com/en/US/products/ps10281/prod_release_notes_list.html

http://www.cisco.com/en/US/products/ps10281/prod_installation_guides_list.html

http://www.cisco.com/en/US/docs/unified_computing/ucs/ts/guide/UCSTroubleshooting.html

http://www.cisco.com/en/US/docs/unified_computing/ucs/ts/faults/reference/ErrMess.html

https://supportforums.cisco.com/community/netpro/data-center/unified-computing






Key Takeaway

Quickly resolve issues to maximise uptime of the UCS as a whole

Distinguish between what is and is not

expected behaviour

Understand the

interactions

between system

components


Q & A


Complete Your Online Session Evaluation

Complete your session evaluation:

Directly from your mobile device by visiting www.ciscoliveaustralia.com/mobile and login by entering your badge ID (located on the front of your badge)

Visit one of the Cisco Live internet stations located throughout the venue

Open a browser on your own computer to access the Cisco Live onsite portal

http://www.ciscoliveaustralia.com/mobile

troubleshooting the cisco unified computing system...–does the cimc have an ip? is the bios...

Documents