cisco networkers 2006 - san-3608 - advanced san troubleshooting

1© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicSAN -360812527_04_2006_c1

Advanced SAN Troubleshooting

SAN-3608


Agenda/First Half

• Overview of Data Gathering Capabilities of SAN/OS

• Network Design—Keep Troubleshooting in Mind

• Analyzer Uses

• Trace and Fabric Manager Tool Uses

• Tie Each Learned Skill to a Lab Exercise


MDS Switch Troubleshooting Levels

• Command line debugs and show commands

Used by network Level 2–3 engineers

• Analyzers and SPAN

Used by Level 3 engineers

• Fabric manager and device manager

Used by network operators and Level 1 engineers

• General statistics gathering from show commands and GUI managers

Network admins and operators Level 1 engineering


Command Line Debugging

• Available debugs depend on features enabled in SAN/OS

• Many, many different options to select when turning on debugs

• Where is it output going?Logfile—Data file in switch memory

Capture to direct to screen via console, telnet or SSH

• Requires admin privileges to run debugs

• Debugs can only be run from CLI

• No debugging available from Fabric Manager or Device Manager

N


Debug Logging

1. Display captured debug to screenTip: use show debug to see name of debug file

TOP-9216i# show debug logfile networkers_debug

2. Copy debug file off MDS to a serverTOP-9216i# copy log:networkers_debugs ftp:

Enter hostname for the ftp server: 10.91.42.166

Enter username: networkers

Password: networkers

To delete the debug logfile TOP-9216i# clear debug-logfile networkers_debugs

Or

TOP-9216i# undebug all

Or

debug logfile will be cleared and over written when next debug logfile is created, only one debug logfile is allowed by system


Debugs to Direct Telnet Window

• Use a telnet/SSH or console application that will capture the expected output to buffer or file

• Undebug all or no debug of specific debug command is required to turn trace off

• The debugs are not persistent across reboots

• Most debugs are very readable and sensible to understand, some not

666N


Debug Outputs

• Some debugs are self-explanatory, others not

• Drilling down to most single item you wish to view like the interface or VSAN helps in narrowing output

• Debugs are usually run as part of a recreate for an issue such as a bad Fibre Channel ID being assigned or a Host login issue

• Some debugs require expertise in state machine and good general knowledge of FC and SCSI standards

N

Debug Example


Show Commands

• Hundreds of show commands

• Unlike debug commands, much information from show commands is available via the fabric manager and device manager

• Many show commands can be several screen pages. Use show terminal command to see information about your telnet terminal properties.

• Screen will pause at each screen unless terminal length is temporally configured different for current output

• Setting terminal length = 0 helpful in gathering long outputs, i.e. show tech-support detail

N


Design for Troubleshooting

• Leverage VSAN design to support troubleshooting mythology

• Have a SPAN port allocated in port count needs

• Integrate a Cisco Port Analyzer Adapter (PAA) into SAN design

• Provision an Analyzer (Finisar, Agilent, other) in the network, keep it operational with FTP access to extract traces

• Design for Syslog servers and scheduled configuration saves (use CFS to simplify)


How Do We Troubleshoot the Network

mds-speed.server2

microsoft:server1

TOP-9216i

mds-speed.server3

Server1

Server2

Clarion Cx300

Seagate 92:7a:47-S

Seagate 92:7b:94-S

Seagate 92:7b:c5-S Seagate 67:67:87-S

BOT-9216i

Ethereal PC

Ethernet

Out-of-Band Management Network

PC

qlogic:networkers1

FC Analyzer Remote

SPAN (Port Monitor) on

Cisco L2 Switch

Debugs and Show Commands, Telnet

or Console

Fabric Manager and Device

Manager

Terminal

FC Analyzer Local

Run Ethereal on IP Host

FC Port Channel

FC ISL

iSCSI iS

CS

I

Server3

Span FC Ports, Port Channels,

iSCSI or FCIP Port to SD Port on MDS

PAA


Gathering Protocol Traces for Analysis

• Using built-in FC Analyzer (CLI)

Linux version from Ethereal.com

• Using Ethereal on PC (local and remote)

• Using the MDS port analyzer adapter w/SPAN

• Using an external FC Analyzer tester in line or with SPAN

Non-Disruptive to Switch Operations and Traffic on the SAN


Note: SPAN Is Used for FC Port to FC Port Monitoring

MDS FC Analyzer (SAN/OS Embedded)

• Output is displayed to the console in readable sniffer-like format

• Is only used to monitor Fibre Channel traffic to and from supervisor on the MDS9000

Traffic-like fabric logins, FSPF routing, switch-to-switch control traffic

• Output can go direct to your console screen or to a workstation running Ethereal program


FC Analyzer Options

• Local or remote—Where to send the trace. Can be to local devices or remote PAA attached to different MDS switch.

• Brief or detailed—Header information vs. full output of frame including hex. Detail is default.

• Limit-captured-frames—Number of frames to capture. Default is FC analyzer will trace 100 frames. Specifying zero is unlimited frame capture.

• Limit-capture-size—Allows to capture N number of bytes of frame. Useful for not capturing frame data when it is not relevant to troubleshooting.


Display-Filter Options

• When not specified, FC analyzer will capture all traffic on VSAN 1Example: fcanalyzer local brief

• To specify a different VSAN, use a display-filter and specify the VSAN to be captured in hex or decimal

Example for 100 (note the two equal signs):

fcanalyzer local display-filter mdshdr.vsan==0x64 or mdshdr.vsan==100

• To capture for a specific address in VSAN 100:Example:

fcanalyzer local brief display-filter ((fc.d_id==64.01.00)or(fc.s_id==64.01.00))

• Recommend to use remote capture method for ease of filter capabilities on the Ethereal GUI


Using Write Option for fcanalyzer

• Using write option sends output of the fcanalyzer to a file on the switch, directory called volatile. This trace file can be then copied off MDS switch and viewed with ethereal app on PC

Example: Capture 250 frames of all traffic on VSAN 200

fcanalyzer local display-filter mdshdr.vsan==0xc8 write volatile:capture.trc limit-captured-frames 250

• The file name on volatile: filesystem will have extra characters appended. Issue following command to see contents of filesystem and then copy the file via tftp/ftp

dir volatile:

copy volatile:capture_00001_20050321172628.trc tftp://<tftp server ip addr>/capture.trc Added Characters

When File Is Written


Use of SPAN Feature

• Used for FC port to FC port analyzing

• Same type of tool as used on Cisco Catalyst® products. Cisco Catalyst uses port monitor.

• Can be left configured on switch

• Ingress and egress ports are sent to an FC-port setup as a SPAN Destination (SD-port type)

• No limits to where the ports are located on the MDS switch network

• Used to output to third-party test equipment or to Cisco Port Analyzer Adapter

N


FCAnalyzer Local Brief

• Display filters are a must for narrowing output on a busy network. See MDS Config Guide for other filter types.

• Capture is done by default to console screen, so make sure you are able to save output to large capture buffer or log with your telnet application

Capture Is Done in Configuration Mode

Brief Option Used to get Single Line Caption Along with a

Display Filter to Narrow Output to only VSAN 2

Control-C Stops Trace Capture

N

TOP-9216i# conf tEnter configuration commands, one per line. End with CNTL/Z.TOP-9216i(config)# fcanalyzer local brief display-filter mdshdr.vsan==2Warning: Couldn't obtain netmask info (eth2: no IPv4 address assigned).Capturing on eth2 2.829871 00.00.00 -> ff.ff.fe 0x2288 0xffff FC ELS FLOGI 2.853261 ff.ff.fe -> 7e.01.00 0x2288 0xc728 FC ELS ACC (FLOGI) 2.853422 7e.01.00 -> ff.ff.fc 0x22a0 0xffff FC ELS PLOGI 2.853592 ff.ff.fc -> 7e.01.00 0x22a0 0xc729 FC ELS ACC (PLOGI) 2.853565 7e.01.00 -> ff.ff.fd 0x22b8 0xffff FC ELS SCR 2.859648 ff.fc.7e -> 7e.01.00 0xc72c 0xffff FC ELS PLOGI 2.860885 7e.01.00 -> ff.fc.7e 0xc72c 0x22d0 FC ELS ACC (PLOGI) 2.861007 ff.fc.7e -> 7e.01.00 0xc72b 0xffff FC ELS PRLI 2.861175 7e.01.00 -> ff.fc.7e 0xc72b 0x22e8 FC ELS ACC (PRLI) 2.862053 7e.01.00 -> ff.ff.fc 0x2300 0xffff dNS RFT_ID 2.865904 ff.fc.7e -> ff.fc.0a 0xc72e 0xffff SW_ILS SW_RSCN 2.865981 ff.fc.7e -> ff.fc.cb 0xc72f 0xffff SW_ILS SW_RSCN 2.866153 ff.fc.0a -> ff.fc.7e 0xc72e 0x77f9 FC Link Ctl, ACK1 2.866297 ff.fc.cb -> ff.fc.7e 0xc72f 0x77fa FC Link Ctl, ACK1 2.866445 ff.fc.0a -> ff.fc.7e 0xc72e 0x77f9 SW_ILS SW_ACC (SW_RSCN) 2.866496 ff.fc.7e -> ff.fc.0a 0xc72e 0x77f9 FC Link Ctl, ACK1 2.866615 ff.ff.fd -> 7e.01.00 0x22b8 0xc72a FC ELS ACC (SCR) 2.868792 ff.fc.cb -> ff.fc.7e 0xc72f 0x77fa SW_ILS SW_ACC (SW_RSCN) 2.868857 ff.fc.7e -> ff.fc.cb 0xc72f 0x77fa FC Link Ctl, ACK1 2.871132 ff.fc.7e -> 7e.01.00 0xc730 0xffff FC ELS LOGO 2.872013 7e.01.00 -> ff.fc.7e 0xc730 0x2318 FC ELS ACC (LOGO) 2.872021 7e.01.00 -> ff.fc.7e 0x2318 0xffff FC ELS PLOGI 2.872139 ff.fc.7e -> 7e.01.00 0x2318 0xc731 FC ELS LS_RJT (PLOGI) 2.872163 ff.fc.cb -> ff.fc.7e 0x77fb 0xffff dNS GE_ID 2.872234 ff.fc.7e -> ff.fc.cb 0x77fb 0xc732 FC Link Ctl, ACK1 2.891239 ff.fc.7e -> ff.fc.cb 0x77fb 0xc732 dNS ACC (GE_ID) 2.891359 ff.ff.fc -> 7e.01.00 0x2300 0xc72d dNS ACC (RFT_ID) 2.891469 7e.01.00 -> ff.ff.fc 0x2330 0xffff dNS RFF_ID 2.891613 ff.fc.cb -> ff.fc.7e 0x77fb 0xc732 FC Link Ctl, ACK1 2.900160 ff.ff.fc -> 7e.01.00 0x2330 0xc733 dNS ACC (RFF_ID) 2.900394 7e.01.00 -> ff.ff.fc 0x2378 0xffff dNS RNN_ID 2.901916 ff.ff.fc -> 7e.01.00 0x2378 0xc734 dNS ACC (RNN_ID) 2.902151 7e.01.00 -> ff.ff.fc 0x23a8 0xffff dNS RSNN_NN 2.908296 ff.ff.fc -> 7e.01.00 0x23a8 0xc735 dNS ACC (RSNN_NN) 2.908444 7e.01.00 -> ff.ff.fc 0x23c0 0xffff dNS GNN_FT 2.919880 ff.ff.fc -> 7e.01.00 0x23c0 0xc736 dNS ACC (GNN_FT)


FCAnalyzer Local Detail(PLOGI)

FCID of HBA Device

N

Port Login

Fabric Domain Controller

Class 3 Device

VSAN 2


TOP-9216i

MDS FC Analyzer Remote

• For capture of internal MDS9000switch traffic, FC traffic to and from supervisor not FC port-to-FC port traffic

• Make sure MDS can ping workstation

Workstation Running EtherealIP Address 10.91.42.167

Management 10/100 Port on MDS Address 10.91.42.183

N

At DOS Prompt:ethereal -i rpcap://10.91.42.183/eth2

Or at Ethereal Interface Drop-Down:rpcap://10.91.52.183/eth2

SCSI

HBA


Ethereal on PC

Same Output as FC Analyzer Local but in Nice Format with the Ethereal GUI

Filters applied after capture or before. Filters are stored in

application cache for later use.


MDS 9000 Port Analyzer Adapter

• MDS Port Analyzer Adapter is a protocol converter

Converts FC frames into Ethernet frames

FC-specific information is carried in encapsulation header

Enables use of Ethernet protocol analyzers to decode FC protocols

• Cisco has released decodes for all FC protocols, the encapsulation header and SCSI decoder to public domain

Part of the popular Ethereal protocol analyzer


FC

FC

TOP-9216i

Port Analyzer Connection Using Port Analyzer with Ethereal

• Configure Port Analyzer Adapter (dip-switch settings, mostly plug and play)

• Configure SPAN port on MDS9000 and what device ports you wish to sniff

• Set Ethereal on PC to capture on local Ethernet interface

Cisco PortAnalyzer Adapter

Workstation L2 Network Running Ethereal

SD Port FC 2/16

Raid Array

For Capture of FC Device to FC Device, Sends Fibre Channel Traffic to an IP-Attached Workstation Running Ethereal

PAA & EtherealCapture on

Ethernet Interface

Connected to PAA

FCPING from Domain Controller in Domain

6f to FCID 6f.02.00

ELS Echo

Boardwalk Is Cisco Code for PAA


TOP-9216i

SPAN Configuration with Single FC Port Analyzer Tool

SPAN DestinationPort (SD) FC 1/13

FC 1/2

FC Analyzer

TX TXRX RX

Dropped

• FC 1/2 RX input is sent to FC 1/13 SD port

• FC 1/2 TX input is sent to FC 1/13 SD port

TX

RX

span session 1

destination interface fc1/13

source interface fc1/2 rx

source interface fc1/2 tx

interface fc1/13

switchport description Span Destination

switchport mode SD

switchport speed 1000

no shutdown


RSPAN Requirements

• All switches must be running MDS code 1.2 or later

• No cable should be attached to the SFP in the ST (SPAN Tunnel) port

• Make sure FC tunnel is UP before starting frame collection

• FC Analyzer can be a PAA and PC running Ethereal, or a Finisar-like analyzer


Fabric Manager/Device Manager

• The MDS Fabric Manager Server is your friend

• Used in combination with CLI interface helps tremendously in situations were you cannot remember all the configuration steps

• Use CLI to verify outputs to the managers but trust the manager displays to be proper


Fabric Manager Views

• Show Tech Support output

• Zoning Checks

• Domain Manager ID across all VSANs

• FSPF Health

• RSCN Statistics and health


Device Manager Views

• Span configuration

• Monitor traffic

• Monitor interface Errors and statistics

• View persistent logs

• Manage files in flash

For Troubleshooting:


Lab 1

• Lab intro

• Debug output example

• Ethereal tool usage

• PAA and SPAN usage

• FM and DM tool navigation

Server2

iqn.1991-05.com.cisco.mds-speed.server3

iqn.1991-05.com.mds-speed.server2

fc1/5 iscsi1/1

fc1/14

ClarionCx300

Server3

fc1/6

Seagate92:7b:c5-S

Seagate92:7a:47-S

Seagate92:7b:94-S

Seagate67:67:87-S

Bot-9216i

Server1 iqn.1991-05.com.microsoft:server2

channel1,channel1

fc1/1

fc1/1 fc1/14

fc1/2

PAA

fc1/5

TOP-9216i

iscsi1/1

iscsi1/1

iscsi1/1

fc1/6

fc1/6

fc1/6

fc1/6

fc1/6

fc1/6

fc1/6


Section Agenda/Second Half

• Fibre Channel Layer Review

• Determining Low-Level Interface Physical Issues

• Best Practices for Troubleshooting Device Connectivity

Lab

• Inter-Switch Link—Understanding and Troubleshooting

• Switch Domains and VSANs

• Zoning Details

• Inter-VSAN Routing—IVRLab


Fibre Channel LayersReview

• FC-0—defines the physical interface characteristicsSignaling rates, cables, connectors, distance capabilities, etc.

• FC-1—defines how characters are encoded/decoded for transmissionTransmission characters are given desirable characters

• FC-2—defines how information is transported

Frames, sequences, exchanges, login sessions

• FC-3—place holder for future functions

• FC-4—defines how different protocols are mapped to use Fiber Channel

SCSI, IP, Virtual Interface Architecture, others

Structure Is Divided into Five Levels of Functionality

FC-PH

FC-0

FC-1

FC-2

FC-3

FC-4s

Common Services

Signaling Protocol

Transmission Code

Physical Interface

IPSCSI


Fibre Channel Transmission Words: FC-1

• Ordered sets are transmitted continuously to indicate that specific conditions within the link are encountered

• Ordered sets transmitted while the condition exist

• Four primitive sequences which can determine where problem exist

Not Operational Sequence (NOS)

Offline Sequence (OLS)

Link Reset Sequence (LR)

Link Reset Response Sequence (LRR)


Link Initialization FlowFibre Channel Layer 1 Protocol (FC-1)

These Are All Special-Ordered Sets of 8B/10B Coding:

Link Failure ConditionAC AC

LFLF

LR

OL

LR

AC

AC

NOS

OLS

LR

Idle

LRR

Idle

Port A Port B

NOS = Not Operational Sequence

OLS = Offline Sequence

LR = Link Reset

LRR = Link Reset Response

AC = Active State

LR = Link Recovery State

LF = Link Failure State

OL = Offline State


Fibre Channel Port Issues

1. The switch port must first acquire bit and word synchronization with the N_Port

2. N_Port must issue a FLOGI to the MDS

Tip: Clear Counters and Monitor to Verify Active Issues, Use Device Manager Monitor Tool to Monitor Live; Set and Activate Threshold Manager to Alert You;

TOP-9216i# clear counters interface fc 1/1

In Order for an F_Port to Come Up on an MDS Switch:

Primitive Sequences Counters Can Determine

Layer 0–1 Problems


Fabric Services Refresher

• FFFFFE—Well-known address of login server

Basic responsibility to assign FCID

• FFFFFC—Well-known address of fabric directory name server

Basic responsibility registration of devices and distribution of database

• FFFFFD—Well-known address of fabric controller

Basic duties responsible for managing fabric, initialization, FSPF routing, responsible for RSCNs and listens for SCNs

FFFcxx—Cisco Specific, Representing Name Server in a VSAN and Domain xx

FC Standards


Line Card Basics

• Show module will display slot locations and type of card

• General information in interface and statistics can be gathered from switch main-level prompt

• More detailed gathering of ASIC counters may be required to troubleshoot difficult issues, attach would then be required

Attach to Module with “Attach” Command, All Modules Can Be Attached to, Including Standby Supervisor and IPS Blade

Exit to Detach


TOP-9216i

Using FC Test Equipment for Physical Issues

• Use MDS show commands and debugs to narrow scope of physical issues

• External analyzers will be required for detail diagnostics of intermittent physical layer issues and use of triggers to trap problem

• Testers must go inline, disruptive to network connection. Plan accordingly.

FC AnalyzerTX TXRX RX

N

FC

FC


Errors on Application Server Logs, User Having Performance Problems

• Use logging capabilities on switch to piece together network issues

• Put together:

1. What interfaces are on the fabric (up, down, flapping)

2. What was health of each interface (virtual and physical interfaces)

3. Examine interface health (FC protocol errors, physical layer errors)

4. Are effects being seen on the complete network fabric or within the VSAN (RSCNs, Zones, ISLs, Errors on common controllers)


Necessary Network Settings

• Switch Fabric clocks synchronized

Use NTP along with CFS to simplify job

Use fabric manager to set all clocks and time zones

• Setup system logs, syslogd server

Use CFS to simplify

• Save off switch configurations or even complete show tech-support details regularly

Use CLI scheduler in SAN/OS to simplify process


Monitor Link Init State (Flow from Bottom to Top)

LINK: 052 022180c5 LR_RECEIVE(03) =>ACTIVE(01)

LINK: 051 022180c3 OLS_TRANSMIT(07) =>LR_RECEIVE(03)

LINK: 050 022180c1 SENTINAL(00) =>OLS_TRANSMIT(07)

LOOP: 049 022180c1 HW_ALPAS(0d) MASTER_LISA_WAIT(1f)=> OLD_PORT(3f)

LOOP: 048 022180c1 HW_ALPAS(0d) MASTER_LISA(1e)=> MASTER_LISA_WAIT(1f)

LOOP: 047 022180c1 HW_ALPAS(0d) MASTER_LIHA_WAIT(1d)=> MASTER_LISA(1e)

LOOP: 046 022180c1 HW_ALPAS(0d) MASTER_LIHA(1c)=> MASTER_LIHA_WAIT(1d)

LOOP: 045 022180c1 HW_ALPAS(0d) MASTER_LIPA_WAIT(1b)=> MASTER_LIHA(1c)

LOOP: 044 022180c1 HW_ALPAS(0d) MASTER_LIPA(1a)=> MASTER_LIPA_WAIT(1b)

LOOP: 043 022180c1 HW_ALPAS(0d) MASTER_LIFA_WAIT(19)=> MASTER_LIPA(1a)

LOOP: 042 022180c1 HW_ALPAS(0d) MASTER_LIFA(18)=> MASTER_LIFA_WAIT(19)

LOOP: 041 022180c1 HW_X_ARB(0c) MASTER_START(17)=> MASTER_LIFA(18)

LOOP: 040 022180c1 HW_LISM0(0a) OPEN_INIT_SELECT_MASTER(06)=> MASTER_START(17)

LOOP: 039 022180bf HW_R_LIP(09) OPEN_INIT_START(05)=> OPEN_INIT_SELECT_MASTER(06)

LOOP: 038 022180bf HW_X_LIP(08) NORMAL_INITIALIZE(04)=> OPEN_INIT_START(05)

LOOP: 037 022180bf HW_R_LIP(09) LPSM_STARTED(01)=> NORMAL_INITIALIZE(04)

LOOP: 036 022180b0 HW_OLDP(07) LPSM_DISABLED(00)=> LPSM_STARTED(01)

LINK: 035 022170e8 ACTIVE(01) =>SENTINAL(00)

Attached to Mod 1 FC Port 1/2

Interface Shut/No Shut via

Configuration

Port Tries Loop First

When Port Is Set to Auto

F-Port Point

to Point Negotiated

module-1# show hardware internal fc-mac port 2 stateinfo (older Line Cards)

module-1# show hardware internal fc-mac2 port 2 state-info-log (newer Line Cards


Show flogi Database

• FLOGI stands for fabric login

• Shows which devices are logged into the fabric

• Only shows devices that have flogi’ed into this switch not complete multi switch network

N

TOP-9216i# show flogi database ---------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME ---------------------------------------------------------------------------fc1/1 2 0x7e00ef 50:06:01:60:30:20:54:f0 50:06:01:60:b0:20:54:f0fc1/2 2 0x7e0100 21:00:00:e0:8b:17:2f:50 20:00:00:e0:8b:17:2f:50fc1/6 5 0xa3026e 21:00:00:04:cf:92:7b:c5 20:00:00:04:cf:92:7b:c5fc1/6 5 0xa30271 21:00:00:04:cf:92:7b:94 20:00:00:04:cf:92:7b:94fc1/6 5 0xa30272 21:00:00:04:cf:92:7a:47 20:00:00:04:cf:92:7a:47fc1/6 5 0xa30273 21:00:00:04:cf:67:67:87 20:00:00:04:cf:67:67:87fc1/14 2 0x7e03ef 50:06:01:69:30:20:54:f0 50:06:01:60:b0:20:54:f0iscsi1/1 2 0x7e0004 21:07:00:0d:ec:20:97:c2 21:08:00:0d:ec:20:97:c2iscsi1/1 5 0xa30002 21:01:00:0d:ec:20:97:c2 21:12:00:0d:ec:20:97:c2iscsi1/1 5 0xa30004 21:07:00:0d:ec:1f:82:42 21:11:00:0d:ec:20:97:c2


Show Fibre Channel Name Server Database

• Logging into fabric with FLOGI and registering with nameserver are separate events

• Shows all devices in entire fabric

• FCID shown is the FCID assigned by the switch that device FLOGI’ed into

• Initiator or target is attribute registered with switch from device during device login

• Vendor field is determined by OUI assigned by IEEE


Show fcns databaseInformation output of show Command

Target or Initiator Passed to Switch via PROCESS ACCEPT and ACCEPT Frames During Nameserver Login

sh fcdomain domain-list vsan 2

OUI is Part of the PWWN and Is IEEE Assigned. 00:e0:8b is Assigned to Qlogic, 00:00:c9 is Assigned to Emulex, and 00:04:cf is Assigned to Seagate, Many Others

You will Require the Domain List from the Switches to Know Where the Devices Are Physically Connected on the SAN, Could Be IVR’ed from Other VSAN


----------------------------------------------------------------------------------VSAN:2 FCID:0x7e0100-----------------------------------------------------------------------------------port-wwn (vendor) :21:00:00:e0:8b:17:2f:50 (Qlogic) node-wwn :20:00:00:e0:8b:17:2f:50class :3node-ip-addr :0.0.0.0ipa :ff ff ff ff ff ff ff fffc4-types:fc4_features :scsi-fcp:init symbolic-port-name :symbolic-node-name :QLA2340 FW:v3.03.13 DVR:v9.0.2.17 (w32)port-type :N port-ip-addr :0.0.0.0fabric-port-wwn :20:02:00:0d:ec:20:97:c0hard-addr :0x000000permanent-port-wwn (vendor) :21:00:00:e0:8b:17:2f:50 (Qlogic)

Show fcns database detail

Gives Detailed Information About Each Device (Only One Shown)

FWWN This Device Attaches to the Fabric via

Registered to Talk SCSI over Fibre Channel

Device Registered and Thus Indicates It Can Converse with Class of Service 3


Debugging Name Server Issues

1. Check if the host and the disk are in the same VSAN (use fcping or fctrace, show VSAN membership)

2. Debug fcns events (this will give the request and response from the fabric name server)

3. Check if the host and the disk are in the same zone (show zoneset active vsan x); look for the * next to each device FCID; missing * means device is not in the fabric

If a Host Is Not Receiving a Proper Response, or if Not Able to Discover a Target, Enable the Following Debug and Shut/No Shut the Host Interface


FCAnalyzer Brief Results During HBA LoginCLI View

From FC-GS-4 standard

Register FC-4 Type, Register Node Name, Register Symbolic Node Name, Get Node Names-FC4 Type, Get Platform Name

Device Logging into Fabric Initial Sends FCID with All 0’s

HBA Does Port Login to the Directory Server and Receives ACC Back

HBA Registers for State Change Registration (SCR) to be Notified by the Switch if There any Changes in Fabric in Same Zone and Receives ACCept from Switch

MDS Begins to Query the Device with RNN_ID, RSNN_NN, GNN_FT

Switch-to-switch communication noted by SW_ILS. Registered state change notifications going out to zone.

Switch Sends ACCept Back andAssigns the HBA a FCID

HBA Registers with Name Server, RFT


HBA LoginEthereal Application View

• HBA and FC port on switch finish physical level handshake and HBA logs into port server a FF.FF.FE

• Login Server assigns FCID from Domain 6f (VSAN 200) 6F.02.00

• HBA logs into Fabric Name Server

• HBA registers for State Change Notification, response comes back in frame 7

• HBA asks Name Server for FC-4 FC devices registered in the server. Response comes back in frame 24

• Name Server from VSAN 200 MDS9216-TOP quires HBA

• Name Server from VSAN 200 MDS9216-Bottom quires HBA

• HBA Port logs to FC-4 devices received from accept frame 24

• HBA talks to SCSI devices

N


Link Events on a Switch

• Per module/per switch command

• Can be taken from Show tech-support detail

• Helps drill down as to what devices were leaving fabric at time of problem and general reason

• Shows both physical and virtual interfaces


Gathering Internal Counter Information for Unknown Issues and Plaguing Connectivity Problems

• Determine what ports on which switches you need to examine, this is done by narrowing to physical switch, VSAN, zone

• Look at path through complete network, initiator and target-side along with ISLs

• One stop shopping on each switch for all the data:Attach to Line Card Module that has the interface you need information on

module-1# terminal length 0

Set scroll back buffer or log to file on telnet tool

module-1# show hardware internal debug-info interface fc1/2

Based on Line Card type proper information will be output


show hardware internal debug-info

Script Runs Giving Outputs Of:

show hardware internal errors

show port-config internal link-events

show port-config internal port-control

show port-config internal port-cfg-error

show hardware internal fc-mac2 port 2 link-status

show hardware internal fc-mac2 port 2 port-info

show hardware internal fc-mac2 port 2 misc-statistics

show hardware internal fc-mac2 port 2 status-reg

show hardware internal fc-mac2 port 2 state-info-log

show hardware internal fc-mac2 port 2 gbic-info

show hardware internal fc-mac2 port 2 statistics

show hardware internal fc-mac2 port 2 flow-control

show hardware internal fc-mac2 port 2 config-reg

show hardware internal fc-mac2 port 2 mask-reg

show hardware internal packet-flow interface fc1/2

We Have Already Looked at Details of

Theses 2 Outputs


Lab 2

• HBA Login Lab

• Physical Layer Troubleshooting

• Debugging Fibre Channel services


ISL Connection

• ISL connections require proper standards operation

FSPF routing

Proper capabilities and parameter exchange

Proper domain ID assignment which could be disruptive or non-disruptive to network

Proper merging of fabric zoning

• ISL will not fully initialize unless FC-SW rules are complete

TOP-9216i BOT-9216i


Switch-to-Switch, the ISL E-Port

• Fibre Channel Standard FC-SW addresses ISL operation

• FC incorporates distributive methods across the SAN Fabric E-Ports seen as SW_ILS in protocol field of traces

• Switches if determine connecting FC port is an ISL link

• Much can go wrong as part of the initial ISL handshake

• ISL needs to be analyzed for situations of domain ID assignment, routing, and zoning distribution issues

N


Principal Switch SelectionReview

• Only one switch is designated principal switch in a fabric

• The switch with the lowest WWN becomes the principal switch at new install of SAN after that other rules apply

• Principal switch makes sure that no new switch is added to the fabric if it has a domain ID conflict with an existing switch in the fabric

• Only one upstream link per switch

N

Switch 1

Switch 3

Switch 4 Switch 5

Switch 6

Switch 2


Fabric Configuration: PS SelectionReview

• A principal switch shall be selected when at least one inter-switch link (link between two E_Port) is established

• The selection process chooses a principal switch, which is then designated to assign domain identifiers to all the switches in the fabric, existing and new


Fabric Configuration ProcessReview

• The fabric configuration process enables a switch port to determine its operating mode, exchange operating parameters, and provides for distribution of addresses

• The process is summarized in the following steps

Establish link parameters and switch port operating mode

Principal switch selection

Domain ID distribution

Path selection

N


Fabric Configuration Disruptive/Non-Disruptive

• One of the following three conditions can trigger Build Fabric, BF (non-disruptive) or Reconfigure Fabric, RCF (disruptive)

Two disjointed fabric are combined together

The principal ISL link fails (upstream or downstream)

A switch with Domain_ID request for another Domain_ID

• Whenever a switch receives a BF/RCF, the switch starts F_S_TOV timer and enters the BF/RCF state; it forwards BF/RCF out of all E_ports except the incoming port (only once) and waits for the timer to expire

• When the timer expires, BF/RCF propagation state is over and principal switch selection begins


Domain IDs

Fabric Manager View

Device Manager View

CLI View


Cisco TE EISL TE Port SPAN to PAA

Domain ID Assign by Existing Principal Switch

Request Domain ID from New Switch

Exchange Fabric Parameters

Exchange Link Parameters

Exchange Switch Capabilities

Zone Merge Request

Exchange Peer Parameters (This Is Proprietary to MDS Only to Determine if Connecting Switch Is Another MDS)

Enhanced Zoning Merge Request Resource Allocation (New in FC-SW3 Standard)

ACK1 Filter Applied

Build Fabric


Domain ID Problems

• Show interface

• Check fabric manager

• Check event logs

MDS9216-Bottom# sh int fc 1/9fc1/9 is down (Isolation due to domain overlap) Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:09:00:0d:ec:0d:0c:c0 Admin port mode is E, trunk mode is off Port vsan is 500 Receive data field Size is 2112 Beacon is turned off 5 minutes input rate 360 bits/sec, 45 bytes/sec, 0 frames/sec 5 minutes output rate 264 bits/sec, 33 bytes/sec, 0 frames/sec 2433 frames input, 198208 bytes 0 discards, 0 errors 0 CRC, 0 unknown class 0 too long, 0 too short 2423 frames output, 128924 bytes 0 discards, 0 errors 19 input OLS, 22 LRR, 1 NOS, 26 loop inits 310 output OLS, 107 LRR, 13 NOS, 10 loop inits 255 receive B2B credit remaining 255 transmit B2B credit remaining


Troubleshooting Zoning

• Show zoning information from CLI on all switches in fabric

Show zone status

Show zoneset active

• Is zone configuration stored and controlled from one switch or any/all switches?

• Make sure you are working with active zone when in CLIzone copy active-zoneset full-zoneset vsan 5

*Key concern when working in mixed vendor interop VSANs

• Verify distribution and default values (permit-deny)

• See event logs for clues about issues

• Verify device in logged into fabric


Zoneset Tools in FM

Backup Zone Configuration to File on ServerWhere Is Active ZoneSet in Each VSAN Applied?

Current Zoneset Status


Show Zoneset Active

• Shows the zoneset activated in the fabric

• An asterisk to left of device indicates that device has registered with the nameserver

• Does not show if a zone has been configured but changes have not been activated

• Zoneset and zone names are case sensitive

• Inter-VSAN routing zone added to ZoneSet via IVR wizard if this VSAN has member being zoned to another VSAN

N

Domain a3(163) Is in VSAN 5


Activate Zoneset Flow Across ISLs

ACC

ACA—Acquire Change Auth

Lock the Fabric

ACC

SFC—Stage Fabric ConfMove the Zone Data to the Switches

ACC

UFC—Update Fabric ConfTrigger Switches to

Activate the New Zoneset

ACC

RCA—Release Change Auth

Unlock the Fabric

Zoneset Distribution Would Go to Every Domain Within the VSAN

ACK1 Frames Are Not Shown Here


Activate Zoneset with FC Analyzer and VSAN Filter

Only VSAN 100

FC interface 1/16


IVR Troubleshooting Tips

• Start out with valid well thought-out IVR design

• Pay close attention to zoning

• Show IVR Tech-Support

• Look at complete IVR path

• Use CFS (Cisco Fabric Services)

• Domain ID limit rules in Interop VSANs also apply to the IVR Domains (97–127 rule)


Which Switch, What Zone, What Feature

TOP-9216i# sh zoneset brief active zoneset name ZoneSet1 vsan 2 zone Server1 zone Server3 zone iSCSIserver3 zone iSCSIserver2 zone IVRZ_Server1-IVR

TOP-9216i# sh ivr zoneset brief active

zoneset name IvrZoneSet1 zone name Server2-IVR

zone name Server1-IVR

CLI Displays Will Show Zones from IVR Feature While Displays on FM Only Display All None IVR Zones


Lab 3

• TE port initialization

Detailed look

• Domain ID assignments

Adding a domain (switch) to the SAN

• Zone merge

Rules of zoning


MDS Documents and Whitepapers on Net

• http://cisco.com/en/US/products/hw/ps4159/ps4358/prod_white_papers_list.html

• http://www.cisco.com/univercd/cc/td/doc/product/sn5000/mds9000/interop/index.htm

• http://www.cisco.com/en/US/products/hw/ps4159/tsd_products_support_category_home.html

• http://www.emc.com/interoperability/index.jsp

• http://www.redbooks.ibm.com/abstracts/sg246397.html?Open

• ftp://service.boulder.ibm.com/storage/san/cisco/

• http://h18006.www1.hp.com/products/storageworks/cmds9509/index.html


Q and A


Recommended Reading

• Continue your Networkers learning experience with further reading from Cisco Press®

• Check the Recommended Reading flyer for suggested books

Available Onsite at the Cisco Company Store


Complete Your Online Session Evaluation!

• Win fabulous prizes; Give us your feedback

• Receive ten Passport Points for each session evaluation you complete

• Go to the Internet stations located throughout the Convention Center to complete your session evaluation

• Drawings will be held in theWorld of Solutions

Tuesday, June 20 at 12:15 p.m.

Wednesday, June 21 at 12:15 p.m.

Thursday, June 22 at 12:15 p.m. and 2:00 p.m.