fabricpath operation and troubleshooting · fabricpath operation and troubleshooting ... acronyms...

91

Upload: hoangdiep

Post on 04-Jun-2018

491 views

Category:

Documents


31 download

TRANSCRIPT

FabricPath Operation and Troubleshooting

Carlo Schmidt, Customer Support Engineer

BRKDCT-3313

Acronyms / Definitions

Acronyms Definitions Acronyms Definitions

ACL Access Control List FP FabricPath

ASIC Application Specific Integrated Circuit FTAG Forwarding Tag

ASID Anycast Switch Identifier LID Local Identifier

BD Bridge Domain LTL Local Target Logic

CE Classical Ethernet MIM MAC-in-MAC (common reference to FP

header)

DBUS / RBUS Data Bus / Result Bus PACL Port-based ACL

DRAP Dynamic Resource Allocation Protocol RACL Router-based ACL

DSID Destination Switch Identifier RPF Reverse Path Forwarding

ELAM Embedded Logic Analyzer Module SoC Switch-On-Chip

ES Emulated Switch SSID Source Switch Identifier

FE Forwarding Engine VACL Vlan-based ACL

FF Flood to Fabric VDC Virtual Device Context

Reference Slide

• FabricPath Overview

Benefits, Restrictions, and Configuration

• Key Concepts

Encapsulation, Trees, Topologies, STP

• Data Plane

Forwarding, Load-Balancing, MAC Learning

• vPC+

Challenges and Operation

• Troubleshooting

Verification steps, tools, and examples

Agenda

FabricPath Benefits

Single path between 2 points in L2 network

• Stability/Resilience at scale

• Disruptive convergence

Shortest path between switches + equal-cost load-balancing

• Core does not need to learn end host MAC addresses

• More resilient to loops

• No topology constraints, L3 anywhere

• Easy scaling / Non-disruptive merge

Existing

Layer2

FabricPath

Fabricpath OverviewUnicast: Known Destination MAC

CE FabricPath CE

MAC A MAC B

Ingress

FabricPath

(Edge) Switch

Egress

FabricPath

(Edge) Switch

DSID comes from

MAC address

table for MAC B

SSID comes

from S10’s

own switchID

TTL

decremented at

every FP switch

Intermediate

switches forward

based on DSID

DMAC B

SMAC A

Payload

DMAC B

SMAC A

Payload

DSID 20

SSID 10 DMAC B

SMAC A

Payload

FabricPath OverviewMultidestination (broadcast, multicast, unicast flood)

MAC A MAC B

DMAC B

SMAC A

Payload

DMAC B

SMAC A

Payload

SSID comes

from S10’s

own switchID

Root switch

for Tree 2

MAC B is

unknown DSID

= FloodSID

Ingress FP

Switch selects

Tree (FTAG)

Root switch

for Tree 1→ FabricPath interface

→ CE interface

→ Tree 1

→ Tree 2DMAC B

SMAC A

Payload

DMAC B

SSID

FTAG 1 DMAC B

SMAC A

Payload

DMAC B

SSID

FTAG 1

CE FabricPath CE

FabricPath support & configuration

• N7K with N7K-F1 linecard as of 5.1.1

• N7K with N7K-F2 linecard as of 6.0.1• N7K + FEX as of 6.1.1 (with N7K-F2) for CE

ports

• F2E as of 6.1.2

• N7K with N7K-F3 linecard as of 6.2.6

• N5500 as of 5.1.3 • no L3 module required

• N5500 + FEX as of 5.1.3 for CE ports

• N6K as of 6.0.2

• Enhanced L2 license required FabricPath

• Packaged as feature-set (plugin)

N7K(config)# install feature-set fabricpath

N7K(config)# feature-set fabricpath

N7K(config)# interface Ethernet4/1

N7K(config-if)# switchport mode fabricpath

...

N7K(config)# vlan 3002

N7K(config-vlan)# mode fabricpath

FabricPath & CE Vlans

• Two types of vlans

CE (Classic Ethernet, default)

FabricPath (FP)

• FP vlans cannot go on M1, M2 modules

• Only FP vlans will be carried over FP interfaces

• FP vlans can be mixed with CE vlans on edge interfaces

N7K(config)# vlan 3002

N7K(config-vlan)# mode ?

ce Classical Ethernet VLAN mode

fabricpath Fabricpath VLAN mode

Classic Ethernet

FabricPath

Port Type VLANs allowed

to be configured

VLANs allowed to

be brought up

N7K-M1, N7K-M2 FP, CE CE

N7K-F1, N7K-F2, N7K-F3 Edge FP, CE FP, CE

N7K-F1, N7K-F2, N7K-F3 Core FP, CE FP

N5500, N6000 Edge FP, CE FP, CE

N5500, N6000 Core FP, CE FP

Core = switchport mode fabricpath

Edge = switchport mode access || trunk

• FabricPath Overview

Benefits, Restrictions, and Configuration

• Key Concepts

Encapsulation, Trees, Topologies, STP

• Data Plane

Forwarding, Load-Balancing, MAC Learning

• vPC+

Challenges and Operation

• Troubleshooting

Verification steps, tools, and examples

Agenda

Encapsulation

Outer SA: SwitchID ingress FP switch system ID

SubswitchID is used in some cases of VPC+

LID is specific to the implementation

• N7K the LID is generally the port index of the ingress interface

• N5K/N6K LID most of the time will be 0

• EndnodeID is not currently used

Outer DA: For known SA/DA is taken from MAC table for DMAC

For broadcast and multicast is the same as DMAC

For unknown unicast DA is 010f.ffc1.01c0 (flood to vlan)

For known unicast DA, but unknown SA is 010f.ffc1.02c0 (flood to fabric)

Example

DMAC SMAC 802.1Q Etype PayloadCRC

(new)

FP

Tag

(32)

Outer

SA

(48)

Outer

DA

(48)

Switch_ID SubSwitch_ID LID

100 1 65535

N7K# show fabricpath switch-id | include SYS|\*

Legend: '*' - this system

SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED

*2028 b414.89e3.a041 Primary Confirmed No No

N7K# sh mac address-table address 0000.1234.5678

VLAN MAC Address Type age Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------------------

3000 0000.1234.5678 dynamic 0 2.0.1054

Local IDSubSwitch

IDSwitch ID

EndnodeID

[ 7:6]

EndnodeID

[ 5:0]

U/L I/G

0

16 bits8 bits12 bits26 1 1

47

1 1

RSVD

OOO

Ethernet II, Src: 02:00:64:01:FF:FF, Dst: 01:00:5e:00:00:02, Type: 0x8903

FabricPath Switch IDs, System IDs … and DRAP

4 5

• Each FP switch is identified by unique number (ID), dynamically assigned or static

• Dynamic Resource Allocation Protocol (DRAP) is responsible for allocating switch IDs and resolving duplicate-ID conflicts. Conflicts are resolved by renumbering switches with higher systemID(DRAP can only auto resolve non-static switch ID)

• When partitioned FP network is merged (or new switch joins the fabric) connecting interface is not enabled for data before all conflicts are resolved

1 2 3

3

+

=

1 2 3

64 5

N7K# show fabricpath switch-id

FABRICPATH SWITCH-ID TABLE

=========================================================================

SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED

----------+----------------+------------+-----------+--------------------

*3 c062.6bac.e343 Primary Confirmed Yes No

30 547f.ee02.ce3c Primary Confirmed Yes No

40 547f.ee04.5cfc Primary Confirmed Yes No

N7K(config-if-range)# no shut

%FABRICPATH-2-

FABRICPATH_LINK_BRINGUP_STALLED_STATIC: Link

bringup stalled due to conflicts

N7K# show fabricpath conflict all

Port State

---------------+------------------------

Ethernet3/31 Suspended due to conflicts

==============================================

Fabricpath Conflicts

SYSTEM-ID SWITCH-ID STATIC

---------------+--------------+---------------

c062.6bac.e343 3 Yes

c062.6bac.e342 3 Yes

Network Merges / Conflict resolution• Goal is to connect two networks with conflicting switch IDs

without incurring packet loss

1) Allocate new switch-id as secondary – tentative• Wait allocate delay time

2) Make new switch-id as secondary - confirmed• Wait transition delay time

3) Swap primary and secondary switch-ids• Wait transition delay time

4) Delete old switch-id (now a secondary switch-id)

More About Graceful Merge

Graceful merge changes the switch-id of a switch to

resolve switch-id collisions

The switch-id to change is based on the system-id

being higher value, or being dynamic

For a time period the switch is identified by two switch-

ids, packets for both are accepted but outgoing packets

only carry the primary switch-id

N7k# show fabricpath switch-id

Legend: '*' - this system

SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED

----------+----------------+------------+-----------+--------------------

*332 b414.89e3.a042 Primary Confirmed Yes No

N7k# show fabricpath isis switch-id

Legend: C - Confirmed, T - tentative, W - swap

S - sticky, E - Emulated Switch

'*' - this system

System-ID Primary Secondary Reachable Bcast-Priority

MT-0

b414.89e3.a042* 332 [C] 0[C] Yes 222 [S]

N7k# show fabricpath timers

Allocate Delay Timer : 10

Transition Delay Timer : 10

Link-up Delay Timer : 10

FabricPath Trees• Known unicast traffic is load-balanced across equal-cost

routes

• FabricPath uses two loop-free trees for unknown unicast, broadcast and multicast traffic

• Two trees are for load-balancing

• For each packet, tree is selected by ingress FP switch and choice is carried in the packet header

• Root of tree1 is the switch with highest Priority (highest sysID for tie)

• Root of tree2 is the switch with 2nd highest Priority (highest sysID for tie)

• Tree is a least-cost-to-the-root graph, with lower sysID used as tie-breaker

• In case of Tree1 root failure both roots are reelected

• Up to 16 trees starting in 7.0 on Nexus 5000 and 6000

→ FabricPath interface

→ Tree 1

→ Tree 2

R

Lower SysID wins

S2

SysID 10S3

SysID 20

S1

SysID 50

S4

SysID 30

Root Election / Tree construction• Every switch advertises its system ID

& Priority

• Once all nodes have spoken Broadcast Root is elected (Highest priority then Highest Mac address wins)

• Broadcast root system will Elect & Advertise Roots for additional multicast Trees (currently only 2 trees)

• Each node will independently run SPF with Tree Root and create 2 Trees

• Since Multicast roots are advertised by Broadcast Root system (Tree 1), in case of failure of the latter both Tree 1 and Tree 2 will re-converge

S101# show fabricpath isis database detail Fabricpath IS-IS domain: default LSP database

LSPID Seq Number Checksum Lifetime A/P/O/T

S1.00-00 0x000000E2 0x0FBB 1054 0/0/0/1

Instance : 0x000000DD

Area Address : 00

NLPID : 0xC0

Hostname : S1 Length : 2

Extended IS : S202.00 Metric : 40

Extended IS : S101.00 Metric : 40

Extended IS : S102.00 Metric : 40

Extended IS : S2.00 Metric : 40

Extended IS : S201.00 Metric : 40

Capability : Device Id: 1 Base Topology

Base Topo Ftag : Graph 1: Root: S1 Primary: 1, Secondary: 0 Nickname 1Graph 2: Root: S2 Primary: 2, Secondary: 0 Nickname 2

Base Topo Trees :

Trees desired: 2 Trees computed: 2 Trees usable: 2

Base Topo Roots : Graph 1: Root Nickname: 1Graph 2: Root Nickname: 2

Version :

Version: 1 Flags: 0

Nickname :

Priority: 0 Nickname: 1 BcastPriority: 255

Nickname Migration :

Swid: 1 Sec. Swid: 0

Encapsulation

• Ethertype for FabricPath packets is 0x8903

• TTL set to 32 and is decremented at every hop. Packet is discarded when TTL reaches 0.

• FTAG: (Forwarding TAG) Used for multidestination traffic; carries the ID of the tree chosen at the FabricPath ingress switch. DRAP is responsible to keep FTAGs unique/consistent. For known unicast, FTAG carries topology ID

Nexus# show fabricpath isis topology summary

Fabricpath IS-IS domain: default FabricPath IS-IS Topology Summary

MT-0

Configured interfaces: Ethernet4/4

Number of trees: 2

Tree id: 1, ftag: 1, root system: 001b.54c2.4244, 4

Tree id: 2, ftag: 2, root system: 001b.54c2.4243, 3

Root for Tree 1, FTAG 1

Root for Tree 2, FTAG 2

Wireshark decodes FP encapsulation (tested on 1.8.3) : EditPreferencesProtocolsCFPEnable Dissector

DMAC SMAC 802.1Q Etype PayloadCRC

(new)

Outer

SA

(48)

Outer

DA

(48)

6 bits10 bits16 bits

FTAG TTLEthertype 0x8903

FP

Tag

(32)

Reverse Path Forwarding Check• RPF: check where the source switch of the packet is

and only accept packets from the interface we would have used if we were to send packet to that source

• At each FP hop RPF check is performed for multidestination traffic against source switchID + FTAG

N7K# show l2 multicast trees

(ftag/2, topo/0, Switch-id 40), uptime: 1w0d, isis

Outgoing interface list: (count: 1, '*' is the preferred interface)

* Interface Ethernet3/39, [admin distance/115] uptime: 1d23h, isis

(ftag/2, topo/0, Switch-id 30), uptime: 1w0d, isis

Outgoing interface list: (count: 1, '*' is the preferred interface)

* Interface Ethernet3/35, [admin distance/115] uptime: 02:56:04, isis

(ftag/2, topo/0, Switch-id 100), uptime: 1w0d, isis

Outgoing interface list: (count: 1, '*' is the preferred interface)

* Interface Ethernet3/39, [admin distance/115] uptime: 1d23h, isis

(ftag/1, topo/0, Switch-id 30), uptime: 02:56:06, isis

Outgoing interface list: (count: 1, '*' is the preferred interface)

* Interface Ethernet3/35, [admin distance/115] uptime: 02:56:06, isis

May also use

show fabricpath isis trees

1 2

34

root

Accept packets from 3

Accept packets from 4,1,2

Accept packets from 1,4

Packets with FTAG==2 from

switch 30 will be accepted from

interface e3/35

Packets with FTAG==1 from

switch 30 will be accepted from

interface e3/35

N7K# show fabricpath topology vlan

Topo-Description Topo-ID Configured VLAN List

-------------------------------- --------- -------------------------------------

0 0 1-99, 200-4095

1 1 100-199

N7K# show fabricpath topology interface

Interface Topo-Description Topo-ID Topo-IF-State

------------------- -------------------------------- ---------- -------------

port-channel1 0 0 Up

Ethernet6/4 0 0 Up

Ethernet6/5 0 0 Up

port-channel1 1 1 Up

Topologies• Routing table & Trees (FTAGs) are per topology

• Switch ID is shared across all topologies

• FP interface may belong to several topologies

• N7K: up to 8 topologies support starting in 6.2

• N5K/N6K: As of 5.2.1 default + 1 extra topology is supported; main use is to permit separate L2 pods to use same local vlanset

Pod 1

Vlan 100-199

Vlan 1000-1099

Pod 2

Vlan 200-299

Vlan 1000-1099

Default Topology allowed

on all FP links

FP links in Topology 0

and Topology 1

fabricpath topology 1

member vlan 100-199

!

interface Port-channel1

switchport mode fabricpath

fabricpath topology 1

Topologies + Vlans• Flood/Multicast/Broadcast trees are per-vlan, made by pruning Topology Tree

• If vlan is not present on the switch, that switch will not be part of per-vlan tree

• This may lead to connectivity issues when not all transit switches in topology have all vlans

• similar to connectivity issues caused by liberal pruning vlans off trunks with MST

• Make sure each vlan exists in every transit switch in a topology

VL10

VL10

VL30

VL20

Topology TreeVLAN 10

VL10

VL20

VL30

VLAN 20 VLAN 30

FabricPath Software Architecture & Hardware tables

on the Supervisor Engine:

• FabricPath IS-IS routing protocol process that forms the core of the FabricPath control plane

• DRAP Dynamic Resource Allocation Protocol, ensures network-wide unique and consistent Switch IDs and FTAGs

• Resolves switch id conflicts

• U2RIB Unicast Layer 2 RIB, containing the “best” unicast Layer 2 routing information

• L2FM Layer 2 forwarding manager, controls MAC address table

on the Linecards:

• U2FIB – Unicast Layer 2 FIB, managing the hardware unicast routing table

• MTM – MAC Table Manager, managing the hardware MAC address table

MAC TableSwitch Table

I/O Module

SupervisorEngine

U2FIB

FabricPath IS-IS

U2RIB L2FM

MTM

DRAP

Other HW

Hardware Drivers

Fabric Path Control Plane initialization flowS101# show processes cpu | egrep "2rib|drap|fab|l2fm|PID"

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

9169 750 16723 0 0.00% 0.00% 0.00% - l2fm

9215 1050 7843 0 0.00% 0.00% 0.00% - m2rib

9555 1050 36161 0 0.00% 0.00% 0.00% - u2rib

9556 14740 163944 0 0.00% 0.00% 0.00% - isis_fabricpath

9557 820 31339 0 0.00% 0.00% 0.00% - drap

----------------------------------------------------------------------------

S101# show fabricpath isis

Fabricpath IS-IS domain : default

System ID : 8478.ac0e.4743 IS-Type : L1 Fabric-Control SVI: Unknown

...

Process is up and running

...

Interfaces supported by Fabricpath IS-IS :

port-channel1

Ethernet6/27

Ethernet6/28

----------------------------------------------------------------------------

S101# show fabricpath switch-id

Legend: '*' - this system

SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED/

ANYCAST

--------------+----------------+------------+-----------+--------------------

* 101 8478.ac0e.4743 Primary Confirmed Yes No

...

Processes start (isis, u2rib, m2rib, drap)

System ID obtained from backplane MAC

Switch ID is obtained from DRAP

As FP interfaces links come up, hellos sent and adjacencies formed

Switch ID conflicts (if any) resolved

FP Interfaces allowed to forward data

Unicast SPF is calculated

Routes installed to U2RIB

Fabric Path Control Plane initialization flowS101# show fabricpath isis interface

Fabricpath IS-IS domain: default

Interface: port-channel1

Status: protocol-up/link-up/admin-up

LSP interval: 33 ms, MTU: 1500

P2P Adjs: 1, AdjsUp: 1, Priority 64

Hello Interval: 10, Multi: 3, Next IIH: 00:00:03

Level Adjs AdjsUp Metric CSNP Next CSNP Last LSP ID

1 1 1 40 60 Inactive ffff.ffff.ffff.ff-ff

Topologies enabled:

Level Topology Metric MetricConfig Forwarding

0 0 4000 no UP

1 0 40 no UP -------------------------------------------

---------------------------------

S101# show fabricpath isis adjacency

Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:

System ID SNPA Level State Hold Time Interface

S102 N/A 1 UP 00:00:25 port-channel1

S1 N/A 1 UP 00:00:28 Ethernet6/27

S2 N/A 1 UP 00:00:27 Ethernet6/28

----------------------------------------------------------------------------

S101# show fabricpath isis spf-log

Fabricpath IS-IS domain: default SPF information

SPF log for Topology 0

Total number of SPF calculations: 55

Log entry (current/max): 20/20

Ago Level Reason Count Total

1d09h 1 New LSP S201.00-00 3 0.001141

1d09h 1 Updated LSP S2.00-00 2 0.000965

Processes start (isis, u2rib, m2rib, drap)

System ID obtained from backplane MAC

Switch ID is obtained from DRAP

As FP interfaces links come up, hellos sent and adjacencies formed

Switch ID conflicts (if any) resolved

FP Interfaces allowed to forward data

Unicast SPF is calculated

Routes installed to U2RIB

Fabric Path Control Plane initialization flowS101# show fabricpath isis route

Fabricpath IS-IS domain: default MT-0

Topology 0, Tree 0, Swid routing table

1, L1

via Ethernet6/27, metric 40

2, L1

via Ethernet6/28, metric 40

200, L1

via Ethernet6/27, metric 80

via Ethernet6/28, metric 80

...

----------------------------------------------------------------------------

S101# show fabricpath route

FabricPath Unicast Route Table

'a/b/c' denotes ftag/switch-id/subswitch-id

'[x/y]' denotes [admin distance/metric]

ftag 0 is local ftag

FabricPath Unicast Route Table for Topology-Default

...

1/102/0, number of next-hops: 1

via Po1, [115/40], 1 day/s 10:01:12, isis_fabricpath-default

1/200/0, number of next-hops: 2

via Eth6/27, [115/80], 1 day/s 10:02:32, isis_fabricpath-default

via Eth6/28, [115/80], 0 day/s 10:20:17, isis_fabricpath-default

Processes start (isis, u2rib, m2rib, drap)

System ID obtained from backplane MAC

Switch ID is obtained from DRAP

As FP interfaces links come up, hellos sent and adjacencies formed

Switch ID conflicts (if any) resolved

FP Interfaces allowed to forward data

Unicast SPF is calculated

Routes installed to U2RIB

How to read

To reach switch 200 in topology 1

send packets to either Eth6/27 or

Eth6/28

FabricPath IP Multicast

• Control plane:• IGMP snooping operates as usual in FabricPath edge switches

• FabricPath IS-IS learns multicast group membership from IGMP snooping on edge switch

• FabricPath edge switch announces group interest by using GM-LSPs, creating “pruned trees” for each group on each multidestination tree

• Data plane:• Hardware selects which multidestination tree to use for each flow based on hash

function

• Once tree is selected, traffic constrained to pruned tree (FTAG) for that IP multicast group, based on MAC table lookup

Key FabricPath Multicast Processeson the Supervisor Engine:

• FabricPath IS-IS routing protocol that forms the core of the FabricPath control plane

• DRAP Dynamic Resource Allocation Protocol, extension to FabricPath IS-IS that ensures network-wide unique and consistent Switch IDs and FTAGs

• IGMP Provides IGMP snooping support for building multicast forwarding database

• M2RIB Multicast Layer 2 RIB, contains the multicast Layer 2 routing information

• L2FM Layer 2 forwarding manager, controls the MAC address table

• MFDM Multicast forwarding distribution manager, connects platform-independent control-plane processes and platform-specific processes on I/O modules

on the Linecards:

• M2FIB – Multicast Layer 2 FIB, manages the hardware multicast routing table

• MTM – MAC table manager, manages the hardware MAC address table

MAC TableSwitch Table Other HW

I/O Module

SupervisorEngine

IGMP

Hardware Drivers

M2FIB

FabricPath IS-IS

MFDM

M2RIB L2FM

MTM

DRAP

FabricPath Multicast Control Plane• IGMP/IGMP snooping tracks connected hosts/routers interest in

receiving multicast

• ISIS distributes information from igmp snooping to other FP nodes using GM-LSPs. Intermediate nodes flood GM-LSPs

• A pruned subtree is created for each group (+flood, OMF) per vlan per FTAG

MAC A MAC B

S10 S30

S1 S2

S20MAC A MAC B

S10 S30

S1 S2

S20

Root

Tree1

Root

Tree2

Vlan FTAG MAC Switches Interfaces

1 1 0100.5e01.0203 S10,S30 E1/1

Vlan FTAG MAC Switches Interfaces

1 1 0100.5e01.0203 S10,S30 E1/10,E1/30

1 2 0100.5e01.0203 S10,S30 E1/2

Vlan FTAG MAC Switches Interfaces

1 1 0100.5e01.0203 S10,S30 E1/1

1 2 0100.5e01.0203 S10,S30 E1/10,E1/30

E1/10 E1/30

E1/2

E1/1 E1/2

E1/10 E1/30

E1/1

Receiver

239.1.2.3

S10 S30

S1 S2

S20Receiver

239.1.2.3

Source

239.1.2.3

Vlan FTAG MAC Switches Interfaces

1 2 0100.5e01.0203 S10,S30 E1/2

STP & FabricPath

• No STP inside FP network

• BPDUs do not traverse FP network

(dropped at FP edge, with the exception of TCNs, see next slide)

• FP network pretends to be 1 switch from STP point of view: all FP edge

switches send BPDUs with the same Bridge ID c84c.75fa.60xx (xx is domain ID

in hex, default 00)

• Before FP ports are up, switch will use its own Bridge ID

(like STP without FP would do)

• Ports inside FP cannot be blocked, FP edge switches will always want to have

STP designated role, if superior BPDU is received such port will be blocked as

L2GW inconsistent

FabricPath

N7K# show spanning-tree interface e3/1 detail

Port 385 (Ethernet3/1) of VLAN2000 is broken (L2 Gateway Backbone Port Inconsistent)Designated root has priority 34768, address c84c.75fa.6000

N7K(config)# spanning-tree vlan 2000 priority 8192

22:27:28 %STP-2-L2GW_BACKBONE_UNBLOCK: L2 Gateway Backbone port inconsistency cleared unblocking port Ethernet3/1 on VLAN2000.

STP, FabricPath & TCNs• When CE STP domains are connected to multiple FP switches STP

TCN handling might be needed to maintain accuracy of MAC address tables inside CE

• Example if link CE1-CE2 goes down, link CE2-CE3 will become forwarding. Now to reach MAC B, switches inside FP need to send traffic to S5 instead of S4…

• To achieve this, FP switches when receiving a TCN from CE will propagate it to all FP switches in the network (via ISIS)

• Each FP switch will flush all remote MAC addresses learned from switches in the same STP domain as domain originating the TCN

• In addition, if FP switch is also part of the same STP domain, it will propagate TCN to the CE domain

• TCNs are not propagated to CE in domain 0 (default domain)

MAC A

MAC B

S1

S3

FabricPath

S3

S4 S5

STP Domain 1

STP Domain 2

CE1 CE3CE2

T

C

N

N7K# conf t

N7K(config)# spanning-tree domain ?

<1-1023> Domain Identifier

N7K# sh spanning-tree summary

Switch is in rapid-pvst mode

L2 Gateway Domain ID: 100...

X

Flush MACs learned from

S4,S5

T

C

N

Flush MACs learned on CE

T

C

N

T

C

N

T

C

N

T C N

Control Plane Protection• Both N7K, N6K, and N5K recognize and protect FP ISIS traffic at COPP level

• COPP needs to be updated when deploying FabricPath; standard profiles are FP-aware as of 5.2(1)

• In case of complex CE-side STP topologies (with blocking ports), usual STP safeguards are recommended (Bridge Assurance & Dispute / UDLD)

• On N7K-F1 cards: rate-limiters allow up to 4500 PPS worth of control plane FabricPath packets

Note: These 4500 PPS include also transit packets

N7K# show policy-map interface control-plane

Control Plane

service-policy input: copp-policy-strict

class-map copp-class-critical (match-any)

match access-group name

copp-acl-mac-fabricpath-isis

set cos 7

police cir 39600 kbps , bc 250 ms

module 1 :

conformed 5136527710 bytes; action: transmit

violated 0 bytes; action: drop

7KN5K# show policy-map interface control-plane class

copp-system-class-isis

Control Plane

service-policy input: copp-system-policy-default

class-map copp-system-class-isis (match-any)

match protocol isis_dce

police cir 1024 kbps , bc 4800000 bytes

conformed 751957 bytes; action: transmit

violated 0 bytes;

5K

6K

• FabricPath Overview

Benefits, Restrictions, and Configuration

• Key Concepts

Encapsulation, Trees, Topologies, STP

• Data Plane

Forwarding, Load-Balancing, MAC Learning

• vPC+

Challenges and Operation

• Troubleshooting

Verification steps, tools, and examples

Agenda

FabricPath: Forwarding Tables

• FabricPath uses 3 tables to forward frames

• MAC address table

VLAN, MAC Address, Port (local or remote), FTAG (for non-unicast)

• Switch-ID table

remote switch-ID, local next-hop interfaces (up to 16)

• Multidestination tree table

Per Tree: remote switch-ID, local next-hop/RPF interface

Tree#1 (broadcast, unknown unicast, IP multicast)

Tree#2 (IP multicast)

MAC TableSwitch Table

I/O Module

SupervisorEngine

U2FIB

FabricPath IS-IS

U2RIB L2FM

MTM

DRAP

Other HW

Hardware Drivers

Forwarding: unicast CEFP

DAKnown

ODA = L2_lookup (DA)

Forward

SAKnown

ODA = MC2 (FF)

Ftag = F(Vlan,SA/DA,…)

Ftag == Vlan2Ftag(Vlan)

ODA = MC1 (Flood2BD)

TTL = 32

unicast

OSA.SW/SubSW = local OSA.LID=LID(ingress_port)

N

Y

N

Y

Unknown unicast

Unknown source Flood to update MACs

Choose FTAG

This is meant to illustrate key decisions in forwarding, some details are abstracted away

FTAG for unicast

is topology ID

DA = Destination Address

SA = Source Address

ODA = Outer Destination Address

OSA = Outer Source Address

MC1 = 010F.FFC1.01C0

MC2 = 010F.FFC1.02C0

Forwarding: broadcast/multicast CEFP

Frame is flooded on CE side as well (based on DA)

Each egress port decides whether to encapsulate the frame in MIM depending on port type (FP,CE)

Forward

Ftag = Hash(Vlan,SA/DA,…)

TTL = 32

BC || MC

OSA.SW/SubSW = local OSA.LID=LID(ingress_port)

Broadcasts are flooded along FTAG1* Exception in vPC+

ODA = DA

Forwarding: FP->FP or FP->CE

Multicast lookups are done using VLAN, FTAG, and ODA(each multicast mac appears twice)

SubSwitchID lookups are omitted here

Remember about special LIDs (Sup, Flood, …)

FF frames are forwarded out of CE ports only when DA is locally learned

TTL<1

Forward

ODA is unicast

Dest = LID orDest = L2_table(DA,VLAN)

NY

RPF is checked against

OSA.SwID + FTAGDecrement(TTL)

RPF

Y

N

Drop

Destination =

L2_Table(Vlan, FTAG, ODA)

Pass

Fail

ODA.SwIDis local

Destination =

Sw_Table(FTAG, ODA.SwID)

N

Y

MIM packet

Load-balancing

• N7K: Unicast and Multicast load-balancing are separate

• N5K/N6K: Unified load-balancing mechanism for unicast and multicast

N7K# show fabricpath load-balance

ECMP load-balancing configuration:

L3/L4 Preference: Mixed

Hash Control: Symmetric

Rotate amount: 6 bytes

Use VLAN: TRUE

Ftag load-balancing configuration:

Hash Control: Symmetric

Rotate amount: 6 bytes

Use VLAN: TRUE

N7K# show fabricpath load-balance unicast forwarding ftag 1 switchid 30 flow l2 src-mac 001c.57ad.ecc3

dst-mac 547f.ee02.ce3c ether-type 0x800 vlan 2000 module 3

128b Hash Key generated : 1ffb80b38f02000019000715eb7b30d5

This flow selects interface Eth3/25

• Symmetric: idea is to make ab and baflows take same path by sorting addresses, before feeding them to hash

• Rotate: polarization avoidance; hash result is rotated by specified number of bytes. Number is derived from unique system MAC

Reducing impact of forwarding loops

• Transient loops might occur during convergence (as with L3 routing)

• To contain impact of these loops FabricPath uses TTL. Starting in 6.2(2), can set the initial TTL via fabricpath [multicast | unicast] ttl

• For Multidestination Trees Reverse Path Forwardingcheck performed on source switch ID

Nexus5k# show platform fwm info asic-errors 0

DROP_TTL_EXPIRED: res0 = 23 res1 = 0 [10]

Nexus7K-F2# show hardware internal errors module 4 | inc ign ttl

47 Ingress redirect due to dtag_ttl check 0000000000000002 41-44 -

MAC Address Learning

• Learning MAC addresses is not required in FabricPath Core as switching is based on Switch ID

• FP Edge switches learn local MAC addresses (behind edge ports) conventionally

• FP Edge devices learn remote addresses (behind Core-facing ports) using conversational learning

• For packets arriving from FP, source MAC (not outer SA!) is learned when destination MAC of the frame is already known on any Edge port of this switch

• No learning from broadcasts (though existing entries will be updated)

• Normal Learning from multicasts (example: HSRP address)

Conversational learning is

disabled on L3 edge

switches (when SVI is up

on FP VLAN)

This does not apply to a

case where F-series is

connected to M-series in

different VDC by external

cable

When M and F are in the

same VDC, special

handling is needed to

forward packets from

MFP core – this is

orchestrated by MCM

(mixed chassis manager)

Conversational MAC Address Learning

A B

• A sends an ARP for B (broadcast)

S1 S2 S3

A BS1 S2 S3

MAC Port

A 1

MAC Port MAC Port

• B sends ARP reply (unicast) to A

A BS1 S2 S3

MAC Port

A 1

B S3.0.1

MAC Port MAC Port

B 1

• A sends unicast packet to B

A BS1 S2 S3

MAC Port

A 1

B S3.0.1

MAC Port MAC Port

B 1

A S1.0.1

MAC Port MAC Port MAC Port

FabricPath Scale

Leaf Layer Optimized conversational learning

Spine No MAC learning (forwarding based on SWID)

VLAN 100

Leaf

VLAN 100VLAN 200 VLAN 200

SpineL2

L3

VLAN 100

Leaf

VLAN 100VLAN 200 VLAN 200

Spine

L3 Spine

Leaf Layer Optimized conversational learning

Spine Learns all MAC addresses in order to route between VLANs

Nexus5500 Nexus6000 N7K-F1 N7K-F2 N7K-F3 N7K-M

series

32K MACs 128K MACs* 16K MACs

per SoC

16K MACs

per SoC

64K MACs

per SoC

128K MACs

Potential bottleneck if

F1/F2 used in L3 Spine

FabricPath Proxy L2 Learn

• Goal: Increase MAC table size in FabricPath for F1/F2E modules

• Solution: Offload MAC learning to M-series module at L2/L3 boundary

• Prerequisites: 6.2(2) on N7K (Spine and Leaf) , M1/M2 + F2E or M1/M2 + F1

L2

L3

VLAN 100

Leaf

VLAN 100VLAN 200 VLAN 200

Spine

SoC

M1/M2 Learn All

Remote MACs

No MAC

Learning

! From default VDC (Prevents F2E/F1 from learning on multicast frames)

no hardware fabricpath mac-learning module <x> [port-group <y>]

! From fabricpath VDC (prevents F2E/F1 from learning remote MACs)

no mac address-table fabricpath remote-learning

Configuration

! If you are using F2 for Leaf core ports to prevent learning from

broadcast/multicast

no hardware fabricpath mac-learning module <x> [port-group <y>]

FabricPath MAC Learning Changes: Why?

• M-Series MAC tables contain VLAN, MAC, and port index (no concept of SWID, SSWID, LID in M-Series MAC table)

• For FP MACs, the destination SWID is mapped to an internal gateway port-channel (GPC) index which is programmed in the M-series MAC table

• FP SoC will translate GPC to SWID before sending out FP port.

• Challenge: No way for FP SoC to determine LID for packet from M-Series module if MAC is not present in local MAC table. Therefore, packet from M-Series sent out FP with flood LID.If FP SoC on destination switch has not learned MAC, then packet will be flooded out local CE ports

• Solution: Sync MACs on CE SoC to FP SoC.

FP

SoC

FP

SoC

CE

SoC

VLAN 100

M1/M2

L2

L3

FP

SoC

VLAN 200

CE

SoC

X, Y, Z A, B, C

S1

S101 S201

VLAN MAC Index

200 A gpc1

GPC SWID

gpc1 S201

M sends

frame to gpc1

F translates

frame to

SWID 201, LID

FFFFMAC miss,

causes flood to

local CE ports

FabricPath MAC Learning Changes

FP

SoC

FP

SoC

CE

SoC

CE

SoC

Learn all MACs on CE

ports. Learn remote

MACs via

conversational learning

No MACs

Learned

FP

SoC

FP

SoC

CE

SoC

CE

SoC

Learn all MACs on CE

ports. Learn remote

MACs via

conversational learning

X, Y, Z X, Y, Z

Sync local CE

MACs to FP SoC

Learns MAC

X,Y,Z

Learns MAC

A,B,C

6.1(2) for F2/F2E

6.2(2) in F1

• To support L2 proxy learning, MACs learned on CE ports will be synced to all SoCs

A, B, CA, B, C

• FabricPath Overview

Benefits, Restrictions, and Configuration

• Key Concepts

Encapsulation, Trees, Topologies, STP

• Data Plane

Forwarding, Load-Balancing, MAC Learning

• vPC+

Challenges and Operation

• Troubleshooting

Verification steps, tools, and examples

Agenda

Fabric Path

VPC+: Why, What and How (1)

• Goal: provide redundant, active-active L2 links to separate FP switcheswith active-active HSRP

• Challenge 1: depending on the path the packet AB takes, switch S3 will learn MAC A behind S1 or S2 (or MAC will be moving)

• Solution: introduce Emulated Switch S100 to represent devices behind VPCs: MAC A will appear behind S100 in S3 MAC address table. HSRP MAC is advertised with emulated switch as a source – taking advantage of VPC+ multipathing

44

S1 S2

S3

MAC A

MAC B

S100

S3# show mac address-table address 0000.0000.000a

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+-------------------

3000 0000.0000.000a dynamic 30 F F 100.0.0

S3# show fabricpath route switchid 100

1/100/0, number of next-hops: 2

via e1/1, [115/20], 1 day/s 05:56:40, isis_fabricpath-default

via e1/2, [115/20], 1 day/s 05:56:38, isis_fabricpath-default

S3# show fabricpath switch-id

SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED

----------+----------------+------------+-----------+--------------------

1 0000.0000.1001 Primary Confirmed Yes No

2 0000.0000.3002 Primary Confirmed Yes No

*3 0000.0000.3003 Primary Confirmed Yes No

100 0000.0000.1010 Primary Confirmed No Yes

Fabric PathVPC VPC+

• To enable VPC+ an Emulated Switch ID must be configured in VPC domain on both peers (must be the same on both peers and globally unique). ES represents ALL VPC+ channels of the domain

• Peer-link and VPC+ ports must be fabric-path capable

• Peer-link is FP interface(no STP, only FP vlans are carried, VPC check is no more ).VPC+ channels are CE

• VPC+ domain must be the root for CE STP, otherwise VPC+ channels will be blocked as L2GW inconsistent

• FP switches use same STP bridge ID but peer-switch is still recommended

S1 S2

S100

S1# show vpcvPC domain id : 2

vPC+ switch id : 100Peer status : peer adjacency formed ok

vPC keep-alive status : peer is alive

vPC fabricpath status : peer is reachable through fabricpath

...

vPC role : primary

Number of vPCs configured : 0

...

Fabricpath load balancing : Disabled

Port Channel Limit : limit to 244

vpc domain 2fabricpath switch-id 100

Fabric Path

HSRP (and VRRP) in VPC+

• HSRP when enabled on VPC+ peers uses Emulated Switch ID as a source switch and thus benefits from VPC+ multipathing

• Control-plane-wise one peer will be active and other will be standby, but data-plane-wise both peers will be forwarding traffic (same as in VPC)

• FabricPath devices will have ECMP route to Emulated Switch

• CE devices will have HSRP VMAC pointing to a port-channel

• If only HSRP active-active is required VPC+ channels are optional

S1 S2

S3

S100

S3# show mac address-table vlan 100 address 0000.0c9f.f064VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

100 0000.0c9f.f064 dynamic 0 F F 100.0.65535

s3# show fabricpath route switchid 1001/100/0, number of next-hops: 2

via e1/1, [115/20], 1 day/s 05:56:40, isis_fabricpath-defaultvia e1/2, [115/20], 1 day/s 05:56:38, isis_fabricpath-default

CE1# show mac address-table vlan 100 address 0000.0c9f.f064VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 100 0000.0c9f.f064 dynamic 0 F F Po1

CE1

VPC+: Why, What and How (2)

• Challenge 2: flooded packets from A (with OSA of S100) might come to S3 from S1 or from S2, but RPF can only be 1 interface

• Solution: S1 and S2 advertise to S3 (via ISIS TLV) an affinity to single FTAG each, S3 will program RPF according to affinity. Multidestination traffic coming from VPC+ will be set to use FTAG 1 for VPC leg on S1 and FTAG 2 for VPC leg on S2

47

S1 S2

S3

MAC A

MAC B

S100

1/1 1/2

Affinity

FTAG1

Affinity

FTAG2

Use FTAG1 Use FTAG2

RPF

FTAG1,S100

RPF

FTAG2,S100

S3# show fabricpath route switchid 100FabricPath Unicast Route Table1/100/0, number of next-hops: 2

via Eth1/1, [115/40], 11 day/s 00:59:35, isis_fabricpath-defaultvia Eth1/2, [115/40], 11 day/s 01:03:27, isis_fabricpath-default

S3# show fabricpath isis database detail | i Affinity|Host|NumgHostname : S1 Length : 2

Affinity :Nickname: 100 Numgraphs: 1 Graph-id: 1

Hostname : S2 Length : 2Affinity :Nickname: 100 Numgraphs: 1 Graph-id: 2

S3# show l2 multicast trees

(ftag/2, topo/0, Switch-id 100), uptime: 1d01h, isisOutgoing interface list: (count: 1, '*' is the preferred interface)

* Interface Ethernet1/2, [admin distance/115] uptime: 1d01h, isis

(ftag/1, topo/0, Switch-id 100), uptime: 6d00h, isisOutgoing interface list: (count: 1, '*' is the preferred interface)

* Interface Ethernet1/1, [admin distance/115] uptime: 6d00h, isis

VPC+: Why, What and How (3)

• Challenge 3: multidestination packets from FP to CE need to be load-balanced too

• Solution: S1 and S2 will each be ‘designated forwarder’ for FTAG of their affinity: traffic for FTAG of affinity will be forwarded out of VPC and other FTAG traffic will be forwarded by peer

S1 S2

S3

MAC A

MAC B

S100

1/1 1/2

Affinity

FTAG1

Affinity

FTAG2

DF: FTAG1 DF: FTAG2

RPF

FTAG1,S100

RPF

FTAG2,S100

Po101

S1# show vpcvPC domain id : 100vPC+ switch id : 100...vPC Peer-link status---------------------------------------------------------------------1 Po1 up 2000-2001,3000-3001

vPC status-------------------------------------------------------------------------id Port Status Consistency Reason Active vlans vPC+ Attrib-- ---------- ------ ----------- ------ ------------ -----------101 Po101 up success success 10 DF: Yes

vPC status-------------------------------------------------------------------------id Port Status Consistency Reason Active vlans vPC+ Attrib-- ---------- ------ ----------- ------ ------------ -----------101 Po101 up success success 10 DF: Partial

vpc domain 100

fabricpath multicast load-balance

Fabric Path

VPC+: Prevention of Duplicate Packets

• How is packet received from VPC+ and flooded on S1 prevented from being flooded on S2 to same VPC+ again?

• N7K-F1 linecards:

Each VPC+ will have its own sub-switch ID. Mac addresses will be learned behind <es_id>.<subsw_id>.<lid>, for example 100.11.65535(emulated switch 100, sub-switch 11, LID 65535). S2 will recognize ES + SubSwitch tuple as its own port and will not flood the frame back to VPC

• N7K-F2, N7K-F3 linecards & N5K, N6K:

By default same as above, as below with ‘fabricpath multicast load-balance’

Each VPC+ peer will be forwarding only for 1 FTAG and traffic coming from other peer will have different FTAG. For example (previous slide) flooded packet coming from S1 will have FTAG1, but S2 will only flood FTAG2 packets out of the VPC

S1 S2

X

Required for FEX FP with N7K-F2

Fabric Path

VPC Failover

• VPC+ member link goes down• Traffic diverted over Peer-Link

• Peer-Link goes down (but Peer-Keepalive up)• Primary: No action

• Secondary: Bring down VPC+ channels

• Stop advertising reachability to Emulated Switch

• Dual active is much less likely than with normal VPC: if Peer-Link and Peer-Keepalive go down, but peer is reachable via FP – secondary will not become primary

S1 S2

S3

S100

S3# show fabricpath route switchid 100

1/100/0, number of next-hops: 1

via e1/1, [115/20], 1 day/s 07:14:24, isis_fabricpath-default

S1# show vpcvPC domain id : 2

vPC+ switch id : 100

Peer status : peer adjacency formed ok

vPC keep-alive status : peer is alive

vPC fabricpath status : peer is reachable through fabricpath

L2

L3

Fabric Path

Anycast HSRP

• Goal: provide N-gateway solution to increase redundancy and bandwidth

• Alternatives: 1. vPC/vPC+ provides 2 active gateways. Failure of a single gateway reduces available inter-vlan traffic by

half

2. GLBP allows more than 2 active gateways. Drawbacks:

• No ECMP load-balancing since a single virtual MAC is assigned to a single SwitchID

• Non-deterministic distribution of virtual MAC addresses (hard to troubleshoot)

• Solution: Anycast HSRP

Active Standby Listen Listen

All 4 devices actively

routing traffic for the HSRP

virtual MAC

L2

L3

Anycast HSRP

• The HSRP virtual MAC is bond to an Anycast SwitchID (ASID)

• ASID uses similar concept to vPC+ ES, where each Anycast gateway advertises the ASID via new Anycast HSRP Sub-TLV

• Each Anycast gateway will actively route traffic for the HSRP virtual MAC

feature interface-vlan

feature hsrp

interface Vlan100

ip address 10.1.100.1/24

hsrp version 2

hsrp 100

ip 10.1.100.254

interface Vlan101

ip address 10.1.101.1/24

hsrp version 2

hsrp 101

ip 10.1.101.254

hsrp anycast 1 ipv4

switch-id 1000

vlan 100-101

no shutdown

Configure HSRP under the

interface - HSRP version2

required

Configured the ASID for this

anycast bundle and

associate vlans

S1 S2 S3 S4

Code RequirementN7K

• 6.2(6)

N5K/N6K

• 6.0(2)N2(1) (SubTLV only)

• 7.0(0)N1(1)

4 Equal Cost

Routes to ASID

ASIDASIDASIDASID

Anycast HSRP

S202# show mac address-table dynamic

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 101 0000.0c9f.f065 dynamic 10 F F 1000.0.65535

* 100 0000.0c9f.f064 dynamic 10 F F 1000.0.65535

S202# show fabricpath isis database detail | i "LSPID|00-00|Nickname: 1000"

LSPID Seq Number Checksum Lifetime A/P/O/T

S1.00-00 0x00000100 0x815E 762 0/0/0/1

Nickname: 1000 Numgraphs: 2 Graph-id: 1, 2

S2.00-00 0x00000103 0xC618 776 0/0/0/1

Nickname: 1000 Numgraphs: 2 Graph-id: 1, 2

...

S202# show fabricpath route switchid 1000

...

1/1000/0, number of next-hops: 4

via Eth1/6, [115/40], 0 day/s 03:00:18, isis_fabricpath-default

via Eth1/7, [115/40], 0 day/s 03:02:55, isis_fabricpath-default

via Eth1/8, [115/40], 0 day/s 03:01:08, isis_fabricpath-default

via Eth1/9, [115/40], 0 day/s 03:03:45, isis_fabricpath-default

Each switch sends ISIS

TLVs advertising ASID

ECMP routes built toward

ASID to increase

redundancy and

bandwidth

HSRP Active Hellos are

sent out with a OSA of the

ASID and SA of the virtual

MAC

• FabricPath Overview

Benefits, Restrictions, and Configuration

• Key Concepts

Encapsulation, Trees, Topologies, STP

• Data Plane

Forwarding, Load-Balancing, MAC Learning

• vPC+

Challenges and Operation

• Troubleshooting

Verification steps, tools, and examples

Agenda

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

FabricPath: Configuration

install feature-set fabricpath

feature-set fabricpath

vlan 100-199

mode fabricpath

fabricpath switch-id 101

vpc domain 100

fabricpath switch-id 100

fabricpath multicast load-balance

! Fabricpath core ports

interface Ethernet6/4 - 5

switchport mode fabricpath

! Peer-link

interface port-channel1

switchport mode fabricpath

! vPCs are CE ports (mode access or mode trunk)

interface port-channel20

switchport

switchport mode trunk

vpc 20

Best practice to manually

configure switch-id

! S1

fabricpath domain default

root-priority 255

! S2

fabricpath domain default

root-priority 254

Configure roots for

FTAG 1 and 2

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

FabricPath: Health Check

S101# sh sys internal plugin info global | begin l2mp | head lines 5

Feature-set id: 2, name: l2mp

vdc: 1 state: PLUGIN_ENABLED_STATE

vdc: 2 state: PLUGIN_ENABLED_STATE

vdc: 3 state: PLUGIN_ENABLED_STATE

FabricPath plugin in good

state

Services running for URIB,

MRIB, DRAP, ISIS

CPU levels are reasonable

Memory below limits

S101# show system internal sysmgr service all | i 2rib|drap|fabric|PID

Name UUID PID SAP state Start count Tag Plugin ID

isis_fabricpath 0x41000243 6475 436 s0009 1 N/A 1

drap 0x0000024E 6476 448 s0009 1 N/A 1

m2rib 0x00000250 6435 449 s0009 1 N/A 1

u2rib 0x00000254 6474 452 s0009 1 N/A 1

S101# show processes cpu | i 2rib|drap|fabric|PID

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

6435 410 335 1 0.00% 0.00% 0.00% - m2rib

6474 170 735 0 0.00% 0.00% 0.00% - u2rib

6475 690 3764 0 0.00% 0.00% 0.00% - isis_fabricpath

6476 200 725 0 0.00% 0.00% 0.00% - drap

S101# show processes memory | i 2rib|drap|fabric|PID

PID MemAlloc MemLimit MemUsed StackBase/Ptr Process

6435 11149312 923422860 273965056 ffd8cb40/ffffffff m2rib

6474 3657728 564849190 262389760 ffbc5b80/ffffffff u2rib

6475 30515200 814058995 479059968 ff8eed50/ffffffff isis_fabricpath

6476 3067904 619628416 262160384 ffa58950/ffffffff drap

FabricPath: Health CheckS101# show fabricpath isis

System ID : 8478.ac0e.4743 IS-Type : L1 Fabric-Control SVI: Unknown

Process is up and running

Interfaces supported by Fabricpath IS-IS :

port-channel1

Ethernet6/27

Ethernet6/28

S101# show fabricpath topology vlan active

Topo-Description Topo-ID Active VLAN List

-------------------------------- --------- -------------------------

0 0 100-199

ISIS is running

system ID is accurate

Interface list matches configuration

Active Vlans match configuration

Interfaces in Up/Ready state

Adjacencies established

Adjacencies stable

S101# show fabricpath isis interface brief

Fabricpath IS-IS domain: default

Interface Type Idx State Circuit MTU Metric Priority Adjs/AdjsUp

--------------------------------------------------------------------------------

port-channel1 P2P 3 Up/Ready 0x01/L1 1500 40 64 1/1

Ethernet6/27 P2P 1 Up/Ready 0x01/L1 1500 40 64 1/1

Ethernet6/28 P2P 2 Up/Ready 0x01/L1 1500 40 64 1/1

S101# show fabricpath isis adjacency detail

Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:

System ID SNPA Level State Hold Time Interface

S102 N/A 1 UP 00:00:25 port-channel1

Up/Down transitions: 1, Last transition: 3w5d ago

Circuit Type: L1

Topo-id: 0, Forwarding-State: UP

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

FabricPath: Health Check

No growing errors on interfaces

All switches and ES are seen and in

confirmed state

S101# show fabricpath isis traffic port-channel 1

Fabricpath IS-IS domain: default

Fabricpath IS-IS Traffic for port-channel1:

PDU Received Sent RcvAuthErr OtherRcvErr ReTransmit

P2P-IIH 734 733 0 0 n/a

CSNP 2 1 0 0 n/a

PSNP 113 113 0 0 n/a

LSP 131 134 0 0 0

S101# show fabricpath switch-id

FABRICPATH SWITCH-ID TABLE

Legend: '*' - this system

'[E]' - local Emulated Switch-id

'[A]' - local Anycast Switch-id

Total Switch-ids: 10

=========================================================================

SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED/

ANYCAST

--------------+----------------+------------+-----------+----------------

1 8478.ac0e.4742 Primary Confirmed Yes No

2 8478.ac5b.2b42 Primary Confirmed Yes No

[E] 100 8478.ac0e.4743 Primary Confirmed No Yes

100 8478.ac5b.2b43 Primary Confirmed No Yes

* 101 8478.ac0e.4743 Primary Confirmed Yes No

102 8478.ac5b.2b43 Primary Confirmed Yes No

200 547f.eed6.70fc Primary Confirmed No Yes

200 547f.eedb.7e7c Primary Confirmed No Yes

201 547f.eed6.70fc Primary Confirmed Yes No

202 547f.eedb.7e7c Primary Confirmed Yes No

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

FabricPath: Unicast Example (MAC)

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

S101# show mac-address-table address-table vlan 100

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link,

(T) - True, (F) - False

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 100 0000.0000.000a dynamic 0 F F Po30

100 0000.0000.000d dynamic 0 F F 200.0.0

S101# show hardware mac address-table 6 vlan 100

FE | Valid| PI| BD | MAC | Index| Stat| SW | ... | SWID| SSWID| LID

| | | | | | ic | | ... | | |

---+------+---+------+---------------+-------+-----+-----+ ... |-----|------|-------

7 1 1 245 0000.0000.000a 0x00408 0 0x089 0x064 0x00b 0x00408

7 1 0 245 0000.0000.000d 0x00000 0 0x009 0x0c8 0x000 0x00000

7K

vPC30 vPC40

MACs are present in software

MAC table

Use Platform Dependent

commands to check hardware

MAC table

On S101, MAC D matches

software remote address (200.0.0)

MAC A has local SWID/SSWID

100.11 with LID 0x408

Hex SWID/SSWID

0xc8 0x00 = 200 0

0x64 0x0b = 64 11

S101# show system internal pixm info ltl 0x408

PC_TYPE PORT LTL RES_ID LTL_FLAG CB_FLAG MEMB_CNT

------------------------------------------------------------------------------

Normal Po30 0x0408 0x1600001d 0x00000000 0x00000002 1

7K

LID 0x408 maps to local Po30

FabricPath: Unicast Example (MAC)

MACs are present in software

MAC table

Use Platform Dependent

commands to check hardware

MAC table

On S202, MAC A matches

software remote address

(100.11.65535)

MAC A has local SWID/SSWID

200.0 with LID 0x15 (0x15 = 21)

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

vPC30 vPC40

S202# show mac address-table vlan 100

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 100 0000.0000.000a dynamic 0 F F 100.11.65535

* 100 0000.0000.000d dynamic 0 F F Po40

S202# show platform fwm info lif port-channel 40 | i local_id

Po40 pd: local_id 21 endnode_id 0 endnode_id_alloced 1 vif_id 0

5K

S202# show platform fwm info hw-stm | i HW|VLAN|_|---|000a|000d

HW STM Contents

dleft loc - bucket_type:line:bucket_number

misc - learn_type:ecc:valid:fcf

cdce format - ig:ul:switch_id:subswitch_id:end_node_id:pbp_idx:local_id

VLAN MAC Address Port loc misc cdce

------+----------------+--------------+--------+-------+--------------------

1.100 0000.0000.000d Po40 1:1111:0 1:0:1:0 2.0.c8.0.0.15 (e:0)

1.100 0000.0000.000a l2mp-nh 1:2918:0 1:0:1:0 2.0.64.b.ff.ff (e:0)

5K

LID 21 maps to local Po40

FabricPath: What command comes from where

MAC TableSwitch Table

I/O Module

SupervisorEngine

U2FIB

FabricPath IS-IS

U2RIB L2FM

MTM

DRAP

Other HW

Hardware Drivers

show fabricpath switch

show fabricpath isis switch

show fabricpath conflict all | link | switch | transitions

show fabricpath isis route

show mac address-table

slot <> show fabricpath unicast routes vdc

slot <> show hardware internal forwarding inst <> table <>

slot <> show hardware mac address-table

show fabricpath isis interface

show fabricpath isis adjacency

show fabricpath isis database

show fabricpath route

show platform fwm info l2mp route ftag <> switch <> hw

show platform fwm info hw-stm

7K

5K

7K

6K

5K 6K

FabricPath: Unicast Example (SWID)

Route for destination SWID present in

ISIS table and U2RIB

S101# show fabricpath isis route

Fabricpath IS-IS domain: default MT-0

Topology 0, Tree 0, Swid routing table

...

200, L1

via Ethernet6/27, metric 80

via Ethernet6/28, metric 80

S101# show fabricpath isis database detail

Fabricpath IS-IS domain: default LSP database

LSPID Seq Number Checksum Lifetime A/P/O/T

S201.00-00 0x00000006 0xF8A7 957 0/0/0/1

Hostname : S201 Length : 4

Capability : Device Id: 201 Base Topology

Affinity :

Nickname: 200 Numgraphs: 1 Graph-id: 1

Nickname :

Priority: 0 Nickname: 201 BcastPriority: 64

Priority: 0 Nickname: 200 BcastPriority: 0

S202.00-00 0x00000007 0x5F3B 884 0/0/0/1

Hostname : S202 Length : 4

Capability : Device Id: 202 Base Topology

Affinity :

Nickname: 200 Numgraphs: 1 Graph-id: 2

Nickname :

Priority: 0 Nickname: 202 BcastPriority: 64

Priority: 0 Nickname: 200 BcastPriority: 0

S101# show fabricpath route switchid 200

FabricPath Unicast Route Table

'a/b/c' denotes ftag/switch-id/subswitch-id

'[x/y]' denotes [admin distance/metric]

...

1/200/0, number of next-hops: 2

via Eth6/27, [115/80], 0 day/s 00:21:58,

isis_fabricpath-default

via Eth6/28, [115/80], 0 day/s 00:21:58,

isis_fabricpath-default

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

vPC30 vPC40

FabricPath: Unicast Example (SWID)

Use Platform Dependent commands to

verify route for destination SWID is

present in hardware

On N7K, first attach to appropriate

module via “attach module x”S202# show platform fwm info l2mp route ftag 1 swid 100

-------------------------------------------------------------------

l2mp_route[0x99f23ac]

route_type: 10 (0xa) merge_version: 1 (0x1)

iic interface: Eth1/7 (0x1a006000)

ftag: 1 (0x1) switchid: 100 (0x64)-> l2mp_nexthop[0x8944dc4]

num_paths: 2

nh[1]: Eth1/7 (0x1a006000)

nh[2]: Eth1/8 (0x1a007000)

5K

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

vPC30 vPC40

module-6# show fabricpath unicast routes vdc 3 ftag 1 switchid 200

Route in VDC 3

--------------------------------------------------------------------------------

FTAG | SwitchID | SubSwitchID | Loc/Rem | RPF | RPF Intf | Num Paths | Merge V

--------------------------------------------------------------------------------

0001 | 0200 | 0000 | Remote | Yes | Eth6/27 | 2 | 1

--------------------------------------------------------------------------------

...

PD Information for ECMP:

Common Info

--------------------------------

AMM key : 0x6000024

--------------------------------

Next Hop | Interface | LID

--------------------------------

0 | Eth6/27 | 0000006a

1 | Eth6/28 | 0000006b

7K

Two equal costs routes via

Eth6/27 and Eth6/28. RPF interface Eth6/27

Two equal costs routes via

Eth1/7 and Eth1/8. RPF interface Eth1/7

FabricPath: what comes from where

show fabricpath isis switch

show fabricpath mroute

show ip igmp snooping groups

show fabricpath isis topology summary

show fabricpath isis tree

show fabricpath isis database mgroup detail

show l2 multicast trees

MAC TableSwitch Table Other HW

I/O Module

SupervisorEngine

IGMP

Hardware Drivers

M2FIB

FabricPath IS-IS

MFDM

M2RIB

MTM

DRAP

show forwarding distribution l2 multicast [vlan <>] 7K

L2FM

S101# show fabricpath isis topology summary

FabricPath IS-IS Topology Summary

Fabricpath IS-IS domain: default

MT-0

Configured interfaces: port-channel1 Ethernet6/27 Ethernet6/28

Max number of trees: 2 Number of trees supported: 2

Tree id: 1, ftag: 1, root system: 8478.ac0e.4742, 1

Tree id: 2, ftag: 2 [transit-traffic-only], root system: 8478.ac5b.2b42, 2

Ftag Proxy Root: 8478.ac0e.4742

FabricPath: Multidestination (Flood)

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

vPC30 vPC40

Check the topology roots for

each FTAG

Map out the active links

How to read: on which

interface in given FTAG will this

switch accept multidestination

traffic from given switch

Example: accept traffic from

switch 100 on E6/19 in FTAG1

S101# show fabricpath isis trees

MT-0

Topology 0, Tree 1, Swid routing table

1, L1

via Ethernet6/27, metric 0

2, L1

via Ethernet6/27, metric 20

102, L1

via Ethernet6/27, metric 40

200, L1

via Ethernet6/27, metric 40

201, L1

via Ethernet6/27, metric 40

202, L1

via Ethernet6/27, metric 40

S1# show fabricpath isis trees

MT-0

Topology 0, Tree 1, Swid routing table

2, L1

via port-channel1, metric 20

100, L1

via Ethernet6/19, metric 40

101, L1

via Ethernet6/19, metric 40

102, L1

via Ethernet6/20, metric 40

200, L1

via Ethernet6/21, metric 40

201, L1

via Ethernet6/21, metric 40

202, L1

via Ethernet6/22, metric 40

Repeat on each

switch to map out

complete

forwarding tree

(FTAG 1)

S101# show fabricpath mroute vlan 100 flood

(vlan/100, *, *), Flood, uptime: 02:01:06, isis

Outgoing interface list: (count: 5)

Switch-id 1, uptime: 02:01:06, isis

Switch-id 2, uptime: 02:01:06, isis

Switch-id 102, uptime: 01:59:40, isis

Switch-id 201, uptime: 02:01:06, isis

Switch-id 202, uptime: 02:01:06, isis

FabricPath: Multidestination (Flood)

Flood entry – traffic that will be flooded to all active ports

(minus receiving port) in a Vlan

(remember about dynamic pruning)

Ignore multiple appearances of the same interface

(interface appears 1 per destination switch)

S101# show fabricpath mroute vlan 100 flood resolved

(ftag/2, vlan/100, *, *), Flood, uptime: 02:01:32, isis

Outgoing interface list: (count: 5)

Interface Ethernet6/28, Switch-id 1, uptime: 02:01:31, isis

Interface Ethernet6/28, Switch-id 2, uptime: 02:01:31, isis

Interface Ethernet6/28, Switch-id 102, uptime: 02:00:07, isis

Interface Ethernet6/28, Switch-id 201, uptime: 02:01:31, isis

Interface Ethernet6/28, Switch-id 202, uptime: 02:01:31, isis

(ftag/1, vlan/100, *, *), Flood, uptime: 02:01:32, isis

Outgoing interface list: (count: 5)

Interface Ethernet6/27, Switch-id 1, uptime: 02:01:31, isis

Interface Ethernet6/27, Switch-id 2, uptime: 02:01:31, isis

Interface Ethernet6/27, Switch-id 102, uptime: 02:00:07, isis

Interface Ethernet6/27, Switch-id 201, uptime: 02:01:31, isis

Interface Ethernet6/27, Switch-id 202, uptime: 02:01:31, isis

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

vPC30 vPC40

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

FabricPath: IP Multicast

vPC30 vPC40

*,G from local IGMP snooping

Local IGMP/snooping entries are

redistributed into FP

L2 multicast prune subtrees built

on each FP switch

S101 hashes multicast to FTAG 1

(remember vPC+ affinity)

Multicast Receiver

Multicast Sender

S202# show ip igmp snooping groups vlan 100

Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core port

Vlan Group Address Ver Type Port list

100 */* - RF Eth1/7

RF Eth1/8

100 239.1.1.1 v2 D Po40

S101# show fabricpath isis database mgroup detail | egrep "LSPID|Group|00-01"

LSPID Seq Number Checksum Lifetime A/P/O/T

S201.00-01 0x00000093 0xEA2C 1092 0/0/0/1

Group-Address : IP Multicast : Vlan : 100 Groups : 1

Group : 239.1.1.1 Sources : 0

S202.00-01 0x00000090 0xBD66 709 0/0/0/1

Group-Address : IP Multicast : Vlan : 100 Groups : 1

Group : 239.1.1.1 Sources : 0

S101# show fabricpath mroute vlan 100

(vlan/100, 0.0.0.0, 239.1.1.1), uptime: 20:35:57, isis

Outgoing interface list: (count: 2)

Switch-id 201, uptime: 20:35:57, isis

Switch-id 202, uptime: 20:35:57, isis

S101# show fabricpath mroute vlan 100 ftag 1

(ftag/1, vlan/100, 0.0.0.0, 239.1.1.1), uptime: 20:47:34, isis

Outgoing interface list: (count: 2)

Interface Ethernet6/27, Switch-id 201, uptime: 22:26:18, isis

Interface Ethernet6/27, Switch-id 202, uptime: 22:26:18, isis

Remember

RPF check

FabricPath: IP Multicast

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

vPC30 vPC40

vPC+ in partial status which

means multidestination traffic is

load-balanced between vPC peers

S201 has affinity for FTAG 1

S202 has affinity for FTAG 2

S201 will forward this frame

Multicast Receiver

Multicast Sender

S202# show vpc 40

vPC status

---------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans vPC+ Attrib

-- ---------- ------ ----------- ------ ------------ -----------

40 Po40 up success success 100-199 DF: Partial,

FP MAC:

200.0.0

S201# show fabricpath isis database detail S201.00-00 | sec Affinity

Affinity :

Nickname: 200 Numgraphs: 1 Graph-id: 1

S201# show fabricpath isis database detail S202.00-00 | sec Affinity

Affinity :

Nickname: 200 Numgraphs: 1 Graph-id: 2

QUIZ

Both S201 and S202 receive multicast

stream, who forwards out vPC 40? x

FabricPath: Hardware Multicast MAC

• Multicast MACs are stored differently from usual 0100.5exx.xxxx

F1

• Each mac appears twice: once per FTAG, use ‘show hard internal forwarding … table mac’ to find which is which

F2 module-6# show hardware mac address-table vlan <vlan> vdc <vdc> fe <fe>

FE | Valid| PI| BD | MAC | Index| Stat| SW | Modi| Age| ... | SWID| SSWID| LID

| | | | | | ic | | fied|Byte| ... | | |

---+------+---+------+---------------+-------+-----+-----+-----+----+ ... |-----|------|-------

7 1 1 245 0000.0000.000a 0x00408 0 0x009 1 199 ... 0x064 0x00b 0x00408

7 1 0 245 0000.0000.000d 0x00000 0 0x009 1 199 ... 0x0c8 0x000 0x00000

7 1 0 245 4180.0f01.0101 0x07fd8 1 0x000 0 0 ... 0x000 0x000 0x07fd8

7 1 0 245 4180.0f01.0101 0x07fda 1 0x000 0 0 ... 0x000 0x000 0x07fda

module-4# show hardware mac address-table vlan <vlan> vdc <vdc> fe <fe>

FE | Valid| PI| BD | MAC | Index|...| PV | RD| NN| UC|PI_E8| SWID| SSWID| LID

| | | | | |...| | | | | | | |

---+------+---+------+---------------+-------|...|----|---|---|---|-----|-----|------|-------

4 1 0 52 0100.ef01.0203 0x07ffb ... 0x00 0 0 0 0 0x000 0x000 0x07ffb

4 1 0 52 0100.ef04.0506 0x07ffb ... 0x00 0 0 0 0 0x000 0x000 0x07ffb

4 1 0 52 0100.ef01.0203 0x07ffb ... 0x00 0 0 0 0 0x000 0x000 0x07ffb

4 1 0 52 0100.ef04.0506 0x07ffb ... 0x00 0 0 0 0 0x000 0x000 0x07ffb

Looking back in time

• show fabricpath isis internal event-history adjacencyevents related to adjacencies (up/down/etc)

• show fabricpath isis internal event-history uribFP events related to URIB updates (for example to see whole history for given switch ID)

• show fabricpath isis internal event-history eventsOverall FP event history: DRAP interactions, switch additions, removals, SPF-related events

• show fabricpath isis internal event-history drapswitch ID, FTAG related events

Tools

Troubleshooting Tools: Pong

• Pong can be equated to L2Ping + L2TraceRoute

• Depends on IEEE 1588v2 HW support F-series, N5500, and N6000 all support PTP, but N5K/N6K at present doesn’t support pong

• Works by sending 2 types of packets: 1 packet to store timestamps at each hop and 2nd to collect stored timestamps

S101# pong destination-swid 2 destination-mac 8478.ac5b.2b42 vlan 100 details

Legend (*) - software delay(not hardware latency)

(#) - reverse path

(NA) - not available

--- ------------------------- --------------------------

Hop System-mac (switch-id) Switching time

(sec, nsec)

--- ------------------------- --------------------------

1 84-78-ac-0e-47-43 ( 101) 5588 353692400

2 84-78-ac-0e-47-42 ( 1) 5588 353692896

3 84-78-ac-0e-47-42 ( 1) 5588 353698488

4 84-78-ac-5b-2b-42 ( 2) 5588 415486312

5 84-78-ac-5b-2b-42 ( 2) 5588 930158536

6 84-78-ac-0e-47-42 ( 1) 5588 868372664

7 84-78-ac-0e-47-42 ( 1) 5588 868378248

8 84-78-ac-0e-47-43 ( 101) 5588 868378768

Round trip time: 0sec 14144 nsec

Send frame to SWID 2

(SysID of SWID 2 = 8478.ac5b.2b42)

* By default, Frame sent on VLAN 1. Be

sure to specify appropriate VLAN

Egress from SWID 101

Ingress SWID 1

Egress SWID 1

Etc..

MACs that can be reached:

- SysID or static

Not supported over ECMP on F2

Troubleshooting Tools: FPOAM

• FPOAM (Fabricpath Operations Administration and Management) is an effective tool set to monitor and diagnose data plane failures in FP networks.

• ping fabricpath

• traceroute fabricpath

• mtrace fabricpath

S202S101

ES S100 ES S200

202# mtrace fabricpath ftag 2 repeat 1

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,

'D' - Destination Unreachable, 'X' - unknown return code,

'V' - VLAN nonexistent, 'v' - VLAN in suspended state,

'm' - malformed request, 'C' - Cross Connect Error,

'U' - Unknown RBridge nickname, 'n' - Not AF,

'*' - Success, Optional Tlv incomplete,

'I' - Interface not in forwarding state,

'S' - Service Tag nonexistent, 's' - Service Tag in suspended state,

'c' - Corrupted Data/Test

Fabricpath mtrace for multicast ftag 2, vlan 1

Code SwitchId Interface State TotalTime

==================================================

! 201 Rcvd on Eth1/2 fwd 3ms

! 101 Rcvd on Eth1/2 fwd 4ms

! 102 Rcvd on Eth1/2 fwd 4ms

S102 S201

Troubleshooting Tools: FPOAM

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

202# show run fabricpath | section "oam profile 2"

fabricpath oam profile 2

vlan 100

flow forward

ether-type 0x800

ip source 100.1.1.20

ip destination 10.1.1.30

mac-address source 0000.1010.1010

mac-address destination 0000.3333.3333

protocol 1

202# traceroute fabricpath switch-id 1034 profile 2

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,

'D' - Destination Unreachable, 'X' - unknown return code,

'V' - VLAN nonexistent, 'v' - VLAN in suspended state,

'm' - malformed request, 'C' - Cross Connect Error,

'U' - Unknown RBridge nickname, 'n' - Not AF,

'*' - Success, Optional Tlv incomplete,

'I' - Interface not in forwarding state,

'S' - Service Tag nonexistent, 's' - Service Tag in suspended state,

'c' - Corrupted Data/Test

Sender handle: 14

Hop Code SwitchId Interface State TotalTime PathId

============================================================

1 ! 2 Rcvd on Eth6/2 fwd 3ms

2 ! 100 Rcvd on Eth1/1 fwd 4ms

• OAM Profiles can be used to replicate dataplane packet and follow the forwarding path

Troubleshooting Tools: Counters

S202(config)# ip access-list test-stats

S202(config-acl)# statistics per-entry

S202(config-acl)# permit ip host 10.1.100.101 host 10.1.100.201

S202(config-acl)# permit ip any any

S202(config-acl)# interface ethernet 1/7

S202(config-if)# ip port access-group test-stats in

S202(config-if)# end

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

vPC30 vPC40

S202# show ip access-lists test-stats

IPV4 ACL test-stats

statistics per-entry

10 permit ip 10.1.100.101/32 10.1.100.201/32 [match=0]

20 permit ip any any [match=0]

! Sent 5000 frames

S202# show ip access-lists test-stats

IPV4 ACL test-stats

statistics per-entry

10 permit ip 10.1.100.101/32 10.1.100.201/32 [match=5000]

20 permit ip any any [match=0]

Find the likely interface to receive packets

(note multidestination traffic might follow

different path sh fab isis trees)

Configure ACL with ‘statistics per-entry’

which explicitly matches traffic in question

Attach ACL to ingress FP port as a PACL

Check the counters

Run test traffic

Check the counters again

Compare

Troubleshooting Tools: CountersS1# attach module 6

module-6# show hardware internal dev-port-map

--------------------------------------------------------------

CARD_TYPE: 48 port 10G

FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF

...

19 4 4 4 4 4 0

20 4 4 4 4 4 0

21 5 5 5 5 5 0

22 5 5 5 5 5 0

...

module-6# test fabricpath unicast configure route-stats vdc 2 ftag 1 switchid 200 fe 5 table [mp | sw] commit

module-6# show fabricpath unicast route-stats vdc 2 ftag 1 switchid 200 fe 5

------------------------------------------

| VDC | FTAG | SwitchID | SubSwitchID |

-------------------------------------------

| 002 | 0001 | 0200 | 000 |

| FE | Adjacency | Statistics |

| 4 | Eth6/21| 0000000000 |

| 4 | Eth6/22| 0000000000 |

module-6# show fabricpath unicast route-stats vdc 2 ftag 1 switchid 200 fe 5

------------------------------------------

| VDC | FTAG | SwitchID | SubSwitchID |

-------------------------------------------

| 002 | 0001 | 0200 | 000 |

| FE | Adjacency | Statistics |

| 4 | Eth6/21| 0000000000 |

| 4 | Eth6/22| 0000000064 |

Find ingress interface & attach to

respective linecard

Find Ingress FE instance

Configure statistics (use FE+1)

Print statistics

Run traffic

Print statistics again – note statistics

are in HEX

Compare

Use MP table to get per next-hop stat

if there is >1 next-hop, else use SW

table

Troubleshooting Tools: Error/Drop Counters

• Usual datapath troubleshooting apply on N7K

• And on N5K/N6K

N5K# sh platform fwm info pif e1/5 | i stats|cdce

Eth1/5 pd: tx stats: bytes 304069130 frames 913992 discard 0 drop 0

Eth1/5 pd: rx stats: bytes 9647836468 frames 8319249 discard 0 drop 1650

Eth1/5 pd cdce_addr: switchid 30 sub-switchid 0, endnodeid 0

Eth1/5 pd cdce_addr: Mcast 0, locally-adm 1, OutOfOrder/don't learn 0

Eth1/5 pd cdce_addr: localid 5, pbp_idx 0

N5K# sh platform fwm info asic-errors 0

Printing non zero Carmel error registers:

DROP_SRC_VLAN_MBR: res0 = 495188 res1 = 0 [12]

DROP_CDCE_SW_TBL_RPF_MISS: res0 = 4 res1 = 0 [30]

DROP_SRC_FTAG_BITMAP_MBR: res0 = 5 res1 = 0 [31]

DROP_SRC_MASK_TO_NULL: res0 = 332912 res1 = 0 [44]

7k# show hardware internal errors module 6 | diff

... send 2000 transit packets using ping with timeout 0 ...

7k# show hardware internal errors module 6 | diff

< 1008 Self-forwarding check OSA drop 0000000287061579 3 –

> 1008 Self-forwarding check OSA drop 0000000287063630 3 -

< 2514 Ingress packets marked with drop_oth sent to IB 0000000002127119 4 –

> 2514 Ingress packets marked with drop_oth sent to IB 0000000002127173 4 -

< 50 smallcnt DSWID/DSSWID miss and DCE frame, def-gw disabled 0000000000000563 5-6 –

> 50 smallcnt DSWID/DSSWID miss and DCE frame, def-gw disabled 0000000000002563 5-6 -

show hardware internal errors

often produces lengthy outputs, use

diff to just see what has changed

between 2 timed samples

(with some test traffic in the middle)

PIF (physical interface) maintains RX/TX

and drop counters

Check if drops are non-zero & growing

(also check the ASIC number)

Use ASIC-errors command to get a

breakdown of drop reasons (and see if any

are growing in with test/ping traffic)

Troubleshooting Tools: ELAM

• Embedded Logic Analyzer Module (ELAM) is an engineering tool that is used to look inside Cisco ASICs.

• ELAM is architecture specific and therefore will have different capabilities and different CLI syntax across different forwarding engines (FE).

• It is possible to use ELAM as a capturing tool to validate:

1. Was the packet received

2. On which interface/VLAN did the packet arrive

3. What did the packet look like

4. How was the packet altered and where was it sent

• It is not intrusive

• It can be used at a very granular level to troubleshoot a single traffic flow which can be an invaluable tool to network administrators.

• When the going gets tough…

ELAM is NOT a supported feature.

It is a diagnostic tool designed for

internal use. Anything and

everything about it may change from

version to version without any notice

Troubleshooting Tool: ELAM Workflow

Identify the expected ingress

Forwarding Engine (FE)

Configure an ELAM trigger to capture specific

frame

Start the ELAM

After ELAM triggers, display and analyze the

data

Once triggered data can be displayed and analyzed

Typical ELAM challenges

Identifying the correct capture point and trigger

Understanding the captured data (for complex cases)

Troubleshooting Tools: ELAM

• Data Bus (DBUS) and Result Bus (RBUS)

The DBUS contains several platform specific internal fields along with the header information from a frame required to make the forwarding decision. We use the DBUS information to validate where the frame was received and basic data about the frame.

The RBUS will contain information about the forwarding decision to help determine if the frame was altered and where it was sent.

• Local Target Logic (LTL)

The LTL is an index used to represent a port or group of ports. The source LTL index and the destination LTL index tell us which port the frame was received and where it was sent.

• Basics to know before performing an ELAM

Troubleshooting Tools: ELAM Example

• Packet from host 10.1.100.101 <-> 10.1.100.201, expected ingress interface Eth6/19 on N7K-F2 linecard of S1

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

vPC30 vPC40

S1# attach module 6

Attaching to module 6 ...

module-6# show hardware internal dev-port-map

+-----------------------------------------------------------------------+

+----------------+++FRONT PANEL PORT TO ASIC INSTANCE MAP+++------------+

+-----------------------------------------------------------------------+

FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF

...

19 4 4 4 4 4 0

...

module-6# elam asic clipper instance 4

module-6(clipper-elam)# layer2

module-6(clipper-l2-elam)# trigger dbus ipv4 ingress if source-ipv4-

address 10.1.100.101 destination-ipv4-address 10.1.100.201

module-6(clipper-l2-elam)# trigger rbus ingress if trig

module-6(clipper-l2-elam)# start

module-6(clipper-l2-elam)# status

L2 DBUS Triggered

L2 RBUS Triggered

Linecard L2/L3 ASIC name

M-series Eureka/Lamira

F1 Orion

F2 Clipper

F3 Flanker

Eth6/19 is on FE instance 4

(code name clipper)

Configure a trigger specific to

this source/destination IP

Start the ELAM, send the

traffic and wait for it to trigger

Troubleshooting Tools: ELAM Example

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

vPC30 vPC40

module-6(clipper-l2-elam)# show dbus

<snip>

port-id : 0x2 last-ethertype : 0x800

vlan : 0x64 destination-index : 0x0

source-index : 0x62 bundle-port : 0x0

status-is-1q : 0x1 trill-encap : 0x0

mac-in-mac-valid : 0x1 dtag-ttl : 0x20

recirc-acos : 0x0 dtag-ftag : 0x1

source-ipv4-address: 10.1.100.101

destination-ipv4-address: 10.1.100.201

mim-destination-mac-address: 0200.c800.0000

mim-source-mac-address: 0200.640b.ffff

destination-mac-address 0000.0000.000d

source-mac-address: 0000.0000.000a

module-6(clipper-l2-elam)# show rbus

<snip>

di-ltl-index : 0x65 l3-multicast-di : 0x0

source-index : 0x62 vlan-id : 0x64

dtag=ftag : 0x1 dtag-ttl : 0x1f

mim-destination-mac-address: 0200.c800.0000

mim-source-mac-address: 0200.640b.ffff

Frame received on VLAN 100 (0x64) from a

source-index of 0x62 (next slide)

mac-in-mac valid (this is a FP frame)

dtag-TTL: fabricpath TTL of 32 (0x20)

ODA (0c8.00.0000) = 200.0.0

OSA (064.0b.ffff) = 100.11.65535

Frame transmitted on vlan 100 (0x64) to a destination

index of 0x65 (next slide)

dtag-TTL: fabricpath TTL decremented to 31 (0xf1)

Troubleshooting Tools: ELAM Example

• ELAM confirms that frame was received on Eth6/19, VLAN 100 with an OSA of 100.11.65535 and ODA of 200.0.0.

• ELAM also confirms that frame was forwarded out Eth6/22 on VLAN 100 with a decremented FP TTL

S202S201S102S101

S2S1

ES S100 ES S200

FP Vlans 100-199

A B C D

vPC30 vPC40

S1# show system internal pixm info ltl 0x62

Member info

------------------

Type LTL

---------------------------------

PHY_PORT Eth6/19

S1# show system internal pixm info ltl 0x65

Member info

------------------

Type LTL

---------------------------------

PHY_PORT Eth6/22

Get mapping of

source index to

physical port

Get mapping of

destination index to

physical port

Troubleshooting Tools: show tech

• show tech fabricpath isis

• show tech fabricpath switch-id

• show tech fabricpath topology

• Neither of these include FP routes, macs or comprehensive forwarding related data. Collect these separately:

• show tech l2fm detail

• show tech l2fm l2dbg

• show tech forwarding l2 unicast

• show tech forwarding l2 multicast

84

Troubleshooting Example: Broken HSRP

• Problem statement: HSRP active & standby do not ‘see’ each other in certain vlans. For example in vlan 1317 standby (S2) ‘sees’ the active (S1), but on active standby is unknown. A number of vlans are affected. This is new deployment.

• Initial assessment: possible reason for HSRP router not ‘seeing’ other router is HSRP hello packets not being received. In our case it is likely active router, not receiving hello packets from standby

• Quick debug on S1 confirms it only sends hellos in vlan 1317

• …and on S2 we see hellos being sent and received…

S4S3

S2S1

S1# debug hsrp engine packet hello interface vlan 1317

10:03:30 hsrp: Vlan1317[17/V4]: Hello out Active pri 100 ip 10.13.17.254

10:03:31 hsrp: Vlan1317[17/V4]: Hello out Active pri 100 ip 10.13.17.254

10:03:32 hsrp: Vlan1317[17/V4]: Hello out Active pri 100 ip 10.13.17.254

S2# debug hsrp engine packet hello interface vlan 1317

10:03:30 hsrp: Vlan1317[17/V4]: Hello in from 10.13.17.1 State Active pri 100 ip 10.13.17.254

10:03:30 hsrp: Vlan1317[17/V4]: Hello out Standby pri 50 ip 10.13.17.254

10:03:31 hsrp: Vlan1317[17/V4]: Hello in from 10.13.17.1 State Active pri 100 ip 10.13.17.254

10:03:31 hsrp: Vlan1317[17/V4]: Hello out Standby pri 50 ip 10.13.17.254

E1/1

Troubleshooting Example: Broken HSRP

• Are the HSRP frames from S2 to S1 getting lost?

• Findings so far:

• Working and Non-working packets may follow different paths

• Time to look at the Trees

S4S3

S2S1

S2# sh fabricpath load-balance multicast ftag-selected flow-type l2 dst-mac

0100.5e00.0002 src-mac 0000.0c00.0123 ether-type 800 vlan 1317 module 1

...

FTAG SELECTED IS : 2

S2# sh fabricpath isis topology summary

MT-0

Configured interfaces: port-

channel1 Ethernet1/1 Ethernet1/2

Number of trees: 2

Tree id: 1, ftag: 1, root system: 0000.0000.0002, 2

Tree id: 2, ftag: 2, root system: 0000.0000.0004, 4

S1# sh fabricpath load-balance multicast ftag-selected flow-type l2 dst-mac

0100.5e00.0002 src-mac 0000.0c07.ac11 ether-type 800 vlan 1317 module 1

...

FTAG SELECTED IS : 1

S2# show fabricpath isis trees

MT-0

Topology 0, Tree 1, Swid routing table

1, L1

via port-channel1, metric 20

...

Topology 0, Tree 2, Swid routing table

1, L1

via Ethernet1/1, metric 40

...

S1S2 FTAG 1 traffic uses Po1

(peer-link)

S1S2 FTAG 2 traffic uses E1/1

(goes through S4)

E1/1

Troubleshooting Example: Broken HSRP

• S4 is transit switch for HSRP S2 S1 traffic, hence we will not see packets in debug. We need to look at the data plane level if hello packet arrives/leaves.

• Options: SPAN, Counters, ELAM

• Let’s try hardware counters…

S4# show hardware internal errors module 1

...

|------------------------------------------------------------------------|

| Device:Orion Fwding Driver Role:L2 Mod: 1 |

| Last cleared @ Thu Apr 11 11:11:11 2011

| Device Statistics Category :: ERROR

|------------------------------------------------------------------------|

Instance:0

ID Name Value Ports

-- ---- ----- -----

29 smallcnt Pkt dropped due to CBL 0000000000001227 1-2 -

2014 Ingress packets marked with drop_oth sent to IB 0000000000001227 1 –

S4# show hardware internal errors module 1 | diff

...wait some seconds...

S4# show hardware internal errors module 1 | diff

< 29 smallcnt Pkt dropped due to CBL 0000000000001229 1-2 –

> 29 smallcnt Pkt dropped due to CBL 0000000000001235 1-2 -

CBL drops grow at about the rate of

HSRP hellos. CBL stands for Color

Blocking logic (or Vlan Blocking

Logic). Essentially, hardware logic

defining whether given port/vlan is

blocking or forwarding packets.

E1/1

S4S3

S2S1

E1/1

S4# show fabricpath mroute vlan 1317

ERROR: Vlan 1317 does not exist

S4# show vlan id 1317

VLAN 1317 not found in current VLAN database

Root cause: Vlan missing from transit switch

All FP vlans must be defined on all FP

switches, otherwise there might be issues

similar to this for flooded traffic. ISIS will

prune off unnecessary flood traffic towards

tree branches that do not have ports behind

them.

Troubleshooting: Common Pitfalls

• All FP Vlans must be present on all FP switches

• else multicast trees might not be correct

• TCNs not propagated to required FP or CE switches. Configure STP domain where TCNs need to be propagated. Else, connectivity might be broken after re-convergence until MACs age out or are relearned

• At power up or reload, CE-side comes up faster than FP-side

• L2GW Inconsistency, ensure that FP switches have been configured with superior priority before connecting to CE switches.

CLI cheatsheet

• Interfaces in FP modeshow fabricpath isis interface [brief]

• ISIS adjacenciesshow fabricpath isis adjacency [detail]

• Root information for the treesshow fabricpath isis topology summary

• RPF information for the treesshow fabricpath isis trees

• OIFs for the treesshow fabricpath mroute

• Affinity to Ftagsshow fabricpath isis database detailshow system internal m2rib ftag

• Pongpong destination-swid <sw#> destination-mac <mac-address> vlan <vlan> count <#> … [detail]

Summary

• Core ConceptsKnown Unicast Best path with ECMP, Rest Tree-balanced

• Control PlaneISIS in the core, STP / IGMP snooping at CE

• Data PlaneMAC address table, SwitchID table, Tree table (RPF)

• TroubleshootingUnderstand what should be happening, verify what is happening, find a deviation, zoom in and repeat

90

Thank you