-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
1/87
January 31 February 3, 2011Hello LONDON!
Ozden Karakok CAE
FCoE Design and Best Practices
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
2/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 2
Agenda
Why are we here?
Background Information
FCoE Building Blocks and Terminology
DCB Standard
FCoE Protocol Information
Design Requirements
Classical Ethernet + Classical Fibre Channel = ??
Single Hop Designs
Multi-Hop Designs
FCoE Deployment Considerations
Questions
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
3/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 3
The Evolving Data Centre AccessThe Consolidated Nexus Edge Layer
slot 1sl
ot 2
slot 3sl
ot 4slot
5sl
ot 6
slot 7sl
ot 8
blade1
blade2
blade3
blade4
blade5
blade6
blade7
blade8
slot 1sl
ot 2
slot 3sl
ot 4slot
5sl
ot 6slot
7
slot 8
blade1
blade2
blade3
blade4
blade5
blade6
blade7
blade8
slot 1sl
ot 2
slot 3
slot 4slot
5sl
ot 6slot
7
slot 8
blade1
blade2
blade3
blade4
blade5
blade6
blade7
blade8
slot 1sl
ot 2
slot 3sl
ot 4sl
ot 5sl
ot 6
slot 7sl
ot 8
blade1
blade2
blade3
blade4
blade5
blade6
blade7
blade8
slot 1sl
ot 2
slot 3sl
ot 4sl
ot 5sl
ot 6sl
ot 7
slot 8
blade1
blade2
blade3
blade4
blade5
blade6
blade7
blade8
slot 1sl
ot 2
slot 3
slot 4sl
ot 5sl
ot 6sl
ot 7
slot 8
blade1
blade2
blade3
blade4
blade5
blade6
blade7
blade8
Core/AggregationLayer
VirtualizedEdge
/AccessLayer
The Access Layer is becoming more thanjust a port aggregator
Edge of the growing Layer 2 topology
Scaling of STP Edge Ports
Virtual embedded switches
vPC and loop free designs
Layer 2 Multi-Pathing (future)
Foundational element for Unified I/O andUnified Wire
DCB and Multi-Hop FCoE Support
Enhanced Multi-hop FCoE with E-NPV
Single Point for Access Management VN-Tag and Port Extension Nexus 2000
(current)
VSM and VN-Link (future)
FC
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
4/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 4
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
5/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 5
Understand the design requirements of a UnifiedNetwork
Be able to design single-hop Unified Networksavailable today which meet the demands of both
SAN and LAN networks
Start the conversation between Network andStorage teams regarding consolidation and FCoEbeyond the access layer
Understand the Operations and Managementaspects of a Unified Network
Why are we here?Session Objectives
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
6/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 6
Agenda
Why are we here?
Background Information
FCoE Building Blocks and Terminology
DCB Standard
FCoE Protocol Information
Design Requirements
Classical Ethernet + Classical Fibre Channel = ??
Single Hop Designs
Multi-Hop Designs
FCoE Deployment Considerations
Questions
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
7/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 7
FCF : Fibre Channel Forwarder (Nexus 5000, Nexus7000, MDS 9000)
FPMA : A unique MAC address that is assigned by anFCF to a single Enode
Enode: a Fiber Channel end node that is able totransmit FCoE frames using one or more Enode MACs.
FCoE Pass-Through : a DCB device capable of passingFCoE frames to an FCF (i.e. FIP-Snooping)
FIP Snooping Bridge
FCoE N-Port Virtualizer
Single hop FCoE : running FCoE between the host and
the first hop access level switch Multi-hop FCoE : the extension of FCoE beyond a
single hop into the Aggregation and Core layers of theData Centre Network
FCoE Building BlocksThe Acronyms Defined
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
8/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 8
Enode MAC AddressFibre Channel over Ethernet Addressing Scheme
Enode FCoE MAC assigned for each FCID Enode FCoE MAC composed of a FC-MAP and
FCID
FC-MAP is the upper 24 bits of theEnodes FCoE MAC
FCID is the lower 24 bits of the EnodesMAC
FCoE forwarding decisions still made based onFSPFand the FCID within the Enode MAC
FC Fabric
Domain ID
FC-MAP(0E-FC-xx)
FC-ID10.00.01
FC-MAC
Address
FC-MAP(0E-FC-xx)
FC-ID10.00.01
Fibre ChannelFCID Addressing
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
9/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 9
Show fcoe
QlogicSanSurferoutput
N5K2-60# show fcoeFCF details for interface san-port-channel 200
FCF-MAC is 00:0d:ec:a4:3b:87FC-MAP is 0e:fc:00
FCF Priority is 128FKA Advertisement period for FCF is 8 seconds
N5K2-60# show fcoe database
-------------------------------------------------------------------------------INTERFACE FCID PORT NAME MAC ADDRESS-------------------------------------------------------------------------------vfc1 0x240101 21:00:00:c0:dd:0a:b8:df 00:c0:dd:0a:b8:dfvfc201 0x240100 21:00:00:c0:dd:12:04:f2 00:c0:dd:12:04:f2
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
10/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 10
Fibre ChannelDrivers
EthernetDrivers
Operating System
PCIe
Ethernet
FibreChannel
10GbE
10GbE
Link
Ethernet Driverbound to EthernetNIC PCI address
FC Driverbound to FC
HBA PCIaddress
Replaces multiple adapters perserver, consolidating both Ethernetand FC on a single interface
Appears to the operation systemas individual interfaces (NICs andHBAs)
First Generation CNAs fromsupport PFC and CIN-DCBX
Second Generation CNAs supportPFC, CEE-DCBX as well as FIP
Single chip implementation
Half Height/Length
Half power consumption
FCoE Building BlocksConverged Network Adapter
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
11/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 11
FCF (Fibre Channel Forwarder) is the Fibre Channel switchingelement inside an FCoE switch
Fibre Channel logins (FLOGIs) happens at the FCF
Consumes a Domain ID
FCoE encap/decap happens within the FCF
Forwarding based on FC information
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Ethernet Bridge
FCport
FCport
FCport
FCport
FCF
FCoE SwitchFC Domain ID : 15
FCoE Building BlocksFibre Channel Forwarder
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
12/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 12
VE_Port
VF_Port
VF_Port
VE_Port
VN_Port
VN_Port
FibreChannel over Ethernet Switch
E_NPVSwitchVF_Port
VNP_PortFCFSwitch
End
Node
EndNode
FCoE Switch : FCF
**Available NOW
FCoE Building BlocksFCoE Port Types
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
13/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 13
Unified I/O using Ethernet as the transport medium in all networkenvironments -- no long needing separate cabling options for LANand SAN networks
Unified Wire a single DCB Ethernet link actively carrying bothLAN and Storage (FC/FCoE/NAS/iSCSI) traffic simultaneously
Unified Dedicated Wire -- a single DCB Ethernet link capable of
carrying all traffic types but actively dedicated to a single traffictype for traffic engineering purposes
Unified Fabric An Ethernet Network made up of Unified Wireseverywhere: all protocols network and storagetransverse all linkssimultaneously
FCoE Building BlocksThe New BuzzwordUnified
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
14/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 14
CNA
Unified DedicatedWire
L2
L3
Core
Aggregation
SharedAccess
Fabric A Fabric B
Unified Wire
Unified Wire to the access switch
cost savings in the reductionof required equipment
cable once for all servers tohave access to both LAN andSAN networks
Unified Dedicated Wire fromaccess to aggregation
separate links for SAN andLAN traffic - both links aresame I/O (10GE)
advanced Ethernet features
can be applied to the LANlinks
maintains fabric isolation
FCoE Building BlocksUnfied Wire vs Unified Dedicated Wire
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
15/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 15
L2
L3
Core
Aggregation
Access
Virtual Port-Channel (VPC)
Ethernet and Storage trafficEVERYWHERE
A single network
All links carry all types of trafficsimultaneously
all/any Storage and Network
protocols Possible reduction of equipment
leading to cost savings
Abolition of Fabric A and Fabric B
Single SAN fabric with
redundant fabric services
FCoE Building BlocksThe Unified Fabric - Definition
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
16/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 16
Unified Technology
L2
L3
Core
Aggregation
Access
Core
Edge
Fabric A Fabric B
Ether-channel Multi-pathing
Virtual Port-Channel (VPC)
NIC/
CNA
CNA
Fibre Channel over Ethernet
SAN
Native Ethernet LAN
LAN and SAN networks share the same Unified I/O building blocks: switches
and cabling
Maintains operations, management and troubleshooting
Takes advantage of the Ethernet Roadmap (10G40G100G)
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
17/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 17
Agenda
Why are we here?
Background Information
FCoE Building Blocks and Terminology
DCB Standard
FCoE Protocol Information
Design Requirements
Classical Ethernet + Classical Fibre Channel = ??
Single Hop Designs
Multi-Hop Designs
FCoE Deployment Considerations
Questions
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
18/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 18
Standards for I/O Consolidation
Standard / Feature Status of the Standard
IEEE 802.1QbbPriority-based Flow Control (PFC)
Passed Sponsor Ballot, awaiting publication
IEEE 802.3bdFrame Format for PFC
Passed Sponsor Ballot, awaiting publication
IEEE 802.1QazEnhanced Transmission Selection (ETS) and
Data Center Bridging eXchange (DCBX)
Entering Sponsor Ballot
IEEE 802.1Qau Congestion Notification Done!
IEEE 802.1Qbh Port Extender In its first task group ballot
Developed by IEEE 802.1 Data Center Bridging Task Group (DCB)
All technically stable
FC-BB-5 standards published by ANSI in May 2010
CEE (Converged Enhanced Ethernet) is an informal group ofcompanies that submitted initial inputs to the DCB WGs.
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
19/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 19
Whats Necessary for FCoE?
FCoE Standard REQUIRES Lossless Ethernet
PFC: necessary to guarantee Ethernetcan provide lossless transport
ETS: nice-to-have for bandwidth management andtraffic separation
QCN: NOT necessary for FCoE today
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
20/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 20
Packet
R_RDY
Fibre Channel
Transmit QueuesEthernet Link
Receive Buffers
EightVirtualLanes
One One
Two Two
Three Three
Four Four
Five Five
Seven Seven
Eight Eight
Six Six
STOP PAUSE
B2B Credits
Enables lossless Ethernet using PAUSE based on a COS as defined
in 802.1p
When link is congested, CoS assigned to no-drop will be PAUSED
Other traffic assigned to other CoS values will continue to transmitand rely on upper layer protocols for retransmission
Not only for FCoE traffic
Priority Flow ControlFCoE Flow Control Mechanism
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
21/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 21
Once feature fcoe is configured, 2 classesare made by default
Priority Flow ControlOperations Configuration Switch Level
DCB Switch
DCB CNA Adapter
class-fcoe is configured to be no-drop withan MTU of 2158
Best Practice - use thedefault COS value of 3 forFCoE/no-drop traffic
Can be changed throughQOS class-map configuration
policy-map type qos default-in-policyclass type qos class-fcoeset qos-group 1
class type qos class-defaultset qos-group 0
class type network-qos class-fcoepause no-dropmtu 2158
N5K# show class-map
Type qos class-maps===================class-map type qos match-any class-fcoematch cos 3
class-map type qos match-any class-default
match any
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
22/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 22
Create classification rules first by defining and applyingpolicy-map type qos
N5010-2(config)# class-map type qos class-losslessN5010-2(config-cmap-qos)# match cos 4N5010-2(config-cmap-qos)# policy-map type qos policy-losslessN5010-2(config-pmap-qos)# class type qos class-losslessN5010-2(config-pmap-c-qos)# set qos-group 4
N5010-2(config-pmap-uf)# system qosN5010-2(config-sys-qos)# service-policy type qos input policy-lossless
DCBX protocol tonegotiate PFC forpriority 4
Define and apply policy-map type network-qos
N5010-2(config-pmap-qos)# class type network-qos policy-losslessN5010-2(config-cmap-uf)# match qos-group 4N5010-2(config-cmap-uf)# policy-map type network-qos policy-losslessN5010-2(config-pmap-uf)# class type network-qos class-losslessN5010-2(config-pmap-uf-c)# pause no-dropN5010-2(config-pmap-uf)# system qosN5010-2(config-sys-qos)# service-policy type network-qos policy-lossless
Priority Flow ControlChanging PFC Settings
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
23/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 23
Checking the PFC settingson an interface
VL bmap = COS set for PFC
Priority Flow ControlVerifying Configurations
VL bmap Binary COS
1 00000001 0
2 00000010 1
4 00000100 2
8 00001000 316 00010000 4
32 00100000 5
64 01000000 6
128 10000000 7
show interface priority-flow-control
Shows ports where PFC is configured,the COS value associated with PFC aswell as the PAUSE packets receivedand sent on that port
N5K1# show interface priority-flow-control======================================================
=
Port Mode Oper(VL bmap) RxPPP
TxPPP
======================================================
=
Ethernet1/1 Auto On (8) 0 0
Ethernet1/2 Auto On (8) 0 0
Ethernet1/3 Auto On (8) 0 0
Ethernet1/4 Auto Off 0 0
Ethernet1/5 Auto Off 0 0
Ethernet1/6 Auto Off 0 0
Ethernet1/7 Auto On (8) 0 0
Ethernet1/8 Auto Off 0 0
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
24/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 24
Offered Traffic
t1 t2 t3
10 GE Link Realized Traffic Utilization
3G/s HPC Traffic3G/s
2G/s
3G/sStorage Traffic3G/s
3G/s
LAN Traffic4G/s
5G/s3G/s
t1 t2 t3
3G/s 3G/s
3G/s 3G/s 3G/s
2G/s
3G/s 4G/s 6G/s
Prevents a single traffic class of hogging all the bandwidth and
starving other classes
When a given load doesnt fully utilize its allocated bandwidth, it isavailable to other classes
Helps accommodate for classes of a bursty nature
Enhanced Transmission SelectionBandwidth Management
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
25/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 25
Enhanced Transmission SelectionBandwidth Management
Once feature fcoe is configured, 2classes are made by default
By default, each class is given 50%of the available bandwidth
1Gig FC HBAs
1Gig Ethernet NICs
Traditional Server
A typical server hasequal BW per traffictype
Best Practice : FCoE and Ethernet each receive 50%
Can be changed through QoS settings when higher demands forcertain traffic exist (i.e. HPC traffic, more Ethernet NICs)
N5K1# show queuing interface ethernet 1/13
Ethernet1/13 queuing information:TX Queuing
qos-group sched-type oper-bandwidth0 WRR 501 WRR 50
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
26/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 26
Create classification rules first by defining and applyingpolicy-map type qos
Define and apply policy-map type queuing to configurestrict priority and bandwidth sharing
N5010-2(config)# class-map type queuing class-voice
N5010-2(config-cmap-que)# match qos-group 2N5010-2(config-cmap-que)# class-map type queuing class-highN5010-2(config-cmap-que)# match qos-group 3N5010-2(config-cmap-que)# class-map type queuing class-lowN5010-2(config-cmap-que)# match qos-group 4N5010-2(config-cmap-que)# exitN5010-2(config)# policy-map type queuing policy-BWN5010-2(config-pmap-que)# class type queuing class-voiceN5010-2(config-pmap-c-que)# priority
N5010-2(config-pmap-c-que)# class type queuing class-highN5010-2(config-pmap-c-que)#bandwidth percent 50N5010-2(config-pmap-c-que)# class type queuing class-lowN5010-2(config-pmap-c-que)#bandwidth percent 20N5010-2(config-pmap-c-que)# class type queuing class-fcoeN5010-2(config-pmap-c-que)#bandwidth percent 30N5010-2(config-pmap-c-que)# class type queuing class-defaultN5010-2(config-pmap-c-que)#bandwidth percent 0N5010-2(config-pmap-c-que)# system qosN5010-2(config-sys-qos)# service-policy type queuing output policy-BW
N5010-2(config-sys-qos)#
Enhanced Transmission SelectionChanging ETS Bandwidth Configurations
FCoE Traffic given30% of the 10GE link
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
27/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 27
Negotiates Ethernet capabilitys PFC, ETS, CoS
values between peer devices Simplifies management of DCB nodes
Allows for configuration and distribution ofparameters from one node to another
Responsible for Logical Link Up/Down signaling ofEthernet and Fibre Channel
Uses Link Layer Discovery Protocol (LLDP) defined by802.1AB to exchange and discover DCB capabilities
DCBX negotiation failures result in:
per-priority-pause not enabled on CoS values
vfc not coming up when DCBX is being used inFCoE environment
Data Center Bridging eXchangeControl Protocol the handshake
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
28/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 28
Agenda
Why are we here?
Background Information
FCoE Building Blocks and Terminology
DCB Standard
FCoE Protocol Information
Design Requirements
Classical Ethernet + Classical Fibre Channel = ??
Single Hop Designs
Multi-Hop Designs
FCoE Deployment Considerations
Questions
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
29/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 29
Understanding FCoE
Fibre Channel is to FCoE
as is to
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
30/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 30
Mapping of FC Frames over
Ethernet
FCoE
FibreChannel
Traffic
Ethernet
Ethernet
Header
FCoE
Header
FC
Header
FC Payload CRC
EOF
FCS
Byte 0 Byte 2229
Completely based on theFC model
Same host-to-switch andswitch-to-switch behavioras FC
WWNs, FC-IDs, hard/softzoning, DNS, RSCN
FCoE is Fibre Channel
Cisco Dell EMC2 EMULEX
Microsoft
Intel QLOGIC Redhat VMWARE
NetAppHP IBM
Fiber Channel over EthernetFC-BB-5 Protocol
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
31/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 31
Mapping of FC Frames over
Ethernet
FCoE
FibreChannel
Traffic
Ethernet
Ethernet
Header
FCoE
Header
FC
Header
FC Payload CRC
EOF
FCS
Byte 0 Byte 2229
Roadmap of Ethernet
Economy of Scale
Massive industry investment
FCoE is Ethernet
Fiber Channel over EthernetFC-BB-5 Protocol
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
32/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 32
From a Fibre Channel standpoint itsFC connectivity over a new type of cable called Ethernet
From an Ethernet standpoints its
Yet another ULP (Upper Layer Protocol) to be transported
FC-0 Physical Interface
FC-1 Encoding
FC-2 Framing & Flow Control
FC-3 Generic Services
FC-4 ULP Mapping
Ethernet Media Access Control
Ethernet Physical Layer
FC-2 Framing & Flow Control
FC-3 Generic Services
FC-4 ULP Mapping
FCoE Logical End Point
Fiber Channel over EthernetProtocol Mapping
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
33/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 33
Both Protocols Have
Two different Ethertypes FIP 0x8914 , FCOE 0x8906
Two different frame formats
Both are defined in FC-BB-5
FCoE itself
Is the data plane protocol
It is used to carry most of the
FC frames and all theSCSI traffic
Uses Fabric Assigned MACaddress (dynamic)
FIP (FCoE InitializationProtocol)
It is the control plane protocol
It is used to discover the FCentities connectedto an Ethernet cloud
It is also used to login to andlogout from the FC fabric
http://www.cisco.biz/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html
Fiber Channel over EthernetData and Control plane
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
34/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 34
FIP discovers other FCoE capable devices within the
Ethernet CloudEnables FCoE adapters (CNAs) to discover FCoE switches(FCFs) on the FCoE VLAN
Establishes a virtual link with between the adapter and FCF orbetween two FCFs
FIP frames use a different Ethertype from FCoE framesmaking FIP-Snooping by DCB capable Ethernet bridges
Building foundation for future multi-hop FCoE topologiesMulti-hop refers to FCoE extending beyond a single hop oraccess switch
Today, multi-hop is achievable with a Nexus 4000 (FIP SnoopingBridge) connected to Nexus 5000 (FCF)
Fibre Channel over Ethernet ProtocolFIP: FCoE Initialization Protocol
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
35/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 35
Step 1: FCoE VLAN Discovery
FIP sends out a multicast toALL_FCF_MAC address lookingfor the FCoE VLAN
FIP frames use the native VLAN
Step 2: FCF Discovery
FIP sends out a multicast to theALL_FCF_MAC address on theFCoE VLAN to find the FCFsanswering for that FCoE VLAN
FCFs responds back with their MACaddress
Step 3: Fabric Login
FIP sends a FLOGI request to theFCF_MAC found in Step 2
Establishes a virtual link betweenhost and FCF
EnodeInitiator
FCoE Switch
FCF
VLANDiscover
y
FLOGI/FDISC
FLOGI/FDISC Accept
FCCommand
FCCommandResponses
FCoE
InitializationProtocol
(FIP)
FCoEProtocol
VLANDiscovery
FCFDiscovery
FCFDiscovery
Fiber Channel over Ethernet ProtocolFCoE Initialization Protocol (FIP)
** FIP does not carry any Fibre Channel frames
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
36/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 36
The FCoE VLAN is manually configured on the Nexus 5000
The FCF-MAC address is configured on the Nexus 5000 by default oncefeature fcoe has been configured
This is the MAC address returned in step 2 of the FIP exchange
This MAC is used by the host to login to the FCoE fabric
Fiber Channel over Ethernet ProtocolFCoE Initialization Protocol (FIP)
** FIP does not carry any Fibre Channel frames
N5K(config)# feature fcoe
N5K(config)# vlan 2
N5K(config-vlan)# fcoe vsan 2
N5K(config-vlan)# show vlan fcoe
Original VLAN ID Translated VSAN ID Association State
---------------- ------------------ -----------------
2 2 Operational
N5K# show fcoeGlobal FCF details
FCF-MAC is 00:0d:ec:d5:fe:00FC-MAP is 0e:fc:00FCF Priority is 128FKA Advertisement period for FCF is 8 seconds
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
37/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 37
Fiber Channel over Ethernet ProtocolFCoE Initialization Protocol (FIP)
Step 3 - login process: show flogi database and show fcoe database
show the logins and associated FCiDs, xWWNs and FCoE MAC addresses
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
38/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 38
Fiber Channel over Ethernet ProtocolConfiguration using Device Manager
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
39/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 39
Fiber Channel over Ethernet ProtocolHost Side FIP and DCBX Configuration
1st portion of theMAC is the FC-MAP of theNexus 5000
FC-MAP
(0E-FC-00)
FC-ID
B1.00.01
FC-MAC
Address
FC-MAP(0E-FC-xx)
FC-IDB1.00.01
2nd portion of the MACis the FC-ID
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
40/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 40
FCoE ForwardingFCF
Intermediate switches in the Ethernet cloudAll are Fibre Channel Aware
EthernetFabric
FC Fabric
FC Domain 7 FC Domain 3MAC A
FCID 7.1.1
FCID 1.1.1MAC C
D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)
FC Frame
D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)
FC Frame
EthernetFabric
FC Domain 1MAC B
FC Storage
FCoEFrame
D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)
Dest. = MAC BSrce. = MAC A
D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)
Dest. = MAC CSrce. = MAC B
FC link
VE_port VE_port VF_port VN_port
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
41/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 41
Agenda
Why are we here?
Background Information
FCoE Building Blocks and Terminology
DCB Standard
FCoE Protocol Information
Design Requirements
Classical Ethernet + Classical Fibre Channel = ??
Single Hop Designs
Multi-Hop Designs
FCoE Deployment Considerations
Questions
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
42/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 42
Ethernet is non-deterministic.Flow control is destination-based
Relies on TCP drop-retransmission / sliding window
Fibre-Channel is deterministic.
Flow control is source-based (B2B credits)
Services are fabric integrated (no loop concept)
The Design RequirementsEthernet vs Fibre Channel
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
43/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 43
Ethernet/IP
Goal is to provide any-to-anyconnectivity
Unaware of packet loss relies on ULPs forretransmission and windowing
Provides the transport without worryingabout the services -services provided byupper layers
East-west vs north-south traffic ratios areundefined
Network design has been optimized for:
High Availability from a transport perspective
by connecting nodes in mesh architectures
Service HA is implemented separately
Takes in to account control protocolinteraction (STP, OSPF, EIGRP, L2/L3boundary, etc)
?
?
?
?
???
??
?
??
Switch Switch
Switch
?
Client/ServerRelationshipsare not pre-
defined
? ?
?
Fabric topology andtraffic flows are highly
flexible
The Design RequirementsClassical Ethernet
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
44/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 44
Servers typically dual homed to twoor more access switches
LAN switches have redundantconnections to the next layer
Distribution and Core can be
collapsed into a single box
L2/L3 boundary typically deployed inthe aggregation layer
Spanning tree or advanced L2technologies (vPC) used to prevent
loops within the L2 boundaryL3 routes are summarized to the core
Services deployed in the L2/L3boundary of the network (load-balancing, firewall, NAM, etc)
L2
L3
Core
Aggregation
Access
Virtual Port-Channel (VPC)
Virtual Port-Channel (VPC)
Outside DataCentercloud
STP
STP
The Design RequirementsLAN Design Access/Aggregation/Core
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
45/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 45
Fibre Channel SAN
Transport and Services are on the samelayer in the same devices
Well defined end device relationships(initiators and targets)
Does not tolerate packet drop requires
lossless transport
Only north-south traffic, east-west trafficmostly irrelevant
Network designs optimized for Scale andAvailability
High availability of network servicesprovided through dual fabric architecture
SAN A and SAN B : physicallyseparate and redundant fabrics
Strict change isolation - end to end drivercertification
Client/Server
Relationships are
pre-defined
I(c)
I(c)
T(s)
T2
I5
I4I3I2
I1
I0
T1T0
Switch Switch
Switch
DNS FSPF
ZoneRSCN DNS
FSPF Zone
RSCN
DNS
Zone
FSPF
RSCN
Fabric topology, servicesand traffic flows are
structured
The Design RequirementsClassical Fibre Channel
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
46/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 46
Edge-Core Topology
Servers connect to the edge switches
Storage devices connect to one ormore core switches
Core switches provide storageservices to one or more edgeswitches, thus servicing more serversin the fabric
ISLs have to be designed so thatoverall fan-in ratio of servers to
storage and overall end-to-endoversubscription are maintained
HA achieved in two physicallyseparate, but identical, redundantSAN fabrics
FC
Core Core
The Design RequirementsSAN Design Two Tier Topology
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
47/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 47
Edge-Core-Edge Topology
For environments where future growthof the network has the number ofstorage devices exceeding the numberof ports available at the core switch
A set of edge switches dedicated toserver connectivity and another set ofdedicated for storage devices
Extra edge can also be services edgefor advanced network services
Core is for transport only, rarelyaccommodates end nodes
HA achieved with dual fabrics
FC
Core Core
The Design RequirementsSAN Design Three Tier Topology
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
48/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 48
T2
I5
I4I3I2
I1
I0
T1T0
Switch Switch
Switch
DNS FSPF
ZoneRSCN DNS
FSPFZone
RSCN
DNS
Zone
FSPF
RSCN
Question Do we build a FC network on topof an Ethernet Cloud? Or and EthernetNetwork on top of a Fibre Channel Fabric?
Unified Fabric design has to incorporate thesuper-set of requirements
Network -- Lossless and Loss full Topologies
Transport undefined (any-to-any) anddefined (one-to-one)
High Availability redundant network topology(mesh/full mesh) and physically separateredundant fabrics
Bandwidth FC fan-in and oversubscriptionratios and Ethernet oversubscription
Security FC controls (zoning, port security,) and IP controls (CISF, ACL, )
Manageability and Visibility Hop by hopvisibility for FC and the cloud for Ethernet
The Design RequirementsClassical Ethernet + Classical Fibre Channel == ??
? ?
?
?
????
?
?
??
Switch Switch
Switch
?
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
49/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 49
Cant we just fold down the dotted line??
FC
Core CoreL2
L3
Core
Aggregation
Access
Virtual Port-Channel (VPC)
Virtual Port-Channel (VPC)
OutsideData Center
cloud
STP
STP
The Design RequirementsClassical Ethernet + Classical Fibre Channel = ??
FoldHere
FoldHere
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
50/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 50
Agenda
Why are we here?
Background Information
FCoE Building Blocks and Terminology
DCB Standard
FCoE Protocol Information
Design Requirements
Classical Ethernet + Classical Fibre Channel = ??
Single Hop Designs
Multi-Hop Designs
FCoE Deployment Considerations
Questions
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
51/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 51
Host connected over unified wire
to first hop access switch Access switch (Nexus 5000) is the
FCF
Fibre Channel ports on the accessswitch can be in NPV or Switchmode for native FC traffic
DCBX is used to negotiate theenhanced Ethernet capabilities
FIP is use to negotiate the FCoEcapabilities as well as the hostlogin process
FCoE runs from host to accessswitch FCF native Ethernet andnative FC break off at the accesslayer
FC
CNA
FC Fabric
ENode
Target
Ethernet Fabric
DCB capable Switchacting as an FCF
Unified Wire
Single Hop DesignTodays Solution
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
52/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 52
The first phase of the Unified Fabricevolution design focused on the fabricedge
Unified the LAN Access and the SANEdge by using FCoE
Consolidated Adapters, Cabling and
Switching at the first hop in the fabrics
The Unified Edge supports multipleLAN and SAN topology options
Virtualized Data Center LAN designs
Fibre Channel edge with direct
attached initiators and targets
Fibre Channel edge-core and edge-core-edge designs
Fibre Channel NPV edge designsThe Unified Edge
Fabric A Fabric BLAN Fabric
FC
FCoEFC
LANAccess/SAN
Edge
Single Hop DesignUnified Wire at the Access
Nexus 5000FCF-A
Nexus 5000FCF-B
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
53/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 53
Fibre ChannelDrivers
EthernetDrivers
Operating System
PCIe
Ethe
rnet
FibreC
hannel
10GbE
10GbE
Link
Converged Network Adapter (CNA)presents two PCI address to theOperating System (OS)
OS loads two unique sets of driversand manages two uniqueapplication topologies
Server participates in bothtopologies separately
Two stacks and thus two views ofthe same unified wire
SAN Multi-Pathing providesfailover between two fabrics(SAN A and SAN B)
NIC Teaming provides failoverwithin the same fabric (VLAN)
Ethernet Driver
bound toEthernet NIC PCI
address
FC Driverbound to FC
HBA PCIaddress
Unified Wireshared by both
FC and IPtopologies
Nexus UnifiedEdge supportsboth FC and IP
topologies
Nexus Edge participates inboth distinct FC and IP Core
topologies
Nexus 5000FCF-A
Nexus 5000FCF-B
Single Hop DesignThe CNA Point of View
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
54/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 54
Fabric A
CEE-DCBX
Generation 1 CNA
CIN-DCBX
Generation 2 CNA
Fabric BLAN Fabric
VN
VFDirect attach
VN_Port to
VF_Port
In this first phase we were limited to direct
attached CNAs at the access Generation 1 CNA
Utilized Cisco, Intel, Nuova DataCenter Bridging Exchange protocol(CIN-DCBX)
Only supports direct attachment of anVN_Port to an VF_Port over theunified wire
Generation 2 CNA
Utilizes Converged Enhanced EthernetData Center Bridging Exchange
protocol (CEE-DCBX) Utilizes FCoE Initialization Protocol
(FIP) as defined by the T.11 FC-BB-5specification
Supports both direct and multi-hopattachment (through a Nexus 4000 FIP
Snooping Bridge)
Single Hop DesignThe CNA Point of View
Nexus 5000FCF-A
Nexus 5000FCF-A
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
55/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 55
VLAN 10,30
VLAN 10,20
A VLAN is dedicated for every
VSAN in the fabric
FIP discovers the FCoE VLAN andsignals it to the hosts
Trunking is not required on the
host driver all FCoE frames aretagged by the CNA
FCoE VLANs must not beconfigured on Ethernet links thatare not designate for FCoE
Maintains isolated edge switchesfor SAN A and B and separateLAN switches for NIC 1 and NIC 2(standard NIC teaming)
! VLAN 20 is dedicated for VSAN 2 FCoE traffic(config)# vlan 20(config-vlan)# fcoe vsan 2
VSAN 2
STP Edge Trunk
Fabric A Fabric BLAN Fabric
Nexus 5000FCF-A
Nexus 5000FCF-B
VSAN3
Single Hop DesignThe FCoE VLAN
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
56/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 56
VLAN 10,30
VLAN 10,20
In order to maintain the integrity of FC
forwarding over FCoE, FCoE VLANsare treated differently than LANVLANs
No flooding, MAC learning,broadcasts, etc.
The FCoE VLAN must not beconfigured as a native VLAN
FIP uses native VLAN
Separate FCoE VLANs must be usedfor FCoE in SAN-A and SAN-B
Unified Wires must be configured astrunk ports and STP edge ports
! VLAN 20 is dedicated for VSAN 2 FCoE traffic(config)# vlan 20(config-vlan)# fcoe vsan 2
VSAN 2
STP Edge Trunk
Fabric A Fabric BLAN Fabric
Nexus 5000FCF
Nexus 5000FCF
VSAN 3
Single Hop DesignThe FCoE VLAN
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
57/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 57
VLAN 10,30
VLAN 10,20
FCoE Fabric A will have a
different VLAN topology thanFCoE Fabric B which are differentfrom the LAN Fabric
PVST+ allows unique topology perVLAN
MST requires that all switches in thesame Region have the samemapping of VLANs to instances
MST does not require that all VLANsbe defined in all switches
A separate instance must beused for FCoE VLANs
Recommended: three separateinstances native EthernetVLANs, SAN A VLANs andSAN B VLANs
spanning-tree mst configurationname FCoE-Fabricrevision 5instance 5 vlan 1-19,40-3967,4048-4093instance 10 vlan 20-29instance 15 vlan 30-39
Fabric A Fabric BLAN Fabric
VSAN 3VSAN 2
VLAN 10
Nexus 5000FCF-A
Nexus 5000FCF-B
Single Hop DesignThe FCoE VLAN and STP
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
58/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 58
Optimal layer 2 LAN design often
leverages Multi-Chassis Etherchannel(MCEC)
Nexus utilizes Virtual Port Channel(vPC) to enable MCEC either betweenswitches or to 802.3ad attached
servers MCEC provides network based load
sharing and redundancy withoutintroducing layer 2 loops in thetopology
MCEC results in diverging LAN andSAN high availability topologies
FC maintains separate SAN A and SANB topologies
LAN utilizes a single logical topology
Direct Attach vPC Topology
MCEC
vPC Peers
vPC Peer Link
Fabric A Fabric BLAN Fabric
Nexus 5000FCF-A
Nexus 5000FCF-B
Single Hop DesignUnified Wires and MCEC
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
59/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 59
vPC enabled topologies with FCoE
must follow specific design andforwarding rules
With the NX-OS 4.1(3) releases avfc interface can only be associatedwith a vPC which has a single [one
(1)] CNA port attached to each edgeswitch
While the port-channel is the sameon N5K-1 and N5K-2, the FCoEVLANs are different
vPC configuration works with Gen-2
FIP enabled CNAs ONLY FCoE VLANs are notcarried on the
vPC peer-link
FCoE and FIP ethertypes are notforwarded over the vPC peer link
Direct Attach vPC Topology
VLAN 10,30
VLAN 10,20STP Edge Trunk
VLAN 10 ONLY HERE!
Fabric A Fabric BLAN Fabric
Nexus 5000FCF-A
Nexus 5000FCF-B
Single Hop DesignUnified Wires and MCEC
vPC contains only 2 X10GE links one to eachNexus 5000
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
60/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 60
Nexus 5000FCF-B
Nexus 5000FCF-A
VLAN 10,30VLAN 10,20
VLAN 10,20,30
Dual CNA (FC initiator) connectedvia an Etherchannel to a single edgeswitch is unsupported
A vfc interface can only bebound to a port channel withone local interface
Not consistent with FibreChannel High Availability designrequirements (No isolation ofSAN A and SAN B)
If SAN design evolves to a shared
physical with only VSAN isolation forSAN A and B this could change(currently this appears to be a big if)
ISLs between the Nexus 5000access switches breaks SAN HArequirements
Single homed dual CNADirect Attach
Topology
Fabric A Fabric BLAN Fabric
Single Hop DesignUnsupported Topologies
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
61/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 61
32 server facing 10Gig/FCoEports
T11 standard based FIP/FCoEsupport on all ports
8 10Gig/FCoE uplink ports forconnections to the Nexus 5000
Management and configurationhandled by the Nexus 5000
Support for Converged
Enhanced Ethernet includingPFC
Part of the Cisco Nexus 2000Fabric Extender family
FEX-2232Remote Line Card of
the Nexus 5000
Single Hop DesignIntroduction of 10Gig/FCoE Fabric Extender
Nexus 2232
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
62/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 62
Nexus 223210GE FEX
SAN BSAN A
Fabric LinksOption 1: Single
Homed PortChannel
Fabric LinksOption 2: Static
Pinned
Server Option 1:FCoE on
individual links.Ethernet traffic isActive/Standby
Server Ethernet driver connectedto the FEX in NIC Teaming (AFT,TLB) or with vPC (802.3ad)
FCoE runs over vPC memberport with a single link from serverto FEX
FEX singlehomed to upstreamNexus 5000
FEX fabric links can beconnected to Nexus 5000with individual links (static
pinning) or a port channel oversubscribed 4:1
Consistent with separate LANAccess and SAN EdgeTopologies
Server Option 2:FCoE on a vPC
member PC witha single link
Nexus 223210GE FEX
Requires FIP enabled CNAs
Single Hop DesignExtending the FCoE Edge Nexus 2232
Nexus 5000FCF-A Nexus 5000
FCF-B
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
63/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 63
Nexus 223210GE FEX
SAN BSAN A
Fabric Links:vPC PortChannel
Nexus 2232 can not be configured
in a dual homed configuration(vPC between two N5K) whenconfigured to support FCoEattached servers
MCEC Port Channel will not keepSAN A and San B traffic isolated
Nexus 2000 not supported withdedicated FCoE and dedicated IPupstream fabric links
Nexus 2232 can only currently beconnected to the Nexus 5000 when
configured to support FCoEattached servers
Nexus 7000 will support Nexus2000 in Ethernet only mode inCY2010 (support for FCoE onFEX targeted for CY2011 on next
generation N7K line cards)
Nexus 223210GE FEX
Nexus 5000 Nexus 7000
Single Hop DesignExtending the FCoE Edge Nexus 2232
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
64/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 64
Agenda
Why are we here?
Background Information
FCoE Building Blocks and Terminology
DCB Standard
FCoE Protocol Information
Design Requirements
Classical Ethernet + Classical Fibre Channel = ??
Single Hop Designs
Multi-Hop Designs
FCoE Deployment Considerations
Questions
What is NPIV? And Why?
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
65/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 65
What is NPIV? And Why?
N-Port ID Virtualization (NPIV) provides a means to assign multipleFCIDs to a single N_Port
Limitation exists in FC where only a single FCID can be handed out per F-port.Therefore and F-Port can only accept a single FLOGI
allows multiple applications to share the same Fiber Channeladapter port
usage applies to applications such as VMWare, MS Virtual Serverand Citrix
Application Server FC NPIV Core Switch
Email
Web
File Services
Email I/O
N_Port_ID 1
Web I/O
N_Port_ID 2
File Services I/O
N_Port_ID 3
F_Port
F_Port
N_Port
What is NPV? And Why?
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
66/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 66
What is NPV? And Why? N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a switch to
act like a server performing multiple logins through a single physical link
Physical servers connected to the NPV switch login to the upstream NPIVcore switch
No local switching is done on an FC switch in NPV mode
FC edge switch in NPV mode does not take up a domain ID
Helps to alleviate domain ID exhaustion in large fabrics
Nexus 5000, MDS 91xx, MDS blade switches,
UCS Fabric Interconnect FC NPIV Core Switch
Eth1/1
Eth1/2
Eth1/3
Server1
N_Port_ID 1
Server2
N_Port_ID 2
Server3
N_Port_ID 3
F_Port
N-Port
F-Port
F-PortNP-Port
M lti H D i
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
67/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 67
FCF
DCB + FIP
SnoopingBridge
What design considerations do we
have when extending FCoE beyondthe Unified Edge?
High Availability for both LAN andSAN
Oversubscription for SAN and LAN
Ethernet Layer 2 and STP design
Where does Unified Wire makesense over Unified Dedicated Wire?
Unified Wire provides for sharing of asingle link for both FC and Ethernet
traffic
Fabric A Fabric BLAN Fabric
Multi - Hop DesignConsiderations for FCoE Multi-hop
M lti H D i
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
68/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 68
Multi-hop FCoE networks allow for
FCoE traffic to extend past theaccess layer (first hop)
In Multi-hop FCoE the role of atransit Ethernet bridge needs to beevaluated
Avoid Domain ID exhaustion
Ease management
FIP Snooping is a minimumrequirement suggestedin FC-BB-5
Fibre Channel over Ethernet NPV(FCoE-NPV) is a new capabilityintended to solve a number ofdesign and managementchallenges
FCF
FCF
SAN BSAN A
DCB CapableEthernet
Switch
DCB CapableEthernetSwitch
VN
VN
VF
VF
Multi - Hop DesignFCoE Pass-through options
M lti H D i
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
69/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 6969
What is FIP-Snooping? Efficient, automatic configuration of ACLs
locks down the forwarding path goingfrom CNA to FCF
accomplished by snooping FIP packets
Why FIP-Snooping? Security - Protection from MAC Address
spoofing of FCoE end devices (ENode)
Fibre Channel links are Point-to-Point
Ethernet bridges can utilize ACLs to provide theequivalent path control (equivalent of point-to-point)
Support for FIP-Snooping?Nexus 4000 (Blade switch for IBM BC H)
Multi - Hop DesignFIP-Snooping
FCF
FCF MAC
0E.FC.00.DD.EE.FF
FIP Capable Multi-Hop
Topology
FIPSnooping
ENode
Spoofed MAC
0E.FC.00.DD.EE.FF
ENode MAC
0E.FC.00.07.08.09
SAN
M lti H D i
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
70/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 70
On the control plane (FIP ethertype), an Fibre Channelover Ethernet NPV bridge improves over a "FIPsnooping bridge" by intelligently proxying FIP functionsbetween a CNA and an FCF
- takes control of how a live network will build FCoEconnectivity
- makes the connectivity very predictable, without the need foran FCF at the next hop from the CNA
On the data plane (FCoE ethertype), an FCoE NPVbridge offers more ways to engineer traffic betweenCNA-facing ports and FCF-facing ports
An FCoE-NPV bridge knows nothing about FibreChannel, and cant parse packets with FCoE ethertype
Multi - Hop DesignFibre Channel over Ethernet NPV Bridge
M lti H D i
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
71/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 7171
Proxys FIP functions between a CNA and an FCFFCoE VLAN configuration and assignment
FCF Assignment
FCoE-NPV load balance logins from the CNAs evenly
across the available FCF uplink portsFCoE-NPV will take VSAN into account when mapping or
pinning logins from a CNA to an FCF uplink
Operations and management process are in line withtodays SAN-Admin practices
Similar to NPV in a native Fibre Channel network
Multi - Hop DesignFibre Channel over Ethernet NPV Bridge
M lti Hop Design
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
72/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 72
FC
Target
FABRIC A
FCoE_NPV bridge
FCFDomain ID andFC-MAP comefrom the FCF
FCoE Pass through deviceAll FCoE Switching is performed atthe upstream FCF
Addressing is pass out by theupstream FCF
more FCoE connectivity to hostswithout
Running into the domain ID issue
Less-expensive
Consistent management
FCoE-NPV is the FIP-SnoopingPlus
FLOGI
VN
FCoE_NPVdoes notconsume adomain ID
E_NodeMAC Address
VF
VF
VNP
Multi - Hop DesignFCoE-NPV - Enode Login Process
Multi Hop Design
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
73/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 73
SAN BSAN A
Nexus 4000 is a Unified Fabric
capable Blade Switch
DCB enabled
FIP Snooping Bridge
Dual Topology requirements forFCoE multi-hop
Servers IP connection to the Nexus4000 is Active/Standby
MCEC is not currently supportedfrom blade server to Nexus 4000
Options 1: Unified Dedicated Wiresfrom Nexus 4000 to Nexus 5000
Options 2: Single Unified Wire PortChannel from Nexus 4000 to Nexus5000
Option 2: SingleHomed Unified
Wire
MezzanineConverged Network
Adapter
Option 1:Unified
Dedicated Wire
PCIe
Ethernet
FibreChannel
10GbE
10GbE
Link
Multi - Hop DesignExtending FCoE with FIP Snooping
Nexus 5000FCF-A
Nexus 5000FCF-B
Nexus 4000FIP Snooping
Bridge-B
Nexus 4000
FIP SnoopingBridge-A
Multi Hop Design
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
74/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 74
DCB + FIPSnooping
Bridge
Extending FCoE Fibre Channel
fabrics beyond direct attach initiatorscan be achieved in two basic ways
Extend the Unified Edge (Stage 1)
Add DCB enabled Ethernetswitches between the VN and
VF ports (stretch the linkbetween the VN_Port and theVF_Port)
Extend Unified Fabric capabilitiesinto the SAN Core
Leverage FCoE wires betweenFibre Channel of Ethernetswitches (VE_Ports)
Fabric ALAN Fabric
Using FCoE
for ISL
between FC
Switches
Extending
FCoE into a
multi-hop
Ethernet
Access Fabric
VN
VF
VE
VE
Fabric B
VE
VE
Multi - Hop DesignExtending FCoE with VE_Ports
Nexus 5000FCF-A
Nexus 5000FCF-A
MDS 9000FCF-A
Multi Hop Design
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
75/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 75
FCoE-NPV
SAN BSAN A
FCoE-NPV
Two basic design options are
possible when we deploy any FCoEmulti-hop configuration
Option 1 Unified Dedicated Wire
Allows MCEC for IP/Ethernet
Dedicated FCoE links forStorage
Option 2Unified Wire
Leverage Server side failovermechanisms for both SAN andLAN
Allows for Unified Wire beyondthe Server to first device
Option 2: SingleHomed Unified
Wire
Option 1:Dedicated Linksand Topologies
Multi - Hop DesignExtending FCoE with FCoE-NPV
Nexus 5000FCF-A
Nexus 5000FCF-B
Multi Hop Design
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
76/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 76
FCF
FIP and
FcoEframes loadshared over
MCEC on a perflow basis
NO SAN A and
SAN B isolation
SAN BSAN A SAN and LAN high availability design
requirements are not always identical
Optimal layer 2 LAN design may notmeet FC high availability andoperational design requirements
Features such as vPC & MCEC are
not viable and not supported beyondthe direct attached server
Server has two stacks andmanages two topologies
Layer 2 network has a single
topology L2MP and TRILL provide options to
change the design paradigm andcome up with potential solutions
FCoE over L2MP/TRILL is not
currently supported
DCBEnabled
Multi - Hop DesignUnsupported Topologies
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
77/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 77
Agenda
Why are we here?
Background Information
FCoE Building Blocks and Terminology
DCB Standard
FCoE Protocol Information
Design Requirements
Classical Ethernet + Classical Fibre Channel = ??
Single Hop Designs
Multi-Hop Designs
FCoE Deployment Considerations
Questions
FCoE Deployment Considerations
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
78/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 78
Where is it efficient to leverage unified wire, shared links for both SAN
and LAN traffic?At the edge of the fabric the volume of end nodes allows for a greater degree ofsharing for LAN and SAN
In the core we will not reduce the number of links and will either maintainseparate FC or FCoE links to the SAN core and Ethernet links to the LAN core
LAN and SAN HA models are very different (and not fully compatible) FC and FCoE are prone to HOLB in the network and therefore we are
limited in the physical topologies we can build
e.g. 10 x 10G uplinks to LAN aggregation will require 10 x 10G links to a nexthop SAN core (with targets attached) No savings, actually spending more toachieve this direct uplinks to SAN core
Targets are attached to the SAN core (the LAN aggregation and SAN corehave different topology functions)
Where is it more beneficial to deploy two cores SAN and LAN over aunified core topology
FCoE Deployment ConsiderationsDedicated Aggregation/Core Devices
FCoE Deployment Considerations
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
79/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 79
Migration to 10G FCoE in place of
4/8G FC links (Ethernet price perbit economics)
Edge switch running as FCF withVE_ports connecting to FCF onCore switch
Must be careful of Domain ID creeping
FSPF forwarding for FCoE trafficis end-to-end
Hosts will log into the FCF which
they are attached to (accessFCF)
Storage devices will log into theFCF at the core/storage edge
Maintains HA requirements from
both LAN and SAN perspective
VE Ports
SAN BSAN A
FCoE Deployment ConsiderationsMigration Strategy for FCoE
Nexus 5000FCF-B
MDS 9000FCF-B
FCoE Deployment Considerations
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
80/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 80
Migration to 10G FCoE in place of
4/8G FC links (Ethernet price per biteconomics)
Edge switch running either as FCFin NPV mode or in FCoE-NPVmode with FCF migrating to theSAN Core
Fibre Channel over Ethernet NPV(FCoE-NPV) is a new constructintended to solve a number ofsystem management problems
Using FCoE_NPV alleviatesdomain ID issue
HA planning for the SAN siderequired
Does loosing a core switch mean the lossof a whole fabric?
FCF
FCF
FCoE-NPV
SAN BSAN A
FCoE Deployment ConsiderationsMigration Strategy for FCoE
FCoE Deployment Considerations
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
81/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 81
SAN BSAN A Does passing FCoE traffic through a
larger aggregation point makesense?
Multiple links required to support theHA models
1:1 ratio between access toaggregation and aggregation to SAN
core is required
SAN is more vulnerable to HOLB soneed to plan for appropriate capacityin any core ISL
When is a direct Edge to Core linksfor FCoE are more cost effective
than adding another hop?
Smaller Edge device more likely tobe able to use under-provisioneduplinks
1:1 Ratio of links
required unless
FCoE-NPVFCoE uplink is
over-provisioned
CORE
Congestion on
Agg-Core links
will HOLB all
attached edge
devices
FCoE Deployment ConsiderationsShared Aggregation/Core Devices
FCoE Deployment Considerations
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
82/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 82
v
Different requirements for LAN
and SAN network designs
Factors that will influence thisuse case
Port density
Operational roles and change
management
Storage device types
Potentially viable for smallerenvironments
Larger environments will need
dedicated FCoE SAN devicesproviding target ports
Use connections to a SAN
Use a storage edge of otherFCoE/DCB capable devices
Direct Attach
FCoE Targets
CORE
Multiple VDCs
FCoE SAN
LAN Agg
LAN Core
FCoE Deployment ConsiderationsShared Aggregation/Core Devices
Nexus 5000
FCF-A
Nexus 5000
FCF-B
FCoE Deployment Considerations
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
83/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 83
FCFFCF
SAN BSAN A
Topology will vary based on scale
(single vs multiple tiers) Architecture as defined for product
development has a dual core
Question - where is the demarkbetween Unified Wire and UnifiedFabric
As the topology grows less UnifiedWire
In all practical terms the edge isthe unified point for LAN and SAN(not the core/agg)
In smaller topologies where core
and edge merge then everythingcollapses but the essential designelements remain
Dedicated SAN and
LAN Core
VLAN 10,30VLAN 10,20
CORE
FCoE Deployment ConsiderationsDedicated Aggregation/Core Devices
FCoE Deployment Considerations
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
84/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 84
Servers, FCoEattached Storage
p yLarger Fabric Multi-Hop Topologies
Multi-hop edge/core/edge topology
Core SAN switches supportingFCoE
N7K with DCB/FCoE line cards
MDS with FCoE line cards(Sup2A)
Edge FC switches supporting either N5K - FCoE-NPV with FCoE
uplinks to the FCoE enabledcore (VNP to VF)
N5K or N7K - FC Switch withFCoE ISL uplinks (VE to VE)
Scaling of the fabric (FLOGI, )will most likely drive the selection ofwhich mode to deploy
N7K or MDS FCoE
enabled FabricSwitches
FC AttachedStorage
Servers
VE
Edge FCFSwitchMode
VE
Edge Switch inFCoE-NPV
Mode
VF
VNPVE
VE
So Remember
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
85/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 85
So Remember
All Unified options are important and have differentplaces within the Data Center Network
FCoE offers a more flexible and cheaper deploymentoption over Fibre Channel
FCoE IS Fibre Channel
Multi-hop FCoE extends the FCoE fabric beyond theaccess
Cisco offers end-to-end FCoE solution with Nexusplatform
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
86/87
2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 86
Question and Answer
Thanks for attending this session.
-
8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF
87/87