5 architecture awareness fcoe design and best practices pdf

8/2/2019 5 Architecture Awareness FCoE Design and Best Practices PDF

1/87

January 31 February 3, 2011Hello LONDON!

Ozden Karakok CAE

FCoE Design and Best Practices


2/87

2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 2

Agenda

Why are we here?

Background Information

FCoE Building Blocks and Terminology

DCB Standard

FCoE Protocol Information

Design Requirements

Classical Ethernet + Classical Fibre Channel = ??

Single Hop Designs

Multi-Hop Designs

FCoE Deployment Considerations

Questions


3/87


The Evolving Data Centre AccessThe Consolidated Nexus Edge Layer

slot 1sl

ot 2

slot 3sl

ot 4slot

5sl

ot 6

slot 7sl

ot 8

blade1

blade2

blade3

blade4

blade5

blade6

blade7

blade8

slot 1sl

ot 2

slot 3sl

ot 4slot

5sl

ot 6slot

7

slot 8

blade1

blade2

blade3

blade4

blade5

blade6

blade7

blade8

slot 1sl

ot 2

slot 3

slot 4slot

5sl

ot 6slot

7

slot 8

blade1

blade2

blade3

blade4

blade5

blade6

blade7

blade8

slot 1sl

ot 2

slot 3sl

ot 4sl

ot 5sl

ot 6

slot 7sl

ot 8

blade1

blade2

blade3

blade4

blade5

blade6

blade7

blade8

slot 1sl

ot 2

slot 3sl

ot 4sl

ot 5sl

ot 6sl

ot 7

slot 8

blade1

blade2

blade3

blade4

blade5

blade6

blade7

blade8

slot 1sl

ot 2

slot 3

slot 4sl

ot 5sl

ot 6sl

ot 7

slot 8

blade1

blade2

blade3

blade4

blade5

blade6

blade7

blade8

Core/AggregationLayer

VirtualizedEdge

/AccessLayer

The Access Layer is becoming more thanjust a port aggregator

Edge of the growing Layer 2 topology

Scaling of STP Edge Ports

Virtual embedded switches

vPC and loop free designs

Layer 2 Multi-Pathing (future)

Foundational element for Unified I/O andUnified Wire

DCB and Multi-Hop FCoE Support

Enhanced Multi-hop FCoE with E-NPV

Single Point for Access Management VN-Tag and Port Extension Nexus 2000

(current)

VSM and VN-Link (future)

FC


4/87



5/87


Understand the design requirements of a UnifiedNetwork

Be able to design single-hop Unified Networksavailable today which meet the demands of both

SAN and LAN networks

Start the conversation between Network andStorage teams regarding consolidation and FCoEbeyond the access layer

Understand the Operations and Managementaspects of a Unified Network

Why are we here?Session Objectives


6/87


Agenda

Why are we here?



DCB Standard


Design Requirements


Single Hop Designs

Multi-Hop Designs


Questions


7/87 2010 Cisco and/or its affiliates. All rights reserved.Data Center Partner Webinar 7

FCF : Fibre Channel Forwarder (Nexus 5000, Nexus7000, MDS 9000)

FPMA : A unique MAC address that is assigned by anFCF to a single Enode

Enode: a Fiber Channel end node that is able totransmit FCoE frames using one or more Enode MACs.

FCoE Pass-Through : a DCB device capable of passingFCoE frames to an FCF (i.e. FIP-Snooping)

FIP Snooping Bridge

FCoE N-Port Virtualizer

Single hop FCoE : running FCoE between the host and

the first hop access level switch Multi-hop FCoE : the extension of FCoE beyond a

single hop into the Aggregation and Core layers of theData Centre Network

FCoE Building BlocksThe Acronyms Defined



Enode MAC AddressFibre Channel over Ethernet Addressing Scheme

Enode FCoE MAC assigned for each FCID Enode FCoE MAC composed of a FC-MAP and

FCID

FC-MAP is the upper 24 bits of theEnodes FCoE MAC

FCID is the lower 24 bits of the EnodesMAC

FCoE forwarding decisions still made based onFSPFand the FCID within the Enode MAC

FC Fabric

Domain ID

FC-MAP(0E-FC-xx)

FC-ID10.00.01

FC-MAC

Address

FC-MAP(0E-FC-xx)

FC-ID10.00.01

Fibre ChannelFCID Addressing



Show fcoe

QlogicSanSurferoutput

N5K2-60# show fcoeFCF details for interface san-port-channel 200

FCF-MAC is 00:0d:ec:a4:3b:87FC-MAP is 0e:fc:00

FCF Priority is 128FKA Advertisement period for FCF is 8 seconds

N5K2-60# show fcoe database

-------------------------------------------------------------------------------INTERFACE FCID PORT NAME MAC ADDRESS-------------------------------------------------------------------------------vfc1 0x240101 21:00:00:c0:dd:0a:b8:df 00:c0:dd:0a:b8:dfvfc201 0x240100 21:00:00:c0:dd:12:04:f2 00:c0:dd:12:04:f2



Fibre ChannelDrivers

EthernetDrivers

Operating System

PCIe

Ethernet

FibreChannel

10GbE

10GbE

Link

Ethernet Driverbound to EthernetNIC PCI address

FC Driverbound to FC

HBA PCIaddress

Replaces multiple adapters perserver, consolidating both Ethernetand FC on a single interface

Appears to the operation systemas individual interfaces (NICs andHBAs)

First Generation CNAs fromsupport PFC and CIN-DCBX

Second Generation CNAs supportPFC, CEE-DCBX as well as FIP

Single chip implementation

Half Height/Length

Half power consumption

FCoE Building BlocksConverged Network Adapter



FCF (Fibre Channel Forwarder) is the Fibre Channel switchingelement inside an FCoE switch

Fibre Channel logins (FLOGIs) happens at the FCF

Consumes a Domain ID

FCoE encap/decap happens within the FCF

Forwarding based on FC information

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Ethernet Bridge

FCport

FCport

FCport

FCport

FCF

FCoE SwitchFC Domain ID : 15

FCoE Building BlocksFibre Channel Forwarder



VE_Port

VF_Port

VF_Port

VE_Port

VN_Port

VN_Port

FibreChannel over Ethernet Switch

E_NPVSwitchVF_Port

VNP_PortFCFSwitch

End

Node

EndNode

FCoE Switch : FCF

**Available NOW

FCoE Building BlocksFCoE Port Types



Unified I/O using Ethernet as the transport medium in all networkenvironments -- no long needing separate cabling options for LANand SAN networks

Unified Wire a single DCB Ethernet link actively carrying bothLAN and Storage (FC/FCoE/NAS/iSCSI) traffic simultaneously

Unified Dedicated Wire -- a single DCB Ethernet link capable of

carrying all traffic types but actively dedicated to a single traffictype for traffic engineering purposes

Unified Fabric An Ethernet Network made up of Unified Wireseverywhere: all protocols network and storagetransverse all linkssimultaneously

FCoE Building BlocksThe New BuzzwordUnified



CNA

Unified DedicatedWire

L2

L3

Core

Aggregation

SharedAccess

Fabric A Fabric B

Unified Wire

Unified Wire to the access switch

cost savings in the reductionof required equipment

cable once for all servers tohave access to both LAN andSAN networks

Unified Dedicated Wire fromaccess to aggregation

separate links for SAN andLAN traffic - both links aresame I/O (10GE)

advanced Ethernet features

can be applied to the LANlinks

maintains fabric isolation

FCoE Building BlocksUnfied Wire vs Unified Dedicated Wire



L2

L3

Core

Aggregation

Access

Virtual Port-Channel (VPC)

Ethernet and Storage trafficEVERYWHERE

A single network

All links carry all types of trafficsimultaneously

all/any Storage and Network

protocols Possible reduction of equipment

leading to cost savings

Abolition of Fabric A and Fabric B

Single SAN fabric with

redundant fabric services

FCoE Building BlocksThe Unified Fabric - Definition



Unified Technology

L2

L3

Core

Aggregation

Access

Core

Edge

Fabric A Fabric B

Ether-channel Multi-pathing


NIC/

CNA

CNA

Fibre Channel over Ethernet

SAN

Native Ethernet LAN

LAN and SAN networks share the same Unified I/O building blocks: switches

and cabling

Maintains operations, management and troubleshooting

Takes advantage of the Ethernet Roadmap (10G40G100G)



Agenda

Why are we here?



DCB Standard


Design Requirements


Single Hop Designs

Multi-Hop Designs


Questions



Standards for I/O Consolidation

Standard / Feature Status of the Standard

IEEE 802.1QbbPriority-based Flow Control (PFC)

Passed Sponsor Ballot, awaiting publication

IEEE 802.3bdFrame Format for PFC

Passed Sponsor Ballot, awaiting publication

IEEE 802.1QazEnhanced Transmission Selection (ETS) and

Data Center Bridging eXchange (DCBX)

Entering Sponsor Ballot

IEEE 802.1Qau Congestion Notification Done!

IEEE 802.1Qbh Port Extender In its first task group ballot

Developed by IEEE 802.1 Data Center Bridging Task Group (DCB)

All technically stable

FC-BB-5 standards published by ANSI in May 2010

CEE (Converged Enhanced Ethernet) is an informal group ofcompanies that submitted initial inputs to the DCB WGs.



Whats Necessary for FCoE?

FCoE Standard REQUIRES Lossless Ethernet

PFC: necessary to guarantee Ethernetcan provide lossless transport

ETS: nice-to-have for bandwidth management andtraffic separation

QCN: NOT necessary for FCoE today



Packet

R_RDY

Fibre Channel

Transmit QueuesEthernet Link

Receive Buffers

EightVirtualLanes

One One

Two Two

Three Three

Four Four

Five Five

Seven Seven

Eight Eight

Six Six

STOP PAUSE

B2B Credits

Enables lossless Ethernet using PAUSE based on a COS as defined

in 802.1p

When link is congested, CoS assigned to no-drop will be PAUSED

Other traffic assigned to other CoS values will continue to transmitand rely on upper layer protocols for retransmission

Not only for FCoE traffic

Priority Flow ControlFCoE Flow Control Mechanism


21/87


Once feature fcoe is configured, 2 classesare made by default

Priority Flow ControlOperations Configuration Switch Level

DCB Switch

DCB CNA Adapter

class-fcoe is configured to be no-drop withan MTU of 2158

Best Practice - use thedefault COS value of 3 forFCoE/no-drop traffic

Can be changed throughQOS class-map configuration

policy-map type qos default-in-policyclass type qos class-fcoeset qos-group 1

class type qos class-defaultset qos-group 0

class type network-qos class-fcoepause no-dropmtu 2158

N5K# show class-map

Type qos class-maps===================class-map type qos match-any class-fcoematch cos 3

class-map type qos match-any class-default

match any


22/87


Create classification rules first by defining and applyingpolicy-map type qos

N5010-2(config)# class-map type qos class-losslessN5010-2(config-cmap-qos)# match cos 4N5010-2(config-cmap-qos)# policy-map type qos policy-losslessN5010-2(config-pmap-qos)# class type qos class-losslessN5010-2(config-pmap-c-qos)# set qos-group 4

N5010-2(config-pmap-uf)# system qosN5010-2(config-sys-qos)# service-policy type qos input policy-lossless

DCBX protocol tonegotiate PFC forpriority 4

Define and apply policy-map type network-qos

N5010-2(config-pmap-qos)# class type network-qos policy-losslessN5010-2(config-cmap-uf)# match qos-group 4N5010-2(config-cmap-uf)# policy-map type network-qos policy-losslessN5010-2(config-pmap-uf)# class type network-qos class-losslessN5010-2(config-pmap-uf-c)# pause no-dropN5010-2(config-pmap-uf)# system qosN5010-2(config-sys-qos)# service-policy type network-qos policy-lossless

Priority Flow ControlChanging PFC Settings


23/87


Checking the PFC settingson an interface

VL bmap = COS set for PFC

Priority Flow ControlVerifying Configurations

VL bmap Binary COS

1 00000001 0

2 00000010 1

4 00000100 2

8 00001000 316 00010000 4

32 00100000 5

64 01000000 6

128 10000000 7

show interface priority-flow-control

Shows ports where PFC is configured,the COS value associated with PFC aswell as the PAUSE packets receivedand sent on that port

N5K1# show interface priority-flow-control======================================================

=

Port Mode Oper(VL bmap) RxPPP

TxPPP

======================================================

=

Ethernet1/1 Auto On (8) 0 0



Ethernet1/4 Auto Off 0 0






24/87


Offered Traffic

t1 t2 t3

10 GE Link Realized Traffic Utilization

3G/s HPC Traffic3G/s

2G/s

3G/sStorage Traffic3G/s

3G/s

LAN Traffic4G/s

5G/s3G/s

t1 t2 t3

3G/s 3G/s

3G/s 3G/s 3G/s

2G/s

3G/s 4G/s 6G/s

Prevents a single traffic class of hogging all the bandwidth and

starving other classes

When a given load doesnt fully utilize its allocated bandwidth, it isavailable to other classes

Helps accommodate for classes of a bursty nature

Enhanced Transmission SelectionBandwidth Management


25/87


Enhanced Transmission SelectionBandwidth Management

Once feature fcoe is configured, 2classes are made by default

By default, each class is given 50%of the available bandwidth

1Gig FC HBAs

1Gig Ethernet NICs

Traditional Server

A typical server hasequal BW per traffictype

Best Practice : FCoE and Ethernet each receive 50%

Can be changed through QoS settings when higher demands forcertain traffic exist (i.e. HPC traffic, more Ethernet NICs)

N5K1# show queuing interface ethernet 1/13

Ethernet1/13 queuing information:TX Queuing

qos-group sched-type oper-bandwidth0 WRR 501 WRR 50


26/87


Create classification rules first by defining and applyingpolicy-map type qos

Define and apply policy-map type queuing to configurestrict priority and bandwidth sharing

N5010-2(config)# class-map type queuing class-voice

N5010-2(config-cmap-que)# match qos-group 2N5010-2(config-cmap-que)# class-map type queuing class-highN5010-2(config-cmap-que)# match qos-group 3N5010-2(config-cmap-que)# class-map type queuing class-lowN5010-2(config-cmap-que)# match qos-group 4N5010-2(config-cmap-que)# exitN5010-2(config)# policy-map type queuing policy-BWN5010-2(config-pmap-que)# class type queuing class-voiceN5010-2(config-pmap-c-que)# priority

N5010-2(config-pmap-c-que)# class type queuing class-highN5010-2(config-pmap-c-que)#bandwidth percent 50N5010-2(config-pmap-c-que)# class type queuing class-lowN5010-2(config-pmap-c-que)#bandwidth percent 20N5010-2(config-pmap-c-que)# class type queuing class-fcoeN5010-2(config-pmap-c-que)#bandwidth percent 30N5010-2(config-pmap-c-que)# class type queuing class-defaultN5010-2(config-pmap-c-que)#bandwidth percent 0N5010-2(config-pmap-c-que)# system qosN5010-2(config-sys-qos)# service-policy type queuing output policy-BW

N5010-2(config-sys-qos)#

Enhanced Transmission SelectionChanging ETS Bandwidth Configurations

FCoE Traffic given30% of the 10GE link


27/87


Negotiates Ethernet capabilitys PFC, ETS, CoS

values between peer devices Simplifies management of DCB nodes

Allows for configuration and distribution ofparameters from one node to another

Responsible for Logical Link Up/Down signaling ofEthernet and Fibre Channel

Uses Link Layer Discovery Protocol (LLDP) defined by802.1AB to exchange and discover DCB capabilities

DCBX negotiation failures result in:

per-priority-pause not enabled on CoS values

vfc not coming up when DCBX is being used inFCoE environment

Data Center Bridging eXchangeControl Protocol the handshake


28/87


Agenda

Why are we here?



DCB Standard


Design Requirements


Single Hop Designs

Multi-Hop Designs


Questions


29/87


Understanding FCoE

Fibre Channel is to FCoE

as is to


30/87


Mapping of FC Frames over

Ethernet

FCoE

FibreChannel

Traffic

Ethernet

Ethernet

Header

FCoE

Header

FC

Header

FC Payload CRC

EOF

FCS

Byte 0 Byte 2229

Completely based on theFC model

Same host-to-switch andswitch-to-switch behavioras FC

WWNs, FC-IDs, hard/softzoning, DNS, RSCN

FCoE is Fibre Channel

Cisco Dell EMC2 EMULEX

Microsoft

Intel QLOGIC Redhat VMWARE

NetAppHP IBM

Fiber Channel over EthernetFC-BB-5 Protocol


31/87


Mapping of FC Frames over

Ethernet

FCoE

FibreChannel

Traffic

Ethernet

Ethernet

Header

FCoE

Header

FC

Header

FC Payload CRC

EOF

FCS

Byte 0 Byte 2229

Roadmap of Ethernet

Economy of Scale

Massive industry investment

FCoE is Ethernet

Fiber Channel over EthernetFC-BB-5 Protocol


32/87


From a Fibre Channel standpoint itsFC connectivity over a new type of cable called Ethernet

From an Ethernet standpoints its

Yet another ULP (Upper Layer Protocol) to be transported

FC-0 Physical Interface

FC-1 Encoding

FC-2 Framing & Flow Control

FC-3 Generic Services

FC-4 ULP Mapping

Ethernet Media Access Control

Ethernet Physical Layer

FC-2 Framing & Flow Control

FC-3 Generic Services

FC-4 ULP Mapping

FCoE Logical End Point

Fiber Channel over EthernetProtocol Mapping


33/87


Both Protocols Have

Two different Ethertypes FIP 0x8914 , FCOE 0x8906

Two different frame formats

Both are defined in FC-BB-5

FCoE itself

Is the data plane protocol

It is used to carry most of the

FC frames and all theSCSI traffic

Uses Fabric Assigned MACaddress (dynamic)

FIP (FCoE InitializationProtocol)

It is the control plane protocol

It is used to discover the FCentities connectedto an Ethernet cloud

It is also used to login to andlogout from the FC fabric

http://www.cisco.biz/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html

Fiber Channel over EthernetData and Control plane


34/87


FIP discovers other FCoE capable devices within the

Ethernet CloudEnables FCoE adapters (CNAs) to discover FCoE switches(FCFs) on the FCoE VLAN

Establishes a virtual link with between the adapter and FCF orbetween two FCFs

FIP frames use a different Ethertype from FCoE framesmaking FIP-Snooping by DCB capable Ethernet bridges

Building foundation for future multi-hop FCoE topologiesMulti-hop refers to FCoE extending beyond a single hop oraccess switch

Today, multi-hop is achievable with a Nexus 4000 (FIP SnoopingBridge) connected to Nexus 5000 (FCF)

Fibre Channel over Ethernet ProtocolFIP: FCoE Initialization Protocol


35/87


Step 1: FCoE VLAN Discovery

FIP sends out a multicast toALL_FCF_MAC address lookingfor the FCoE VLAN

FIP frames use the native VLAN

Step 2: FCF Discovery

FIP sends out a multicast to theALL_FCF_MAC address on theFCoE VLAN to find the FCFsanswering for that FCoE VLAN

FCFs responds back with their MACaddress

Step 3: Fabric Login

FIP sends a FLOGI request to theFCF_MAC found in Step 2

Establishes a virtual link betweenhost and FCF

EnodeInitiator

FCoE Switch

FCF

VLANDiscover

y

FLOGI/FDISC

FLOGI/FDISC Accept

FCCommand

FCCommandResponses

FCoE

InitializationProtocol

(FIP)

FCoEProtocol

VLANDiscovery

FCFDiscovery

FCFDiscovery

Fiber Channel over Ethernet ProtocolFCoE Initialization Protocol (FIP)

** FIP does not carry any Fibre Channel frames


36/87


The FCoE VLAN is manually configured on the Nexus 5000

The FCF-MAC address is configured on the Nexus 5000 by default oncefeature fcoe has been configured

This is the MAC address returned in step 2 of the FIP exchange

This MAC is used by the host to login to the FCoE fabric


** FIP does not carry any Fibre Channel frames

N5K(config)# feature fcoe

N5K(config)# vlan 2

N5K(config-vlan)# fcoe vsan 2

N5K(config-vlan)# show vlan fcoe

Original VLAN ID Translated VSAN ID Association State

---------------- ------------------ -----------------

2 2 Operational

N5K# show fcoeGlobal FCF details

FCF-MAC is 00:0d:ec:d5:fe:00FC-MAP is 0e:fc:00FCF Priority is 128FKA Advertisement period for FCF is 8 seconds


37/87



Step 3 - login process: show flogi database and show fcoe database

show the logins and associated FCiDs, xWWNs and FCoE MAC addresses


38/87


Fiber Channel over Ethernet ProtocolConfiguration using Device Manager


39/87


Fiber Channel over Ethernet ProtocolHost Side FIP and DCBX Configuration

1st portion of theMAC is the FC-MAP of theNexus 5000

FC-MAP

(0E-FC-00)

FC-ID

B1.00.01

FC-MAC

Address

FC-MAP(0E-FC-xx)

FC-IDB1.00.01

2nd portion of the MACis the FC-ID


40/87


FCoE ForwardingFCF

Intermediate switches in the Ethernet cloudAll are Fibre Channel Aware

EthernetFabric

FC Fabric

FC Domain 7 FC Domain 3MAC A

FCID 7.1.1

FCID 1.1.1MAC C

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

FC Frame

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

FC Frame

EthernetFabric

FC Domain 1MAC B

FC Storage

FCoEFrame

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

Dest. = MAC BSrce. = MAC A

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

Dest. = MAC CSrce. = MAC B

FC link

VE_port VE_port VF_port VN_port


41/87


Agenda

Why are we here?



DCB Standard


Design Requirements


Single Hop Designs

Multi-Hop Designs


Questions


42/87


Ethernet is non-deterministic.Flow control is destination-based

Relies on TCP drop-retransmission / sliding window

Fibre-Channel is deterministic.

Flow control is source-based (B2B credits)

Services are fabric integrated (no loop concept)

The Design RequirementsEthernet vs Fibre Channel


43/87


Ethernet/IP

Goal is to provide any-to-anyconnectivity

Unaware of packet loss relies on ULPs forretransmission and windowing

Provides the transport without worryingabout the services -services provided byupper layers

East-west vs north-south traffic ratios areundefined

Network design has been optimized for:

High Availability from a transport perspective

by connecting nodes in mesh architectures

Service HA is implemented separately

Takes in to account control protocolinteraction (STP, OSPF, EIGRP, L2/L3boundary, etc)

?

?

?

?

???

??

?

??

Switch Switch

Switch

?

Client/ServerRelationshipsare not pre-

defined

? ?

?

Fabric topology andtraffic flows are highly

flexible

The Design RequirementsClassical Ethernet


44/87


Servers typically dual homed to twoor more access switches

LAN switches have redundantconnections to the next layer

Distribution and Core can be

collapsed into a single box

L2/L3 boundary typically deployed inthe aggregation layer

Spanning tree or advanced L2technologies (vPC) used to prevent

loops within the L2 boundaryL3 routes are summarized to the core

Services deployed in the L2/L3boundary of the network (load-balancing, firewall, NAM, etc)

L2

L3

Core

Aggregation

Access



Outside DataCentercloud

STP

STP

The Design RequirementsLAN Design Access/Aggregation/Core


45/87


Fibre Channel SAN

Transport and Services are on the samelayer in the same devices

Well defined end device relationships(initiators and targets)

Does not tolerate packet drop requires

lossless transport

Only north-south traffic, east-west trafficmostly irrelevant

Network designs optimized for Scale andAvailability

High availability of network servicesprovided through dual fabric architecture

SAN A and SAN B : physicallyseparate and redundant fabrics

Strict change isolation - end to end drivercertification

Client/Server

Relationships are

pre-defined

I(c)

I(c)

T(s)

T2

I5

I4I3I2

I1

I0

T1T0

Switch Switch

Switch

DNS FSPF

ZoneRSCN DNS

FSPF Zone

RSCN

DNS

Zone

FSPF

RSCN

Fabric topology, servicesand traffic flows are

structured

The Design RequirementsClassical Fibre Channel


46/87


Edge-Core Topology

Servers connect to the edge switches

Storage devices connect to one ormore core switches

Core switches provide storageservices to one or more edgeswitches, thus servicing more serversin the fabric

ISLs have to be designed so thatoverall fan-in ratio of servers to

storage and overall end-to-endoversubscription are maintained

HA achieved in two physicallyseparate, but identical, redundantSAN fabrics

FC

Core Core

The Design RequirementsSAN Design Two Tier Topology


47/87


Edge-Core-Edge Topology

For environments where future growthof the network has the number ofstorage devices exceeding the numberof ports available at the core switch

A set of edge switches dedicated toserver connectivity and another set ofdedicated for storage devices

Extra edge can also be services edgefor advanced network services

Core is for transport only, rarelyaccommodates end nodes

HA achieved with dual fabrics

FC

Core Core

The Design RequirementsSAN Design Three Tier Topology


48/87


T2

I5

I4I3I2

I1

I0

T1T0

Switch Switch

Switch

DNS FSPF

ZoneRSCN DNS

FSPFZone

RSCN

DNS

Zone

FSPF

RSCN

Question Do we build a FC network on topof an Ethernet Cloud? Or and EthernetNetwork on top of a Fibre Channel Fabric?

Unified Fabric design has to incorporate thesuper-set of requirements

Network -- Lossless and Loss full Topologies

Transport undefined (any-to-any) anddefined (one-to-one)

High Availability redundant network topology(mesh/full mesh) and physically separateredundant fabrics

Bandwidth FC fan-in and oversubscriptionratios and Ethernet oversubscription

Security FC controls (zoning, port security,) and IP controls (CISF, ACL, )

Manageability and Visibility Hop by hopvisibility for FC and the cloud for Ethernet

The Design RequirementsClassical Ethernet + Classical Fibre Channel == ??

? ?

?

?

????

?

?

??

Switch Switch

Switch

?


49/87


Cant we just fold down the dotted line??

FC

Core CoreL2

L3

Core

Aggregation

Access



OutsideData Center

cloud

STP

STP

The Design RequirementsClassical Ethernet + Classical Fibre Channel = ??

FoldHere

FoldHere


50/87


Agenda

Why are we here?



DCB Standard


Design Requirements


Single Hop Designs

Multi-Hop Designs


Questions


51/87


Host connected over unified wire

to first hop access switch Access switch (Nexus 5000) is the

FCF

Fibre Channel ports on the accessswitch can be in NPV or Switchmode for native FC traffic

DCBX is used to negotiate theenhanced Ethernet capabilities

FIP is use to negotiate the FCoEcapabilities as well as the hostlogin process

FCoE runs from host to accessswitch FCF native Ethernet andnative FC break off at the accesslayer

FC

CNA

FC Fabric

ENode

Target

Ethernet Fabric

DCB capable Switchacting as an FCF

Unified Wire

Single Hop DesignTodays Solution


52/87


The first phase of the Unified Fabricevolution design focused on the fabricedge

Unified the LAN Access and the SANEdge by using FCoE

Consolidated Adapters, Cabling and

Switching at the first hop in the fabrics

The Unified Edge supports multipleLAN and SAN topology options

Virtualized Data Center LAN designs

Fibre Channel edge with direct

attached initiators and targets

Fibre Channel edge-core and edge-core-edge designs

Fibre Channel NPV edge designsThe Unified Edge

Fabric A Fabric BLAN Fabric

FC

FCoEFC

LANAccess/SAN

Edge

Single Hop DesignUnified Wire at the Access

Nexus 5000FCF-A

Nexus 5000FCF-B


53/87


Fibre ChannelDrivers

EthernetDrivers

Operating System

PCIe

Ethe

rnet

FibreC

hannel

10GbE

10GbE

Link

Converged Network Adapter (CNA)presents two PCI address to theOperating System (OS)

OS loads two unique sets of driversand manages two uniqueapplication topologies

Server participates in bothtopologies separately

Two stacks and thus two views ofthe same unified wire

SAN Multi-Pathing providesfailover between two fabrics(SAN A and SAN B)

NIC Teaming provides failoverwithin the same fabric (VLAN)

Ethernet Driver

bound toEthernet NIC PCI

address

FC Driverbound to FC

HBA PCIaddress

Unified Wireshared by both

FC and IPtopologies

Nexus UnifiedEdge supportsboth FC and IP

topologies

Nexus Edge participates inboth distinct FC and IP Core

topologies

Nexus 5000FCF-A

Nexus 5000FCF-B

Single Hop DesignThe CNA Point of View


54/87


Fabric A

CEE-DCBX

Generation 1 CNA

CIN-DCBX

Generation 2 CNA

Fabric BLAN Fabric

VN

VFDirect attach

VN_Port to

VF_Port

In this first phase we were limited to direct

attached CNAs at the access Generation 1 CNA

Utilized Cisco, Intel, Nuova DataCenter Bridging Exchange protocol(CIN-DCBX)

Only supports direct attachment of anVN_Port to an VF_Port over theunified wire

Generation 2 CNA

Utilizes Converged Enhanced EthernetData Center Bridging Exchange

protocol (CEE-DCBX) Utilizes FCoE Initialization Protocol

(FIP) as defined by the T.11 FC-BB-5specification

Supports both direct and multi-hopattachment (through a Nexus 4000 FIP

Snooping Bridge)

Single Hop DesignThe CNA Point of View

Nexus 5000FCF-A

Nexus 5000FCF-A


55/87


VLAN 10,30

VLAN 10,20

A VLAN is dedicated for every

VSAN in the fabric

FIP discovers the FCoE VLAN andsignals it to the hosts

Trunking is not required on the

host driver all FCoE frames aretagged by the CNA

FCoE VLANs must not beconfigured on Ethernet links thatare not designate for FCoE

Maintains isolated edge switchesfor SAN A and B and separateLAN switches for NIC 1 and NIC 2(standard NIC teaming)

! VLAN 20 is dedicated for VSAN 2 FCoE traffic(config)# vlan 20(config-vlan)# fcoe vsan 2

VSAN 2

STP Edge Trunk


Nexus 5000FCF-A

Nexus 5000FCF-B

VSAN3

Single Hop DesignThe FCoE VLAN


56/87


VLAN 10,30

VLAN 10,20

In order to maintain the integrity of FC

forwarding over FCoE, FCoE VLANsare treated differently than LANVLANs

No flooding, MAC learning,broadcasts, etc.

The FCoE VLAN must not beconfigured as a native VLAN

FIP uses native VLAN

Separate FCoE VLANs must be usedfor FCoE in SAN-A and SAN-B

Unified Wires must be configured astrunk ports and STP edge ports

! VLAN 20 is dedicated for VSAN 2 FCoE traffic(config)# vlan 20(config-vlan)# fcoe vsan 2

VSAN 2

STP Edge Trunk


Nexus 5000FCF

Nexus 5000FCF

VSAN 3

Single Hop DesignThe FCoE VLAN


57/87


VLAN 10,30

VLAN 10,20

FCoE Fabric A will have a

different VLAN topology thanFCoE Fabric B which are differentfrom the LAN Fabric

PVST+ allows unique topology perVLAN

MST requires that all switches in thesame Region have the samemapping of VLANs to instances

MST does not require that all VLANsbe defined in all switches

A separate instance must beused for FCoE VLANs

Recommended: three separateinstances native EthernetVLANs, SAN A VLANs andSAN B VLANs

spanning-tree mst configurationname FCoE-Fabricrevision 5instance 5 vlan 1-19,40-3967,4048-4093instance 10 vlan 20-29instance 15 vlan 30-39


VSAN 3VSAN 2

VLAN 10

Nexus 5000FCF-A

Nexus 5000FCF-B

Single Hop DesignThe FCoE VLAN and STP


58/87


Optimal layer 2 LAN design often

leverages Multi-Chassis Etherchannel(MCEC)

Nexus utilizes Virtual Port Channel(vPC) to enable MCEC either betweenswitches or to 802.3ad attached

servers MCEC provides network based load

sharing and redundancy withoutintroducing layer 2 loops in thetopology

MCEC results in diverging LAN andSAN high availability topologies

FC maintains separate SAN A and SANB topologies

LAN utilizes a single logical topology

Direct Attach vPC Topology

MCEC

vPC Peers

vPC Peer Link


Nexus 5000FCF-A

Nexus 5000FCF-B

Single Hop DesignUnified Wires and MCEC


59/87


vPC enabled topologies with FCoE

must follow specific design andforwarding rules

With the NX-OS 4.1(3) releases avfc interface can only be associatedwith a vPC which has a single [one

(1)] CNA port attached to each edgeswitch

While the port-channel is the sameon N5K-1 and N5K-2, the FCoEVLANs are different

vPC configuration works with Gen-2

FIP enabled CNAs ONLY FCoE VLANs are notcarried on the

vPC peer-link

FCoE and FIP ethertypes are notforwarded over the vPC peer link

Direct Attach vPC Topology

VLAN 10,30

VLAN 10,20STP Edge Trunk

VLAN 10 ONLY HERE!


Nexus 5000FCF-A

Nexus 5000FCF-B

Single Hop DesignUnified Wires and MCEC

vPC contains only 2 X10GE links one to eachNexus 5000


60/87


Nexus 5000FCF-B

Nexus 5000FCF-A

VLAN 10,30VLAN 10,20

VLAN 10,20,30

Dual CNA (FC initiator) connectedvia an Etherchannel to a single edgeswitch is unsupported

A vfc interface can only bebound to a port channel withone local interface

Not consistent with FibreChannel High Availability designrequirements (No isolation ofSAN A and SAN B)

If SAN design evolves to a shared

physical with only VSAN isolation forSAN A and B this could change(currently this appears to be a big if)

ISLs between the Nexus 5000access switches breaks SAN HArequirements

Single homed dual CNADirect Attach

Topology


Single Hop DesignUnsupported Topologies


61/87


32 server facing 10Gig/FCoEports

T11 standard based FIP/FCoEsupport on all ports

8 10Gig/FCoE uplink ports forconnections to the Nexus 5000

Management and configurationhandled by the Nexus 5000

Support for Converged

Enhanced Ethernet includingPFC

Part of the Cisco Nexus 2000Fabric Extender family

FEX-2232Remote Line Card of

the Nexus 5000

Single Hop DesignIntroduction of 10Gig/FCoE Fabric Extender

Nexus 2232


62/87


Nexus 223210GE FEX

SAN BSAN A

Fabric LinksOption 1: Single

Homed PortChannel

Fabric LinksOption 2: Static

Pinned

Server Option 1:FCoE on

individual links.Ethernet traffic isActive/Standby

Server Ethernet driver connectedto the FEX in NIC Teaming (AFT,TLB) or with vPC (802.3ad)

FCoE runs over vPC memberport with a single link from serverto FEX

FEX singlehomed to upstreamNexus 5000

FEX fabric links can beconnected to Nexus 5000with individual links (static

pinning) or a port channel oversubscribed 4:1

Consistent with separate LANAccess and SAN EdgeTopologies

Server Option 2:FCoE on a vPC

member PC witha single link

Nexus 223210GE FEX

Requires FIP enabled CNAs

Single Hop DesignExtending the FCoE Edge Nexus 2232

Nexus 5000FCF-A Nexus 5000

FCF-B


63/87


Nexus 223210GE FEX

SAN BSAN A

Fabric Links:vPC PortChannel

Nexus 2232 can not be configured

in a dual homed configuration(vPC between two N5K) whenconfigured to support FCoEattached servers

MCEC Port Channel will not keepSAN A and San B traffic isolated

Nexus 2000 not supported withdedicated FCoE and dedicated IPupstream fabric links

Nexus 2232 can only currently beconnected to the Nexus 5000 when

configured to support FCoEattached servers

Nexus 7000 will support Nexus2000 in Ethernet only mode inCY2010 (support for FCoE onFEX targeted for CY2011 on next

generation N7K line cards)

Nexus 223210GE FEX

Nexus 5000 Nexus 7000

Single Hop DesignExtending the FCoE Edge Nexus 2232


64/87


Agenda

Why are we here?



DCB Standard


Design Requirements


Single Hop Designs

Multi-Hop Designs


Questions

What is NPIV? And Why?


65/87


What is NPIV? And Why?

N-Port ID Virtualization (NPIV) provides a means to assign multipleFCIDs to a single N_Port

Limitation exists in FC where only a single FCID can be handed out per F-port.Therefore and F-Port can only accept a single FLOGI

allows multiple applications to share the same Fiber Channeladapter port

usage applies to applications such as VMWare, MS Virtual Serverand Citrix

Application Server FC NPIV Core Switch

Email

Web

File Services

Email I/O

N_Port_ID 1

Web I/O

N_Port_ID 2

File Services I/O

N_Port_ID 3

F_Port

F_Port

N_Port

What is NPV? And Why?


66/87


What is NPV? And Why? N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a switch to

act like a server performing multiple logins through a single physical link

Physical servers connected to the NPV switch login to the upstream NPIVcore switch

No local switching is done on an FC switch in NPV mode

FC edge switch in NPV mode does not take up a domain ID

Helps to alleviate domain ID exhaustion in large fabrics

Nexus 5000, MDS 91xx, MDS blade switches,

UCS Fabric Interconnect FC NPIV Core Switch

Eth1/1

Eth1/2

Eth1/3

Server1

N_Port_ID 1

Server2

N_Port_ID 2

Server3

N_Port_ID 3

F_Port

N-Port

F-Port

F-PortNP-Port

M lti H D i


67/87


FCF

DCB + FIP

SnoopingBridge

What design considerations do we

have when extending FCoE beyondthe Unified Edge?

High Availability for both LAN andSAN

Oversubscription for SAN and LAN

Ethernet Layer 2 and STP design

Where does Unified Wire makesense over Unified Dedicated Wire?

Unified Wire provides for sharing of asingle link for both FC and Ethernet

traffic


Multi - Hop DesignConsiderations for FCoE Multi-hop

M lti H D i


68/87


Multi-hop FCoE networks allow for

FCoE traffic to extend past theaccess layer (first hop)

In Multi-hop FCoE the role of atransit Ethernet bridge needs to beevaluated

Avoid Domain ID exhaustion

Ease management

FIP Snooping is a minimumrequirement suggestedin FC-BB-5

Fibre Channel over Ethernet NPV(FCoE-NPV) is a new capabilityintended to solve a number ofdesign and managementchallenges

FCF

FCF

SAN BSAN A

DCB CapableEthernet

Switch

DCB CapableEthernetSwitch

VN

VN

VF

VF

Multi - Hop DesignFCoE Pass-through options

M lti H D i


69/87


What is FIP-Snooping? Efficient, automatic configuration of ACLs

locks down the forwarding path goingfrom CNA to FCF

accomplished by snooping FIP packets

Why FIP-Snooping? Security - Protection from MAC Address

spoofing of FCoE end devices (ENode)

Fibre Channel links are Point-to-Point

Ethernet bridges can utilize ACLs to provide theequivalent path control (equivalent of point-to-point)

Support for FIP-Snooping?Nexus 4000 (Blade switch for IBM BC H)

Multi - Hop DesignFIP-Snooping

FCF

FCF MAC

0E.FC.00.DD.EE.FF

FIP Capable Multi-Hop

Topology

FIPSnooping

ENode

Spoofed MAC

0E.FC.00.DD.EE.FF

ENode MAC

0E.FC.00.07.08.09

SAN

M lti H D i


70/87


On the control plane (FIP ethertype), an Fibre Channelover Ethernet NPV bridge improves over a "FIPsnooping bridge" by intelligently proxying FIP functionsbetween a CNA and an FCF

- takes control of how a live network will build FCoEconnectivity

- makes the connectivity very predictable, without the need foran FCF at the next hop from the CNA

On the data plane (FCoE ethertype), an FCoE NPVbridge offers more ways to engineer traffic betweenCNA-facing ports and FCF-facing ports

An FCoE-NPV bridge knows nothing about FibreChannel, and cant parse packets with FCoE ethertype

Multi - Hop DesignFibre Channel over Ethernet NPV Bridge

M lti H D i


71/87


Proxys FIP functions between a CNA and an FCFFCoE VLAN configuration and assignment

FCF Assignment

FCoE-NPV load balance logins from the CNAs evenly

across the available FCF uplink portsFCoE-NPV will take VSAN into account when mapping or

pinning logins from a CNA to an FCF uplink

Operations and management process are in line withtodays SAN-Admin practices

Similar to NPV in a native Fibre Channel network

Multi - Hop DesignFibre Channel over Ethernet NPV Bridge

M lti Hop Design


72/87


FC

Target

FABRIC A

FCoE_NPV bridge

FCFDomain ID andFC-MAP comefrom the FCF

FCoE Pass through deviceAll FCoE Switching is performed atthe upstream FCF

Addressing is pass out by theupstream FCF

more FCoE connectivity to hostswithout

Running into the domain ID issue

Less-expensive

Consistent management

FCoE-NPV is the FIP-SnoopingPlus

FLOGI

VN

FCoE_NPVdoes notconsume adomain ID

E_NodeMAC Address

VF

VF

VNP

Multi - Hop DesignFCoE-NPV - Enode Login Process

Multi Hop Design


73/87


SAN BSAN A

Nexus 4000 is a Unified Fabric

capable Blade Switch

DCB enabled

FIP Snooping Bridge

Dual Topology requirements forFCoE multi-hop

Servers IP connection to the Nexus4000 is Active/Standby

MCEC is not currently supportedfrom blade server to Nexus 4000

Options 1: Unified Dedicated Wiresfrom Nexus 4000 to Nexus 5000

Options 2: Single Unified Wire PortChannel from Nexus 4000 to Nexus5000

Option 2: SingleHomed Unified

Wire

MezzanineConverged Network

Adapter

Option 1:Unified

Dedicated Wire

PCIe

Ethernet

FibreChannel

10GbE

10GbE

Link

Multi - Hop DesignExtending FCoE with FIP Snooping

Nexus 5000FCF-A

Nexus 5000FCF-B

Nexus 4000FIP Snooping

Bridge-B

Nexus 4000

FIP SnoopingBridge-A

Multi Hop Design


74/87


DCB + FIPSnooping

Bridge

Extending FCoE Fibre Channel

fabrics beyond direct attach initiatorscan be achieved in two basic ways

Extend the Unified Edge (Stage 1)

Add DCB enabled Ethernetswitches between the VN and

VF ports (stretch the linkbetween the VN_Port and theVF_Port)

Extend Unified Fabric capabilitiesinto the SAN Core

Leverage FCoE wires betweenFibre Channel of Ethernetswitches (VE_Ports)

Fabric ALAN Fabric

Using FCoE

for ISL

between FC

Switches

Extending

FCoE into a

multi-hop

Ethernet

Access Fabric

VN

VF

VE

VE

Fabric B

VE

VE

Multi - Hop DesignExtending FCoE with VE_Ports

Nexus 5000FCF-A

Nexus 5000FCF-A

MDS 9000FCF-A

Multi Hop Design


75/87


FCoE-NPV

SAN BSAN A

FCoE-NPV

Two basic design options are

possible when we deploy any FCoEmulti-hop configuration

Option 1 Unified Dedicated Wire

Allows MCEC for IP/Ethernet

Dedicated FCoE links forStorage

Option 2Unified Wire

Leverage Server side failovermechanisms for both SAN andLAN

Allows for Unified Wire beyondthe Server to first device

Option 2: SingleHomed Unified

Wire

Option 1:Dedicated Linksand Topologies

Multi - Hop DesignExtending FCoE with FCoE-NPV

Nexus 5000FCF-A

Nexus 5000FCF-B

Multi Hop Design


76/87


FCF

FIP and

FcoEframes loadshared over

MCEC on a perflow basis

NO SAN A and

SAN B isolation

SAN BSAN A SAN and LAN high availability design

requirements are not always identical

Optimal layer 2 LAN design may notmeet FC high availability andoperational design requirements

Features such as vPC & MCEC are

not viable and not supported beyondthe direct attached server

Server has two stacks andmanages two topologies

Layer 2 network has a single

topology L2MP and TRILL provide options to

change the design paradigm andcome up with potential solutions

FCoE over L2MP/TRILL is not

currently supported

DCBEnabled

Multi - Hop DesignUnsupported Topologies


77/87


Agenda

Why are we here?



DCB Standard


Design Requirements


Single Hop Designs

Multi-Hop Designs


Questions



78/87


Where is it efficient to leverage unified wire, shared links for both SAN

and LAN traffic?At the edge of the fabric the volume of end nodes allows for a greater degree ofsharing for LAN and SAN

In the core we will not reduce the number of links and will either maintainseparate FC or FCoE links to the SAN core and Ethernet links to the LAN core

LAN and SAN HA models are very different (and not fully compatible) FC and FCoE are prone to HOLB in the network and therefore we are

limited in the physical topologies we can build

e.g. 10 x 10G uplinks to LAN aggregation will require 10 x 10G links to a nexthop SAN core (with targets attached) No savings, actually spending more toachieve this direct uplinks to SAN core

Targets are attached to the SAN core (the LAN aggregation and SAN corehave different topology functions)

Where is it more beneficial to deploy two cores SAN and LAN over aunified core topology

FCoE Deployment ConsiderationsDedicated Aggregation/Core Devices



79/87


Migration to 10G FCoE in place of

4/8G FC links (Ethernet price perbit economics)

Edge switch running as FCF withVE_ports connecting to FCF onCore switch

Must be careful of Domain ID creeping

FSPF forwarding for FCoE trafficis end-to-end

Hosts will log into the FCF which

they are attached to (accessFCF)

Storage devices will log into theFCF at the core/storage edge

Maintains HA requirements from

both LAN and SAN perspective

VE Ports

SAN BSAN A

FCoE Deployment ConsiderationsMigration Strategy for FCoE

Nexus 5000FCF-B

MDS 9000FCF-B



80/87


Migration to 10G FCoE in place of

4/8G FC links (Ethernet price per biteconomics)

Edge switch running either as FCFin NPV mode or in FCoE-NPVmode with FCF migrating to theSAN Core

Fibre Channel over Ethernet NPV(FCoE-NPV) is a new constructintended to solve a number ofsystem management problems

Using FCoE_NPV alleviatesdomain ID issue

HA planning for the SAN siderequired

Does loosing a core switch mean the lossof a whole fabric?

FCF

FCF

FCoE-NPV

SAN BSAN A

FCoE Deployment ConsiderationsMigration Strategy for FCoE



81/87


SAN BSAN A Does passing FCoE traffic through a

larger aggregation point makesense?

Multiple links required to support theHA models

1:1 ratio between access toaggregation and aggregation to SAN

core is required

SAN is more vulnerable to HOLB soneed to plan for appropriate capacityin any core ISL

When is a direct Edge to Core linksfor FCoE are more cost effective

than adding another hop?

Smaller Edge device more likely tobe able to use under-provisioneduplinks

1:1 Ratio of links

required unless

FCoE-NPVFCoE uplink is

over-provisioned

CORE

Congestion on

Agg-Core links

will HOLB all

attached edge

devices

FCoE Deployment ConsiderationsShared Aggregation/Core Devices



82/87


v

Different requirements for LAN

and SAN network designs

Factors that will influence thisuse case

Port density

Operational roles and change

management

Storage device types

Potentially viable for smallerenvironments

Larger environments will need

dedicated FCoE SAN devicesproviding target ports

Use connections to a SAN

Use a storage edge of otherFCoE/DCB capable devices

Direct Attach

FCoE Targets

CORE

Multiple VDCs

FCoE SAN

LAN Agg

LAN Core

FCoE Deployment ConsiderationsShared Aggregation/Core Devices

Nexus 5000

FCF-A

Nexus 5000

FCF-B



83/87


FCFFCF

SAN BSAN A

Topology will vary based on scale

(single vs multiple tiers) Architecture as defined for product

development has a dual core

Question - where is the demarkbetween Unified Wire and UnifiedFabric

As the topology grows less UnifiedWire

In all practical terms the edge isthe unified point for LAN and SAN(not the core/agg)

In smaller topologies where core

and edge merge then everythingcollapses but the essential designelements remain

Dedicated SAN and

LAN Core

VLAN 10,30VLAN 10,20

CORE

FCoE Deployment ConsiderationsDedicated Aggregation/Core Devices



84/87


Servers, FCoEattached Storage

p yLarger Fabric Multi-Hop Topologies

Multi-hop edge/core/edge topology

Core SAN switches supportingFCoE

N7K with DCB/FCoE line cards

MDS with FCoE line cards(Sup2A)

Edge FC switches supporting either N5K - FCoE-NPV with FCoE

uplinks to the FCoE enabledcore (VNP to VF)

N5K or N7K - FC Switch withFCoE ISL uplinks (VE to VE)

Scaling of the fabric (FLOGI, )will most likely drive the selection ofwhich mode to deploy

N7K or MDS FCoE

enabled FabricSwitches

FC AttachedStorage

Servers

VE

Edge FCFSwitchMode

VE

Edge Switch inFCoE-NPV

Mode

VF

VNPVE

VE

So Remember


85/87


So Remember

All Unified options are important and have differentplaces within the Data Center Network

FCoE offers a more flexible and cheaper deploymentoption over Fibre Channel

FCoE IS Fibre Channel

Multi-hop FCoE extends the FCoE fabric beyond theaccess

Cisco offers end-to-end FCoE solution with Nexusplatform


86/87


Question and Answer

Thanks for attending this session.


87/87

5 architecture awareness fcoe design and best practices pdf

Documents