cfp380 fcip theory - brocade community...
TRANSCRIPT
FCIP Theory
Revision 0210
CFP380
1
FCIP Theory
Revision 0210
CFP380
2
FCIP Theory
Revision 0210
CFP380
3
Footnote 1: Distances that utilize native FC can span 500km; these solutions incorporate dark fiber, C/DWDM, and form a single fabric.
For additional FCIP details, reference RFC 3821 – Fibre Channel Over TCP/IP (FCIP).
Brocade does not recommend FCIP for use in every distance extension scenario: no technical solution can be all things to all people. FCIP has inherent performance, reliability, data integrity, and manageability limitations when compared to native FC solutions. Delay and packet loss may create bottlenecks in IP networks. FCIP can support very long distances, as long as the carrier network is extremely high performance and reliable. FCIP is typically deployed when long-haul applications are not business critical, and do not need especially high performance. FCIP may not be suitable for tape, since tape usage will often fail if packets are dropped. In addition to its performance limitations, FCIP troubleshooting and performance analysis requires evaluating all aspects of the IP LAN and WAN networks in addition to all FC nodes, switches, and routers, which can make it more complex to manage than other extension options.
FCIP Theory
Revision 0210
CFP380
4
FC traffic over an IP network
• Interconnection of islands of Fibre Channel storage area networks over IP-based networks
• Distance Extension over IP LAN/WAN/MAN
FCIP Theory
Revision 0210
CFP380
5
After FC frames destined for devices at the remote side are encapsulated into TCP packets, a standard IP header is added to each packet. The packet is then sent to the next hop (usually an Ethernet router).
FCIP Theory
Revision 0210
CFP380
6
FCIP Theory
Revision 0210
CFP380
7
FCIP Theory
Revision 0210
CFP380
8
FCIP Theory
Revision 0210
CFP380
9
Footnote 1: An Upgrade License is required to enable all 16 FC ports and 6 GbE ports.
Footnote 2: Two GbE ports can be configured for copper (via built-in RJ45) or optical (via SFP module) cable connectivity. The other four GbE ports offer optical SFP module only. SFP ports will also accept 1 Gbps copper SFPs, Avago part number ABCU-5700RZ-BR1.Footnote 3: Link speeds of 1 Gbps require a 4 Gbps Brocade-branded SFP.
FCIP Theory
Revision 0210
CFP380
10
Footnote 1: The top two GbE ports (GE0 and GE1) are configured for copper via built-in RJ45. In standard configuration, users have the option of using either the top two GbE ports, which are configured for copper SFPs, or the bottom two, left-most, GbE ports (GE0 and GE1) which are configured for optical SFPs. The remaining four GbE ports can use either optical or copper via SFP module up to 1 Gbps. It is possible to configure GE0 as copper and GE1 as optical and vice versa.
FCIP Theory
Revision 0210
CFP380
11
Footnote 1: Fabric OS identifies power supplies left to right as Power Supply #1 and Power Supply #2. The Brocade 7800 requires a minimum of one power supply (connected and powered on) for switch operation. Footnote 2: Fabric OS identifies the fan assemblies left to right as Fan #1 and Fan #2.
Weight: 9.3 kg (20.6 lb)1Dimensions:Width: 42.87 cm (16.88 in)
Height: 4.30 cm (1.70 in) or 1UDepth: 61.0 cm (24.02 in)
Power Consumption: 84 Watts nominal, 91 Watts maximum
Both weight and power consumption numbers assume Brocade 7800 with two power supply/fan FRUs and zero SFPs installed.
FCIP Theory
Revision 0210
CFP380
12
The base unit has 2 GbE ports available for use. They can be the two copper ports, or the two fibre ports. The default is copper.
The upgrade license is named the 7800 Upgrade license.
FCIP Theory
Revision 0210
CFP380
13
Please refer to the Release Notes for the most up-to-date information.
Enabling either of the 2 x10 GbE ports requires a 10 GbE license, which is a slot-based license. Footnote 1: The supported operational modes are:
• 10 x 1 GbE port
• 10 x 1 GbE ports and 1 x 10 GbE port• 2 x 10 GbE ports
FCIP Theory
Revision 0210
CFP380
14
Each Cavium processor has 4G of DDR memory.
FCIP Theory
Revision 0210
CFP380
15
Footnote 1: Typically, port groups for Condor2 ASICs are 8-port trunk groups. There are 40 ports on a Condor2 ASIC. As seen in the notes section of the previous slide, there is a 5-port trunk to each of the four Blaster FPGAs. By definition, a trunk must be created from the same octet. This means that each octet from a Blaster trunk octet has 3 ports remaining in the octet. Thus, Brocade engineers used the available ports to create 2 other 2-port trunks to use some of the remaining ports.
Weight: 3.2 kg (7.0 lb)1Dimensions:
Width: 3.60 cm (1.41 in)Height: 42.06 cm (16.56 in)
Depth: 29.89 cm (11.77 in)Power Consumption: 235 Watts nominal Approximate weight with 0 SFPs installed. The minimum power consumption of the Brocade FX8-24 Extension Blade is 235 watts with 0 optical SFPs installed running at 8 Gbps.
FCIP Theory
Revision 0210
CFP380
16
Footnote 1: Requires the 10Gbps license.
FCIP Theory
Revision 0210
CFP380
17
FCIP Theory
Revision 0210
CFP380
18
Footnote 1:
FCIP Tunnels: •B7800 - up to 8 VE_Ports
•FX8-24 - up to 20 VE_PortsFCIP Trunking:
•B7800 - up to 4 circuits per trunk•FX8-24 - up to 4 circuits per trunk for 1 GbE; up to 10 circuits per trunk for 10 GbE
Footnote 2: 10 GbE, Advanced Extension and FICON Accelerator licenses for FX8-24 supported in DCX and DCX-4S are slot based licenses
Footnote 3: Not supported in initial Fabric OS v6.3.0 release.
FCIP Theory
Revision 0210
CFP380
19
Certified (Support is through the SFP vendor)
1/2/4 Gbps ELWL - 50km
1/2/4 Gbps ELWL - 100km
1/2/4 Gbps CWDM - 1470nm - 80km
1/2/4 Gbps CWDM - 1490nm - 80km
1/2/4 Gbps CWDM - 1510nm - 80km
1/2/4 Gbps CWDM - 1530nm - 80km
1/2/4 Gbps CWDM - 1550nm - 80km
1/2/4 Gbps CWDM - 1570nm - 80km
1/2/4 Gbps CWDM - 1590nm - 80km
1/2/4 Gbps CWDM - 1610nm - 80km
1/2/4 Gbps DWDM 80km (25 wavelengths from 1530nm-1560nm)
2/4/8 Gbps CWDM 40km(8 wavelengths from 1290nm-1430nm)
For the most up-to-date support, refer to the following site:
http://www.brocade.com/products-solutions/products/tranceivers/index.page
FCIP Theory
Revision 0210
CFP380
20
FCIP Theory
Revision 0210
CFP380
21
FCIP Theory
Revision 0210
CFP380
22
C2 and GE2 = Condor2 and GoldenEye2 ASICs
FCIP Theory
Revision 0210
CFP380
23
FCIP Theory
Revision 0210
CFP380
24
Footnote 1: FC flow control mechanisms include R_RDYs and ACKs. FC communications also utilize long distance modes, BB credits, and VC channels.
FCIP Theory
Revision 0210
CFP380
25
Footnote 1: There are 8 VE_Ports for 6 physical ge ports on the 7800 16/6. An FX8-24 blade can support 20 VE_Ports, and therefore 20 FCIP tunnels. Each FCIP tunnel is associated with a specific VE_Port. On FX8-24 blades, and on the 7800 switch, VE_Ports do not have to be associated with a particular GbE port.VE_Ports 12 through 21 may use GbE ports ge0 through ge9, or they may use XGE port 1. VE_Ports 22 through 31 can only be used by XGE port 0. The total bandwidth cannot exceed 20 Gbps.
FCIP Theory
Revision 0210
CFP380
26
Footnote 1: As of Fabric OS v6.3.0, VEX_Ports are not supported on the FX8-24.
FCIP Theory
Revision 0210
CFP380
27
The 7500E and 7500 switches and the FR4-18i blade support only one connection per GbE port, so strictly speaking, the FCIP circuit concept does not apply
FCIP Theory
Revision 0210
CFP380
28
VE ports and GbE ports are no longer 1:1 associated as they were on the 7500 and FR4-18i.
VE and GbE ports can have a 1:1 association, but they are not limited by the design.
Footnote 1: Configuring a tunnel with more than one circuit requires an Advanced Extension license. Without a license present, a second circuit will not be allowed to be configured. The administrator will receive a message stating as such.
FCIP Theory
Revision 0210
CFP380
29
FCIP trunking requires multiple FCIP circuits, therefore FCIP trunking cannot be implemented on the 7500E, 7500 switch, and the FR4-18i blade.
Footnote 1: Configuring a tunnel with more than one circuit requires an Advanced Extension license. Without a license present, a second circuit will not be allowed to be configured. The administrator will receive a message stating as such.
FCIP Theory
Revision 0210
CFP380
30
Time out values:
• Default FICON is 1 sec• Default FC is 4 sec
• Depending on the solution being extended, the time out values may need to be changed from default.
FCIP Theory
Revision 0210
CFP380
31
Example tunnel/circuit creation:
A tunnel using VE_Port 16 is created with an initial circuit 0 with a maximum rate of 1Gbps, metric of 0.
• IPPM_LINK_UP.bandwidth = 1GbpsCircuit 1 is added to Tunnel 16 with a max rate of 500Mbs, metric 0.
• TUNNEL_UPDATE.bandwidth = 1.5 GbpsCircuit 2 is added to Tunnel 16 with a max rate of 1Gps, metric 1.
• Since this is a higher metric, it is considered a standby and no TUNNEL_UPDATE is generated.
Circuit 3 is added to Tunnel 16 with a max rate of 1Gps, metric 1.• Since this is a higher metric, it is considered a standby and no
TUNNEL_UPDATE is generated.
FCIP Theory
Revision 0210
CFP380
32
Example circuit failures/recovery (after initial creation state)
Circuit 1 fails.• Since this was a low metric circuit (active) we will try to re-establish, no
TUNNEL_UPDATE is sent to CP (and no updated will be sent once it re-establishes).
Circuit 0 fails, 1 is still failed.
• Since this is the last of the low metric circuits, the tunnel will failover to the next lowest circuits, which are circuits 2 and 3. Since the combined bandwidth of circuits 2 and 3 is different than circuits 0 and 1, a TUNNEL_UPDATE.bandwidth = 2.0 Gbps is generated to the CP.
Circuit 0 and 1 come back on line.• Since these are again the low metric circuits, we will resume traffic on these
circuits and 2 and 3 become the hot standbys. Since the bandwidth has changed again, a TUNNEL_UPDATE.bandwidth = 1.5 Gbps is generated to the CP. Note that the bandwidth was reduced.
Example circuit deletions (after initial creation state);
Circuit 1 is deleted.• Since this was a low metric circuit, all traffic will be directed to circuit 0. Again,
the bandwidth has changed so the tunnel will generate a TUNNEL_UPDATE.bandwidth = 1.0 Gbps to the CP.
Deleting circuits 2 or 3 at this point would have no affect since they are standby and the bandwidth is not included in the calculations.
Example circuit metric changes (after initial creation state);
Circuit 1’s metric is changed to 1.• Since this was a low metric circuit, all traffic will be directed to circuit 0. The
bandwidth has changed, so the tunnel will generate a TUNNEL_UPDATE.bandwidth = 1.0 Gbps to the CP. This would basically have the same affect on bandwidth as if deleting the circuit instead.
Circuit 3’s is changed to metric 0 (circuit 1 still has a metric of 1)• Since this was a high metric circuit changed to a low. The bandwidth has
changed, so FTNL will generate a TUNNEL_UPDATE.bandwidth 2.0 Gbps to the CP (circuit 0 and circuit 3 are now “active”, circuits 1 and 2 are now in “standby” mode).
FCIP Theory
Revision 0210
CFP380
33
Footnote 1: Each 10 GbE port is assigned to a different FCIP subsystem. Because of this, tunnels cannot be created traversing the two ports. All circuits that reside in a tunnel must belong to the same FCIP subsystem.
FCIP Theory
Revision 0210
CFP380
34
FCIP Theory
Revision 0210
CFP380
35
3 Modes of operation for the Ethernet ports on FX8-24:
1G Only Mode: 10 x 1 Gbps ports are available for use (GE0 through GE9)10G Only: 2 x 10 Gbps ports are available for use (XGE0 and XGE1)1
Dual Mode: 10 x 1 Gbps ports and 1 x 10 Gbps port available for use (GE0 through GE9, XGE0)
FCIP Theory
Revision 0210
CFP380
36
Footnote 1: IP address subnets are used so that the TCP supervisor can route packets to the destination subnets. The FCIP subsystem contains an IP routing table that directs packets with a tunnel destination to the correct circuits and tunnels.Footnote 2: The B7800 contains 6 x 1 Gbps connections between field-programmable gate arrays (FPGAs) and the Ethernet ports.Footnote 3: If multiple circuits are configured on the same physical port, they can contain addresses from the same subnet or different subnets.
Footnote 4: For the FX8-24, each FCIP complex has ten 1 Gbps connections between FPGAs and the Ethernet ports. This is the reason that the XGE ports require 10 circuits to achieve 10 Gbps bandwidth.
FCIP Theory
Revision 0210
CFP380
37
• Packet loss re-transmissions are compounded when errors are bursty. Selective Acknowledgement (SACK) is an extension to a protocol which allows the acknowledge reception of specific packets or messages.The SACK option RFC 2883 [18] allows the receiver to acknowledge multiple lost packets in a single ACK, enabling faster recovery. An FCIP Entity may negotiate use of TCP SACK and use it for faster recovery from lost packets and holes in TCP sequence number space.Footnote 1: SACK improves loss detection, retransmission techniques, and enables faster recovery.
FCIP Theory
Revision 0210
CFP380
38
Hardware compression is performed at FC ingress and de-compression is performed at FC egress.
Compression before encapsulation allows the TCP and FCIP headers to be visible on the network.
FCIP Theory
Revision 0210
CFP380
39
Footnote 1: A value of 1 enables hardware compression. The 7800 switch provides two additional levels of software compression. Settings 2 and 3 provide incrementally higher compression ratios that can be used to improve performance on slower links. A value of 0 disables compression.
FCIP Theory
Revision 0210
CFP380
40
FCIP Theory
Revision 0210
CFP380
41
FCIP Theory
Revision 0210
CFP380
42
FCIP Theory
Revision 0210
CFP380
43
FCIP Theory
Revision 0210
CFP380
44
FCIP Theory
Revision 0210
CFP380
45
Ports ge4 and ge9 are available to be used, but are not in the example above.
FCIP Theory
Revision 0210
CFP380
46
FCIP Theory
Revision 0210
CFP380
47
FCIP Theory
Revision 0210
CFP380
48
Best practice is to use a commit rate that uses 90% of the bandwidth allocated to the FCIP traffic. In this example, both devices are configured for 100% of the total bandwidth of the WAN gateway, which means that the router is oversubscribed 2:1. This configuration can lead to many dropped frames and errors.
FCIP Theory
Revision 0210
CFP380
49
The configuration above is a better solution than the previous slide, but still has shortcomings. While the router is not oversubscribed, the bandwidth available for the WAN is not fully utilized when there is either a failure of one of the devices, or a simple case of a device needing less bandwidth than what is configured. This can leave the link underutilized during certain times.
FCIP Theory
Revision 0210
CFP380
50
Footnote 1: Best practice dictates setting maximum commit rates to 90% of physical connection.
The best solution would to use Adaptive Rate Limiting (ARL) to better utilize the available WAN bandwidth. By configuring each device with a minimum commit rate of 50% of the allocated bandwidth, each device is then guaranteed a minimum amount of bandwidth at all times. If an outage occurs in one of the devices, or a device is not using all of its minimum committed rate bandwidth, the other active device can utilize the extra bandwidth up to its configured maximum commit rate.
FCIP Theory
Revision 0210
CFP380
51
FCIP Theory
Revision 0210
CFP380
52
FCIP Theory
Revision 0210
CFP380
53
FCIP Theory
Revision 0210
CFP380
54
Brocade’s original virtual channel model (pre-Condor2/GoldenEye2) divides an ISL into 8 virtual channels to insure that traffic of multiple priorities can travel across the link at the same time, without being disrupted, or disrupting other traffic.
���������������� �����������
VC # Assigned to
0 Class F
1 Class 2 Ack/Link Control
2 Data
3 Data
4 Data
5 Data
6 Class 3 Multicast
7 Broadcast/Multicast
All Data Traffic
FCIP Theory
Revision 0210
CFP380
55
The QoS feature only comes into play if there is contention on the link. If there is no congestion on the link QoS will not engage.
The order of operations during congestion is as follows and repeats as necessary:1. VC0 then, 2. 6 frames of High priority traffic then,
3. 3 frames of Medium priority traffic then, 4. 1 frame of Low priority traffic
Results of QoS on a congested link:
FCIP Theory
Revision 0210
CFP380
56
FCIP Theory
Revision 0210
CFP380
57
Footnote 1: TCP port used is 3225.
FCIP Theory
Revision 0210
CFP380
58
FCIP Theory
Revision 0210
CFP380
59
FCIP Theory
Revision 0210
CFP380
60
FCIP Theory
Revision 0210
CFP380
61
FCIP Theory
Revision 0210
CFP380
62
FCIP Theory
Revision 0210
CFP380
63
FCIP Theory
Revision 0210
CFP380
64
FCIP Theory
Revision 0210
CFP380
65
FCIP Theory
Revision 0210
CFP380
66
FCIP Theory
Revision 0210
CFP380
67
FCIP Theory
Revision 0210
CFP380
68
FCIP Theory
Revision 0210
CFP380
69
FCIP Theory
Revision 0210
CFP380
70
FCIP Theory
Revision 0210
CFP380
71
FCIP Theory
Revision 0210
CFP380
72
FCIP Theory
Revision 0210
CFP380
73
FCIP Theory
Revision 0210
CFP380
74