© 2013 IBM Corporation

IBM Easy-ConnectSystem Networking- Transparent Mode- Multi-chassis Mode (PureFlex ONLY)- Customer Examples

Design and configuration by Scott Irwin & Igor Marty

EN/CN4093 and Virtual Fabric Switch Module

© 2013 IBM Corporation

What is Easy-Connect

2

• Clever pass-through = Simple I/O module connecting servers, storage and the Core network

• Aggregation = Server ports aggregated together towards Core Network

• No network integration = Seen as one big pipe with server traffic coming in and out.

• No Spanning-Tree = No loops, no network integration

• Similar to Cisco FEX – with one important exception, server to server traffic stays local to chassis.

• Specific Networking features (Virtual Fabric Mode, VMready etc…) can still be activated on selected ports when/if needed.

© 2013 IBM Corporation

Transparent Mode

© 2013 IBM Corporation

Net

wo

rkin

g In

fras

tru

ctu

reEasy-Connect- Transparent ModeEasy-Connect Transparent Mode

• Looks like a “dumb” device to Nexus 5K/Edge Switch• No Spanning Tree Protocol (STP) – eliminates Network admin loop concerns• Provides traffic consolidation in the chassis to minimize ToR port utilization• Provides intra-chassis switching, even in Transparent Mode

Client Benefits• Allows client to manage networking from the Edge/Nexus management system• Allows for intra-chassis switching for optimal performance (i.e. vMotion)• Allows for pay as you grow expandability in the chassis, both to the network and the compute

nodes.• VLAN’s are transparent when in “Transparent Mode” between the NIC and Edge Switch• Allows for pNIC or Switch Independent vNIC to be utilized

• Note: Power Nodes supported in pNIC mode only• Allows for growth into “Virtual Fabric Mode” using multiple groups and uplinks• Allows for FCoE if ONLY using a single Uplink (FCoE currently not supported over PortChannel)

How to set the I/O Switch into Transparent Mode• Set the uplinks as one logical unit via Static Port-Channel (note: VFSM support LACP when using

vNIC Groups)• Turn on vNIC and configure all internal “ports” and external “port-channel” into vNIC “Group 1”• Turn on Spanning Tree BPDU Guard and Edge on the Edge/Nexus Switch for additional

protectionNote: If connecting to a Nexus 2k both BPDU Guard and Edge are already enabled by default and

cannot be disabled.• Works with PureFlex Chassis EN/CN4093 and Virtual Fabric Switch Modules• If using FCoE over a single link simply enable CEE Globally (cee enable)

© 2013 IBM Corporation

Net

wo

rkin

g In

fras

tru

ctu

reEasy-ConnectConfiguration to convert from Switch Mode to Transparent ModeConfiguration for the IBM VFSM and EN/CN4093•ssh to each VFSM,EN/CN4093•boot the VFSM,EN/CN4093 to use isCLI (Cisco-Like CLI) for a more familiar look

• /boot/mode iscli• /boot/reset• /boot/prompt enable enable if using FSM on PureFlex

•Set the following commands to convert to “Transparent Mode”• ssh to each VFSM• enter into enable mode (e.g. enable)• enter into configuration terminal mode (e.g. config t)• type the following; (start with factory default configuration)

“boot conf factory”===========Begin Script ===========spanning-tree mode disableinterface port ext1-ext10lacp key 1001lacp mode active vnic enable vnic vnicgroup 1 vlan 4091 port int1-int14 adminkey 1001 enable failoverexitwrite memory===========End Script ===========

Chassis

Node

Switch Switch

VFSM 2VFSM 1

vLAGvPC

LACP

© 2013 IBM Corporation

Net

wo

rkin

g In

fras

tru

ctu

reEasy-ConnectConfiguration to convert from Switch Mode to Transparent Mode w/FCoEConfiguration for the IBM VFSM and EN/CN4093•ssh to each VFSM•boot the VFSM to use isCLI (Cisco-Like CLI) for a more familiar look

• /boot/mode iscli• /boot/reset

•Set the following commands to convert the VFSM to “Transparent Mode”• ssh to each VFSM• enter into enable mode (e.g. enable)• enter into configuration terminal mode (e.g. config t)• type the following; (start with factory default configuration)

“boot conf factory”===========Begin Script ===========spanning-tree mode disableinterface port ext1-ext4lacp key 1001lacp mode active vnic enable vnic vnicgroup 1 vlan 4091 port int1-int14 adminkey 1001 enable failoverexitcee enablewrite memory===========End Script ===========

vLAGvPC

LACP

Chassis

Node

Switch Switch

VFSM 2VFSM 1

© 2013 IBM Corporation

Multi-Chassis Mode

© 2013 IBM Corporation

Net

wo

rkin

g In

fras

tru

ctu

reEasy-Connect- Multi-chassis Mode

Easy-Connect Multi-chassis Mode• Allows G8264 and EN/CN4093 look like a “dumb” device to Edge/Nexus Switches• No Spanning Tree Protocol (STP) Required – eliminates Network admin loop concerns• Provides traffic consolidation in the chassis to minimize ToR port utilization• Provides intra chassis switching, even in Easy-Connect Mode• Works with EN/CN4093 ONLY as VFSM does NOT support vLAG

Client Benefits• Allows client to manage networking from the Edge/Nexus management system• Allows for intra/external chassis switching for optimal performance (i.e. vMotion)• Allows for pay as you grow expandability in the chassis, both to the network and the

compute nodes.• VLAN’s are transparent on both G8264 and EN/CN4093 when in “Easy-Connect Mode”

between the NIC and Upstream AGG/Core Switch• Allows for pNIC or Switch Independent vNIC to be utilized• Allows for growth into Virtual Fabric Mode• Note: Power Nodes supported in pNIC mode only

Use cases for Multi-Chassis Mode• Works with environments that require Active/Active NIC Teaming and or Bonding using

802.3ad and or Static EtherChannels on the NODE Ports• Works with both Intel and Power Nodes• ALL local Layer 2 Traffic, even if using different I/O Bays, will remain within the same

enclosure with the use of the ISL Peer-Links between the EN/CN4093’s• Each Chassis appears to upstream Network as a single entity

© 2013 IBM Corporation

Easy-ConnectWhat you end up with in Multi-chassis Mode with vLAG and vPC

How-To: EN/CN4093 Easy-Connect Multi-chassis Mode

Disable Global Spanning Tree

Set all VFSM ports, both EXT Ports facing the Network (Top of Rack) and INT Ports, into tagpvid-ingress using VLAN 4091 as the PVID

Enable TAGGING on the two EXT Ports being used as the vLAG Peer Link and set 4090 as the PVID adding VLAN 4091 as a TAGGED/TRUNKED member (VLAN 4090 is used as the vLAGISL VLAN)

Configure all required LACP PortChannels (Peer Link, EXT, and INT Ports)

Configure a dummy IP Address used by the MGT EXT Port vLAG Health Check (e.g. 1.1.1.1 (SW1) and 1.1.1.2 (SW2))

Configure vLAG ISL, Health Check peer-ip, and all associated vLAG pairs

NOTE: Turning on Spanning Tree BPDU Guard and Edge on the EDGE G8264 / Nexus 5K Switches can provide further protection

Chassis

NodeX or P

Switch Switch

EN/CN4093EN/CN4093

LACP

LACP

vLAGor vPC

vLAG

© 2013 IBM Corporation

Easy-ConnectIf using G8264’s in ToR also in Multi-Chassis Transparent Mode

Chassis

NodeX or P

G8264-1 G8264-2

EN/CN4093EN/CN4093

LACP

AGG/Core AGG/Core

LACP LACP

vLAG

LACP

vPC

vLAG

How-To: G8264 Easy-Connect Multi-chassis Mode

Disable Global Spanning Tree (not required (default is using Rapid PVST+ (i.e. PVRST)))

Set all G8264 ports, both uplink facing network and enclosure facing EN/CN4093’s, into tagpvid-ingress using VLAN 4091 as the PVID

Enable TAGGING on the two ISL Ports being used as the vLAG Peer-Link and set 4090 as the PVID adding VLAN 4091 as a TAGGED/TRUNKED member (VLAN 4090 is used as the vLAGISL VLAN)

Configure all required LACP PortChannels (Peer Link, Uplinks, and EN/CN4093 facing ports)

Configure a dummy IP Address used by the MGT Port vLAG Health Check (e.g. 1.1.1.1 (SW1) and 1.1.1.2 (SW2))NOTE: if using customers MGT Network set IP’s appropriately

Configure vLAG ISL, Health Check peer-ip, and all associated vLAG pairs

NOTE: Turning on Spanning Tree BPDU Guard and Edge on the EDGE G8264 / Nexus 5K Switches can provide further protection

© 2013 IBM Corporation

Customer requirements to implement Easy-Connect

11

1. If using FSM in same chassis – since the FSM is NOT capable of TAGGING (Cisco Trunking) customer MUST enable the Top-of-Rack Port “Native VLAN ID” with the VLAN that the FSM needs to be configured on.

2. ALL other OS’s (i.e. VMware, VIO, and even Bare Metal OS’s) within the PureFlex Chassis are required to TAG/Trunk VLAN’s.

3. If using multiple vNIC Groups for traffic either traffic separation and or if using Virtual Fabric Mode - each vNIC Group requires it’s own uplink/PortChannel or DUM (Dedicated Uplink Mode)

4. If Multi-Tenant security is a concern within the same PureFlex Chassis Easy-Connect may/may not be a good option. Why, because each vNIC Group is a single broadcast domain.

5. If IGMP Multicast Snooping is required Easy-Connect may not be a best option. Since Easy-Connect is VLAN Agnostic IGMP Snooping is not supported in any Q-n-Q Architecture. However, Multicast will still pass through the Easy Connect enabled Switches.

© 2013 IBM Corporation

Easy Connect

Customer Examples

© 2013 IBM Corporation

Easy-Connect EN/CN4093Transparent Mode – Telecom Customer (Very Simple)

Requirements included:•No Spanning Tree and or any other protocols seen by the network (Easy-Connect)•Connection into Nexus 2k (no vPC or vLAG requirement)•EN4093 is to be a total transparent device requiring no management by any group

© 2013 IBM Corporation

Easy-Connect EN/CN4093Transparent Mode – State Government

Requirements included:•Using LoM in Virtual Fabric Mode so bandwidth can be adjusted on the fly for each vNIC as required•Dedicated uplink vPC PortChannel from each EN4093 for each vNIC Group for separation of traffic•EN4093’s in Transparent Mode with minimal hands-on requirement (vNIC BW ONLY)

© 2013 IBM Corporation

Easy-Connect EN/CN4093Storage Mode w/NIC Redundancy – Medical Center

Requirements included:•Dedication and Separation of Fiber Channel and Ethernet from Each NODE and Enclosure•Total Hardware Redundancy including NIC and ASIC on every NODE (CN4054 Mezz)•Transparency on both Ethernet (Easy-Connect) and Fiber Channel (NPV)

© 2013 IBM Corporation

Backup Slides

© 2013 IBM Corporation17

Four Scalable switches enable high speed connectivity – Ethernet (FCoE, iSCSI), Fibre Channel and InfiniBand

17

Four high performance

Scalable Switch Modules

1 23 4

© 2013 IBM Corporation

1 3 2 4

Redundant pairRedundant pairasic level redundancy

Adapter level redundancy

Robust connectivity: Switch, asic and adapter level redundancy

CN4054

CN4054

asic 1

asic 1

asic 2

asic 2E

N40

93 (

bas

e)

EN

4093

(b

ase)

EN

4093

(b

ase)

EN

4093

(b

ase)

EN

4093

(U

pg

rad

e 1

)

EN

4093

(U

pg

rad

e 1

)

EN

4093

(U

pg

rad

e 1

)

EN

4093

(U

pg

rad

e 1

)