de-pn-dg dell emc data center interconnect fabric for os10

Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRailPowered by Dell EMC PowerSwitch and Pluribus Netvisor ONE

Abstract

Design Guide

Networking from Dell Technologieswww.DellTechnologies.com/Networking

This deployment guide describes how to deploy the Pluribus Adaptive Cloud Fabric to interconnect multiple geographically distributed data center sites running Dell EMC SmartFabric OS10 in the leaf/spine switches with an application focus on stretched VxRail Clusters to support workload mobility and active-active resiliency.

April 2021

H18513Dell Technologies Networking Infrastructure Solutions

Table of Contents

Notes, cautions, and warnings

Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 2 1 | Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail

Overview 2

Edge DCI Interconnection for multiple local data center fabrics 3

Pluribus Edge DCI Solution Overview 3

Pluribus Edge DCI Deployment Overview 4

Detailed Underlay (Day-0) configurations steps: 10

Detailed Overlay (Day-1/2) configurations steps: 21

Pluribus UNUM – Unified Automation, Management and Analytics 26

Support and references 31

A NOTE indicates important information that helps you make better use of your product.A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the problem.A WARNING indicates a potential for property damage, personal injury, or death.

Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 2

Overview This guide describes how to configure and implement the Pluribus Edge Data Center Interconnect (DCI) solution to interconnect multiple data centers running Dell EMC OS10 on the leaf/spine switches. The DCI solution uses the Pluribus Adaptive Cloud Fabric (ACF) so�ware to provide an SDN-automated VXLAN over IP DCI fabric between all of the DC sites. The application focus is on stretching layer 2 VLANs to support VxRail vSAN clusters across multiple datacenters to support workload mobility and active-active resiliency in conjunction with a VxRail Witness Host.

The local fabrics in each data center are interconnected across the WAN by means of connecting from the border leaf switch pair at each DC location with standard VLANs to pair of collocated Dell ON switches which are running Pluribus Netvisor ONE operating system and acting as the DCI switches. The Pluribus Adaptive Cloud Fabric so�ware is deployed between all of the DCI switches and is a distributed architecture which delivers a radically simplified approach to building large scale DCI solutions leveraging industry standard Virtual Extensible LAN (VxLAN) encapsulation virtualized overlay network featuring SDN automation, transported over an IP underlay network.

The Pluribus DCI solution enables organizations to simplify their data center network with a distributed fabric bringing unified management across multiple datacenters and unparalleled operational simplification and agility.

Supporting up to 64 geographically distributed data centers, the Pluribus Edge DCI solution allows the extension of Layer 2 & Layer-3 services across multiple locations connected via an existing WAN orcore network.

Pluribus ACF can be managed via CLI, RESTful API or Pluribus UNUM virtual appliance and also supports scripting languages such as Python and automation frameworks such as Ansible. UNUM is a unified fabric management, automation and analytics platform that o�ers a rich graphical user interface and enables administrators to further reduce day 0, 1 and 2 DCI operations.

Figure 1 Pluribus So�ware-Defined DCI Solution

3 | Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail

Dell switches used for DCI must support Pluribus Netvisor ONE R5.2 or later. A list of compatible switches can be found here https://www.pluribusnetworks.com/assets/PN-DS-Pluribus_Netvisor_Open_Networking_Edition.pdf. Pluribus recommends S52xx switches or later be used when possible.

Hardware and So�ware Compatibility

There are several methods to interconnect data center sites with VxRail clusters directly connected to Dell OS10 or SFS fabrics. In this design guide we are assuming that each site VxRail site fabric is based on a leaf-spine fabric topology (based on one of the Dell Technologies network solutions mentioned above). Each site fabric has a designated pair of border leaf switches, which connect via industry standard layer 2 VLANs to a pair of Dell Open Networking switches running Pluribus Netvisor ONE OS with ACF activated. These switch pairs, running Pluribus ACF at each site, then connect to the WAN network and stretch the VxRail networks over the WAN using a VXLAN based overlay.

Edge DCI Interconnection for multiple local data center fabrics

The reference topology used in this design guide is shown in the following figure, which describes the physical interconnections between the border leaf and the Edge DCI switches in each of the two data centers with the VxRail infrastructure, as well as the third site hosting the vSAN witness host, whose function is to protect against split-brain in a vSAN Stretched Clusters.

Pluribus Edge DCI Solution Overview

Pluribus Edge DCI solution provides a highly available, highly scalable distributed fabric to stretch a VxRail vSAN cluster across the WAN interconnecting the datacenters. Based on Pluribus AdaptiveCloud Fabric, the DCI solution allows organizations to simplify the deployment and migration of VMware workloads and vSAN storage across any site to optimize compute resources and enhance business continuity.

The Pluribus Edge DCI solution allows the extension of Layer-2 and Layer-3 network service across the IP WAN to support stretching vSAN services across multiple locations connected via any existing WAN or Core network.

The Pluribus Edge DCI solution interconnection provides the following features: • Unified and centralized management among all the data centers • Simple, cloud-style provisioning and monitoring the overlay network services • Adapts and interoperate with any WAN network • Scales to a large number (64) of sites • Provides site-to-site tra�ic visibility and analytic

The Pluribus Edge DCI with Dell OS10 Hands-on Lab is available on the Demo Center and can help you understand how to configure a data center fabric to achieve so�ware-defined networks. See Demo Center for information about the Hands-on Lab.


Pluribus Edge DCI Deployment Overview Demo Center Hands-on Lab includes full configurations for all the Pluribus Edge DCI nodes. The configuration steps assume Netvisor ONE OS is already loaded on the Dell switches and is shown both for CLI as well as via the UNUM Fabric Manager and cover the following tasks: • Creation of underlay connections for Pluribus Edge DCI fabric • Creation of VTEP (Virtual Tunnel End Point) to support the overlay network required to stretch the VxRail cluster • Creation of stretched VLANs and subnet objects • Creation of a distributed VRF service to provide inter-vlan routing for the VxRail components

The following figure shows configuration details for the pair of Pluribus Edge DCI nodes referred to as Core-1 (for Sites A), Core-3 (for Site-B) and Core-2 (for Site-C). The three sites are unified and managed with a single ACF instance.

Each of the Dell PODs in the primary and secondary datacenters has a pair of designated leaf switches acting as border nodes to hand-o� the vSAN VxRail tra�ic to the Pluribus Edge DCI Fabric as depictedin Figure 1.

The following configuration details refers to the four switches, running Dell OS10, directly connected to the Dell DCI switches running Pluribus Netvisor.

Dell EMC SmartFabric OS10 POD Switch Configuration

Figure 2 Pluribus Interconnection details


4112F-01# show running-configuration

S4112F-02# show running-configuration



! Version 10.5.2.2!ip vrf default!iscsi enableiscsi target port 860iscsi target port 3260hostname S4112F-01interface breakout 1/1/13 map 100g-1xinterface breakout 1/1/14 map 100g-1xinterface breakout 1/1/15 map 100g-1xno multicast snooping flood-restrictsystem-user linuxadmin password ****username admin password **** role sysadmin priv-lvl 15aaa authentication login default localaaa authentication login console local!class-map type application class-iscsi!policy-map type application policy-iscsi!interface vlan1 no shutdown!interface vlan30 description Engineering- VLAN

! Version 10.5.2.2!ip vrf default!iscsi enableiscsi target port 860iscsi target port 3260hostname S4112F-02interface breakout 1/1/13 map 100g-1xinterface breakout 1/1/14 map 100g-1xinterface breakout 1/1/15 map 100g-1xno multicast snooping flood-restrictsystem-user linuxadmin password ****username admin password **** role sysadmin priv-lvl 15aaa authentication login default localaaa authentication login console local!class-map type application class-iscsi!policy-map type application policy-iscsi!interface vlan1 no shutdown!interface vlan30 description Engineering- VLAN

! Version 10.5.2.2!ip vrf default!iscsi enableiscsi target port 860iscsi target port 3260hostname S4128F-01interface breakout 1/1/25 map 100g-1xinterface breakout 1/1/26 map 100g-1x

no multicast snooping flood-restrictsystem-user linuxadmin password ****username admin password **** role sysadmin priv-lvl 15aaa authentication login default localaaa authentication login console local!class-map type application class-iscsi!policy-map type application policy-iscsi!interface vlan1 no shutdown!interface vlan30 description Engineering- VLAN

! Version 10.5.2.2!ip vrf default!iscsi enableiscsi target port 860iscsi target port 3260hostname S4128F-02interface breakout 1/1/25 map 100g-1xinterface breakout 1/1/26 map 100g-1x

no multicast snooping flood-restrictsystem-user linuxadmin password ****username admin password **** role sysadmin priv-lvl 15aaa authentication login default localaaa authentication login console local!class-map type application class-iscsi!policy-map type application policy-iscsi!interface vlan1 no shutdown!interface vlan30 description Engineering- VLAN





no shutdown mtu 9216!interface vlan40 description Marketing-VLAN no shutdown mtu 9216!interface vlan1111 description Mgmt-VLAN no shutdown mtu 9216!interface vlan1312 description vSAN-VLAN no shutdown mtu 9216!interface vlan1412 description vMotion-VLAN no shutdown mtu 9216!interface vlan3939 description VxRail-Dis-VLAN no shutdown mtu 9216!interface port-channel101 description Channel-2- Pluri-Core-1 no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 vlt-port-channel 101!interface mgmt1/1/1 no shutdown no ip address dhcp ip address 172.17.180.31/24 ipv6 address autoconfig!interface ethernet1/1/1 no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 flowcontrol receive on!

interface ethernet1/1/2 no shutdown

no shutdown mtu 9216!interface vlan40 description Marketing-VLAN no shutdown mtu 9216!interface vlan1111 description Mgmt-VLAN no shutdown mtu 9216!interface vlan1312 description vSAN-VLAN no shutdown mtu 9216!interface vlan1412 description vMotion-VLAN no shutdown mtu 9216!interface vlan3939 description VxRail-Dis-VLAN no shutdown mtu 9216!interface port-channel101 description Channel-2- Pluri-Core-1 no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 vlt-port-channel 101!interface mgmt1/1/1 no shutdown no ip address dhcp ip address 172.17.180.32/24 ipv6 address autoconfig!interface ethernet1/1/1 no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 flowcontrol receive on!

interface ethernet1/1/2 no shutdown

no shutdown mtu 9216!interface vlan40 description Marketing-VLAN no shutdown mtu 9216!interface vlan1111 description Mgmt-VLAN no shutdown mtu 9216!interface vlan1312 description vSAN-VLAN no shutdown mtu 9216!interface vlan1412 description vMotion-VLAN no shutdown mtu 9216!interface vlan3939 description VxRail-Disc-VLAN no shutdown mtu 9216!interface port-channel102 description Po-2-Core-3

no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 vlt-port-channel 102!interface mgmt1/1/1 no shutdown no ip address dhcp ip address 172.17.180.36/24 ipv6 address autoconfig!interface ethernet1/1/1 no shutdown switchport access vlan 3939 mtu 9216 flowcontrol receive on

!interface ethernet1/1/2 description Link-2-VxRail- Nodes no shutdown switchport mode trunk

no shutdown mtu 9216!interface vlan40 description Marketing-VLAN no shutdown mtu 9216!interface vlan1111 description Mgmt-VLAN no shutdown mtu 9216!interface vlan1312 description vSAN-VLAN no shutdown mtu 9216!interface vlan1412 description vMotion-VLAN no shutdown mtu 9216!interface vlan3939 description VxRail-Disc-VLAN no shutdown mtu 9216!interface port-channel102 description Upstream- LAG-102 no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 vlt-port-channel 102!interface mgmt1/1/1 no shutdown no ip address dhcp ip address 172.17.180.35/24 ipv6 address autoconfig!interface ethernet1/1/1 no shutdown switchport access vlan 3939 mtu 9216 flowcontrol receive on

!interface ethernet1/1/2 description Link-2-VxRail- Nodes no shutdown switchport mode trunk






switchport mode trunk switchport access vlan 1

switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 flowcontrol receive on!interface ethernet1/1/3

no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 flowcontrol receive on!interface ethernet1/1/4 no shutdown

switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216

flowcontrol receive on!interface ethernet1/1/5 shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/6 shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/7 shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/8 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/9 description Link-2-Priv-172- VM no shutdown switchport mode trunk

switchport mode trunk switchport access vlan 1

switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 flowcontrol receive on!interface ethernet1/1/3

no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 flowcontrol receive on!interface ethernet1/1/4 no shutdown

switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216

flowcontrol receive on!interface ethernet1/1/5 shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/6 shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/7 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/8 shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/9 no shutdown

switchport access vlan 1 flowcontrol receive on

switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216

flowcontrol receive on!interface ethernet1/1/3 description Link-2-VxRail- Nodes no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 flowcontrol receive on!interface ethernet1/1/4 description Link-2-VxRail_ Nodes no shutdown switchport mode trunk switchport access vlan 1

switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 flowcontrol receive on!interface ethernet1/1/5 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/6 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/7 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/8 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/9

no shutdown switchport access vlan 1

switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216

flowcontrol receive on!interface ethernet1/1/3 description Link-2-VxRail- Nodes no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 flowcontrol receive on!interface ethernet1/1/4 description Link-2-VxRail- New no shutdown switchport mode trunk switchport access vlan 1

switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 flowcontrol receive on!interface ethernet1/1/5 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/6 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/7 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/8 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/9







switchport access vlan 1 switchport trunk allowed vlan 1111 flowcontrol receive on!interface ethernet1/1/10 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/11 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/12 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/13 no shutdown no switchport flowcontrol receive on

!interface ethernet1/1/14

description Po101-link- members no shutdown channel-group 101 mode active no switchport mtu 9216 negotiation o� flowcontrol receive on!interface ethernet1/1/15 no shutdown switchport access vlan 1 flowcontrol receive on!interface virtual-network3939 description VxRail-NW

no shutdown!

management route 0.0.0.0/0 172.17.180.254!ip route 172.11.12.0/24 172.11.11.254!vlt-domain 1

discovery-interface ethernet1/1/13

flowcontrol receive on!

interface ethernet1/1/10 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/11 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/12 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/13 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/14


flowcontrol receive on


no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/16 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/17 no shutdown switchport access vlan 1


interface ethernet1/1/18





interface ethernet1/1/10 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/11 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/12 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/13 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/14


flowcontrol receive on


no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/16 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/17 no shutdown switchport access vlan 1








no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/11 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/12 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/13 no shutdown no switchport flowcontrol receive on!interface ethernet1/1/14 description Po101-link- members no shutdown channel-group 101 mode active no switchport

mtu 9216 negotiation o�

flowcontrol receive on!interface ethernet1/1/15 no shutdown switchport access vlan 1 flowcontrol receive on!interface virtual-network3939 description VxRail-NW no shutdown!management route 0.0.0.0/0 172.17.180.254!ip route 172.11.12.0/24 172.11.11.254!

vlt-domain 1 discovery-interface ethernet1/1/13!snmp-server contact "Contact Support"!





!snmp-server contact "Contact Support"!telemetry


flowcontrol receive on!interface ethernet1/1/20 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/21 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/22 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/23 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/24 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/25 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/26 description Po102-link- member no shutdown channel-group 102 mode active no switchport mtu 9216 negotiation o� flowcontrol receive on!interface ethernet1/1/27 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/28 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/29 no shutdown no switchport


flowcontrol receive on!interface ethernet1/1/20 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/21 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/22 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/23 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/24 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/25 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/26 description Po102-link- member no shutdown channel-group 102 mode active no switchport mtu 9216 negotiation o� flowcontrol receive on!interface ethernet1/1/27 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/28 no shutdown switchport access vlan 1 flowcontrol receive on!interface ethernet1/1/29 no shutdown no switchport

telemetryS4112F-02#






flowcontrol receive on!interface ethernet1/1/30 no shutdown switchport access vlan 1 flowcontrol receive on!management route 0.0.0.0/0 172.17.180.254!ip route 172.11.12.0/24 172.11.11.254!vlt-domain 1 discovery-interface ethernet1/1/29!snmp-server contact "Contact Support"!telemetryS4128F-01#

flowcontrol receive on!interface ethernet1/1/30 no shutdown switchport access vlan 1 flowcontrol receive on!management route 0.0.0.0/0 172.17.180.254!ip route 172.11.12.0/24 172.11.11.254!vlt-domain 1 discovery-interface ethernet1/1/29!snmp-server contact "Contact Support"!telemetryS4128F-02#

The Pluribus Ede DCI switches in Figure 2 are directly connected to a border-leaf (Dell) devices in a datacenter. All the Pluribus core switches in Figure 2 are running Pluribus Netvisor operating system and provides connectivity between data centers.

Configuring the underlay network involves the configuration of (at least) the following base features:

The following configuration is for switches Site-A:Core-1, Site-C:Core-2(witness) and Site-B:Core-3 only.

Detailed Underlay (Day-0) configurations steps:


Enable physical port, speed & mtu for jumbo frames

Configure port channel

Configure the in-band IP for fabric communication

Enable OSPF to ensure in-band IP reachability across the all core node

Create the Fabric and define all the in-band subnets of the nodes that will be part of the Fabric

Configure the rest of the core switch across sites to join the existing Fabric

Configure the VXLAN Tunnel End-Points (VTEPs)

[Optional] Reserve a loopback port for VXLAN routing packet recirculation. Not required with Dell S52xx and S41xx ON switches.

Step 1Step 2Step 3

Step 4

Step 5

Step 6Step 7Step 8

About this taskNOTE: The Pluribus cluster (High-Availability) configuration is not described in this guide, however Pluribus recommended to use cluster for redundancy and high-availability.

Core-1 Core-3 Core-2 (witness)

!Enable Port connecting to Core-1 and Leaf1 (S4112F-01) & Leaf2 (S4112F-02)

CLI (network-admin@Core-1) > port-config-modify port 33 speed 100g enable jumbo


!Enable Port connecting to Core-2





Enable Port connecting to Core-3 and Leaf3 (s4128F-01) & Leaf4 (s4128F-02)








!Enable Port connecting to Core-2 and Leaf1








!Port-phy-show

CLI (network-admin@Core-1) > port-phy-show port 9,33,41,89port state speed eth-mode max-frame def-vlan ---- ----- ------ -------- --------- -------- 9 up 40000 sr4 9412 4092 33 up 100000 caui4 9412 1 41 up 100000 caui4 9412 1 89 up 40000 sr4 9412 1

!Port-phy-show

CLI (network-admin@Core-3) > port-phy-show port 9,89,113,121port state speed eth-mode max-frame def-vlan ---- ----- ------ -------- --------- -------- 9 up 40000 sr4 9412 4092 89 up 40000 sr4 9412 1 113 up 100000 caui4 9412 1 121 up 100000 caui4 9412 1

!Port-phy-show

CLI (network-admin@Core-2) > port-phy-show port 1,2,57,61port state speed eth-mode max-frame def-vlan ---- ----- ----- -------- --------- -------- 1 up 10000 xfi 9412 1 2 up 10000 xfi 9412 1 57 up 40000 sr4 9412 4091 61 up 40000 sr4 9412 4092


Step 1 : Enable Physical port, speed and MTU for jumbo frames

A�er plugging in the cables, login to each Pluribus core switch and use the following command to enable port, speed, and increase the MTU value to jumbo.

Use the following sample configuration for Core-1, Core-2 (witness)and Core-3

Validation Steps:Use the following command to verify on Core-1, Core-2 (witness)and Core-3

NOTE: VXLAN encapsulation adds between 50 and 54 bytes of additional header information to the original Ethernet frame. Because this can result in Ethernet frames that exceed the default 1514-byte MTU, best practice is to implement jumbo frames throughout the network.

Core-1

!Enable Port-101 connecting to Core-1 and Leaf1 (S4112F-01) & Leaf2 (S4112F-02)

CLI (network-admin@Core-1) >trunk-create name PO-101 ports 33,41 jumbo lacp-mode active

Core-3

!Enable Port-102 connecting to Core-2 and Leaf3 (s4128F-01) & Leaf4 (s4128F-02)

CLI (network-admin@Core-3) >trunk-create name PO-102 ports 113,121 jumbo lacp-mode active

Site-1 Core-1

!Verify Port-Channel 101

CLI (network-admin@Core-1) > trunk-show name PO-101 layout verticaltrunk-id: 272switch: Core-1name: PO-101ports: 33,41speed: 100gbw-bps: 200Genable: onlacp-mode: activelacp-timeout: slowlacp-fallback: bundlelacp-fallback-timeout: 50edge-switch: novxlan-termination: onstatus: up,uplink,STP-BPDUsconfig: fd,jumbodefer-bringup: nohost-enable: yesCLI (network-admin@Core-1*) >

Site-2 Core-3

!Verify Port-Channel 102

CLI (network-admin@Core-3) > trunk-show name PO-102 layout vertical trunk-id: 272switch: Core-3name: PO-102ports: 113,121speed: 100gbw-bps: 200Genable: onlacp-mode: activelacp-timeout: slowlacp-fallback: bundlelacp-fallback-timeout: 50edge-switch: novxlan-termination: onstatus: up,uplink,STP-BPDUsconfig: jumbodefer-bringup: nohost-enable: yesCLI (network-admin@Core-3*) >

Step 2 : Configure Port-Channel between Core and Border-Leaf’s

Configure Port-channel with LACP to provide more bandwidth, load balance tra�ic across multiple physical connected links and redundancy.

Use the following sample configuration for Core-1 and Core-3

Step 3 : Configure the in-band IP for fabric communication

This step focuses on setting up the Adaptive Cloud Fabric control plane communication using the in-band management interface which is reachable through the switch front panel ports carrying the network tra�ic.

In order to behave as a unified single-logical system, all switches that are part of an Adaptive Cloud Fabric need to exchange fabric-related messages such as: control-plane messages, configuration transactions, network and endpoint state notifications, remote procedure calls and file system replication messages. This communication is always encrypted and can be enabled via an out-of-band management network or via the in-band network. In-band communication has two main advantages: first it removes the dependence on an external management network, second it leverages the high-availability and resilience of production network.

The following steps configure a dedicated in-band management network on each switch using an internal VLAN.

Use the following sample configuration for Core-1,Core-2 (witness) and Core-3


Validation Steps:Use the following command to verify Port-Channel on Core-1 and Core-3


!Modify switch In-band IP

CLI (network-admin@Core-1) > switch-setup-modify in-band-ip 10.20.1.1/24

!Configure Internal VLAN Fabric Communication

CLI (network-admin@Core-1) > fabric-local-modify vlan 4050

CLI (network-admin@Core-1) > vlan-create id 4050 scope local

!Configure Internal VLAN SVI & Switch-Route for reachability

CLI (network-admin@Core-1) > vrouter-interface-add vrouter-name Core-1 nic eth1.4050 ip 10.20.1.2/24 vlan 4050

CLI (network-admin@Core-1) > switch-route-create network 10.20.0.0/16 gateway-ip 10.20.1.2

















Site-1 Core-1 Site-2 Core-3 Site-3 Core-2 (witness)

Switch-setup-show

CLI (network-admin@Core-1) > switch-route-show network gateway-ip nic ----------- ---------- ----- 10.20.0.0/16 10.20.1.2 vdata0

CLI (network-admin@Core-1*) > CLI (network-admin@Core-1*) > CLI (network-admin@Core-1*) > switch-setup-showswitch-name: Core-1mgmt-ip: 172.17.180.14/24mgmt-ip-assignment: staticmgmt-ip6: fe80::56bf:64ff:febd:bc40/64mgmt-ip6-assignment: autoconfmgmt-link-state: upmgmt-link-speed: 1gin-band-ip: 10.20.1.1/24in-band-ip6: fe80::640e:94ff:fedf:bf65/64in-band-ip6-assign: autoconf gateway-ip: 172.17.180.254ntp-server: 0.ubuntu.pool.ntp.orgntp-secondary-server: 1.ubuntu.pool.ntp.orgtimezone:

!Switch-setup-show

CLI (network-admin@Core-3) > switch-setup-show switch-name: Core-3mgmt-ip: 172.17.180.15/24mgmt-ip-assignment: staticmgmt-ip6: fe80::56bf:64ff:febd:aec0/64mgmt-ip6-assignment: autoconfmgmt-link-state: upmgmt-link-speed: 1gin-band-ip: 10.20.3.1/24in-band-ip6: fe80::640e:94ff:fede:f6a5/64in-band-ip6-assign: autoconfgateway-ip: 172.17.180.254ntp-server: 0.ubuntu.pool.ntp.orgntp-secondary-server: 1.ubuntu.pool.ntp.orgtimezone: America/Los_Angelesdate: 2021-04-10,17:53:03hostid: 184558814location-id: 3enable-host-ports: yesbanner: * Welcome to Pluribus Networks Inc.

!Switch-setup-show

CCLI (network-admin@Core-2) > switch-setup-show switch-name: Core-2mgmt-ip: 169.254.1.1/24mgmt-ip-assignment: dhcpmgmt-ip6: fe80::1618:77ff:fe18:ab8/64mgmt-ip6-assignment: autoconfmgmt-link-state: upmgmt-link-speed: 1gin-band-ip: 10.20.2.1/24in-band-ip6: fe80::640e:94ff:fef8:f86d/64in-band-ip6-assign: autoconfntp-server: 0.ubuntu.pool.ntp.orgntp-secondary-server: 1.ubuntu.pool.ntp.orgtimezone: America/Los_Angelesdate: 2021-04-10,19:03:01hostid: 184558840location-id: 4


Validation Steps:Use the following command to verify on Core-1, Core-2 (witness) and Core-3


America/Los_Angelesdate: 2021-04-03,05:52:47hostid: 184558815location-id: 2enable-host-ports: yesbanner: * Welcome to Pluribus Networks Inc. Netvisor(R). This is a monitored system. *device-id: GGLQG02ntp: onbanner: * ACCESS RESTRICTED TO AUTHORIZED USERS ONLY *banner: * By using the Netvisor(R) CLI,you agree to the terms of the Pluribus Networks *banner: * End User License Agreement (EULA). The EULA can be accessed via *banner: * http://www.pluribusnet-works.com/eula or by using the command "eula-show" *

CLI (network-admin@Core-1*) >

!Vrouter-interface-show

CLI (network-admin@Core-1) > vrouter-interface-show nic eth1.4050 vrouter-name nic ip mac vlan vlan-type nic-state fabric-nic mtu priority-tag ------------ --------- ------------ ----------------- ---- --------- --------- ---------- ---- ------------ Core-1 eth1.4050 10.20.1.2/24 66:0e:94:df:f4:f5 4050 public up fabric-nic 1500 off CLI (network-admin@Core-1) >

!Switch-route-show

CLI (network-admin@Core-1) > switch-route-show network gateway-ip nic ----------- ---------- ------ 10.20.0.0/16 10.20.1.2 vdata0

Netvisor(R). This is a monitored system. *device-id: GGHQG02ntp: onbanner: * ACCESS RESTRICTED TO AUTHORIZED USERS ONLY *banner: * By using the Netvisor(R) CLI,you agree to the terms of the Pluribus Networks *banner: * End User License Agreement (EULA). The EULA can be accessed via *banner: * http://www.pluribusnet-works.com/eula or by using the command "eula-show" *CLI (network-admin@Core-3*) >


CLI (network-admin@Core-3) > vrouter-interface-show nic eth1.4050 vrouter-name nic ip mac vlan vlan-type nic-state fabric-nic mtu priority-tag ------------ --------- ------------ ----------------- ---- --------- --------- ---------- ---- ------------ Core-3 eth1.4050 10.20.3.2/24 66:0e:94:de:6c:fc 4050 public up fabric-nic 1500 off CLI (network-admin@Core-3) >

!Switch-route-show

CLI (network-admin@Core-3) > switch-route-show network gateway-ip nic ------------ ---------- ------ 10.20.0.0/16 10.20.3.2 vdata0 CLI (network-admin@Core-3) >

enable-host-ports: yesbanner: * Welcome to Pluribus Networks Inc. Netvisor(R). This is a monitored system. *device-id: 9K31XC2ntp: onbanner: * ACCESS RESTRICTED TO AUTHORIZED USERS ONLY *banner: * By using the Netvisor(R) CLI,you agree to the terms of the Pluribus Networks *banner: * End User License Agreement (EULA). The EULA can be accessed via *banner: * http://www.pluribusnet-works.com/eula or by using the command "eula-show" *

CLI (network-admin@Core-2) >


CLI (network-admin@Core-2*) > vrouter-interface-show nic eth0.4050 vrouter-name nic ip mac vlan vlan-type nic-state fabric-nic mtu priority-tag ------------ --------- ------------ ----------------- ---- --------- --------- ---------- ---- ------------ Core-2 eth0.4050 10.20.2.2/24 66:0e:94:f8:98:4a 4050 public up fabric-nic 1500 off CLI (network-admin@Core-2) >

!Switch-route-show

CLI (network-admin@Core-2) > switch-route-show network gateway-ip nic ----------- ---------- ------ 10.20.0.0/16 10.20.2.2 vdata0 CLI (network-admin@Core-2) >


Step 4 : Enable OSPF and advertise network to ensure In-band IP reachability across all the Core nodes

Use the following sample configuration for Core-1, Core-2 (witness) and Core-3


!Configure VLAN & SVI (Layer-3) connecting to Core-3

CLI (network-admin@Core-1) > vlan-create id 4000 scope local ports 89,97

CLI (network-admin@Core-1) > vrouter-interface-add vrouter-name Core-1 vlan 4000 ip 10.1.3.1/30 mtu 9216

!Note: 2 links between Core-1 & Core-3 will become auto-trunk

!Assign IP-address on physical connecting to Core-2

CLI (network-admin@Core-1) > vrouter-interface-add vrouter-name Core-1 l3-port 9 ip 10.1.2.1/30 mtu 9216

!Enable OSPF and Redistribute routes

CLI (network-admin@Core-1) > vrouter-modify name Core-1 router-id 10.10.1.1 ospf-redistribute connected ospf-redistribute connected

!Configure point-to-point OSPF and BFD connecting to Core-2

CLI (network-admin@Core-1) > vrouter-interface-config-add vrouter-name Core-1 l3-port 9 ospf-hello-interval 2 ospf-dead-interval 8 ospf-network-type point-to-point ospf-bfd default

Configure point-to-point OSPF and BFD connecting to Core-3

CLI (network-admin@Core-1) > vrouter-interface-config-add vrouter-name Core-1 nic eth2.4000 ospf-hello-interval 2 ospf-dead-interval 8 ospf-network-type point-to-point ospf-bfd default

!Advertise connected network into OSPF

CLI (network-admin@Core-1) > vrouter-ospf-add vrouter-name Core-1 network 10.1.2.0/30 ospf-area 0CLI (network-admin@Core-1) > vrouter-ospf-add vrouter-name Core-1 network 10.1.3.0/30 ospf-area 0

!Configure VLAN & SVI (Layer-3) connecting to Core-1

CLI (network-admin@Core-3) > vlan-create id 4000 scope local ports 89,97

CLI (network-admin@Core-3) > vrouter-interface-add vrouter-name Core-3 vlan 4000 ip 10.1.3.2/30 mtu 9216

!Note: 2 links between Core-1 & Core-3 will become auto-trunk






CLI (network-admin@Core-3) > vrouter-interface-config-add vrouter-name Core-3 nic eth2.4000 ospf-hello-interval 2 ospf-dead-interval 8 ospf-network-type point-to-point ospf-bfd default

Configure point-to-point OSPF and BFD connecting to Core-2


















Vrouter-interface show

e,mtunetwork-admin@Core-1) > vrouter-interface-show format nic,ip,vlan,nic-state,mtuvrouter-name nic ip vlan nic-state mtu ------------ --------- ---------------- ---- --------- ---- Core-1 eth0.4092 10.1.2.1/30 4092 up 9216 Core-1 eth2.4000 10.1.3.1/30 4000 up 9216

!Vrouter-ospf-interface-show

CLI (network-admin@Core-1) > vrouter-ospf-interface-show vrouter-name nic l3-port area mtu state hello-due neighbor ------------ --------- ------- ------- ---- -------------- --------- -------- Core-1 eth0.4092 9 0.0.0.0 9216 point-to-point 0.864s 1 Core-1 eth2.4000 0.0.0.0 9216 point-to-point 1.168s 1

!Vrouter-ospf-neighbor-show

CLI (network-admin@Core-1) > vrouter-ospf-neighbor-show format neighbor,pri,State,address,interface,vlanvrouter-name neighbor pri State address interface vlan ------------ --------- --- ------------ -------- ------------------ ---- Core-1 10.10.3.1 1 Full/DROther 10.1.3.2 eth2.4000:10.1.3.1 4000 Core-1 10.10.2.1 1 Full/DROther 10.1.2.2 eth0.4092:10.1.2.1

!Vrouter-routes-show

CLI (network-admin@Core-1) > vrouter-routes-show vrouter-name network type interface next-hop distance metric vrf ------------ -------------- --------- --------- -------- -------- ------ --- Core-1 10.1.2.0/30 connected eth0.4092 Core-1 10.1.3.0/30 connected eth2.4000

!Vrouter-interface show

CLI (network-admin@Core-3) > vrouter-interface-show format nic,ip,vlan,nic-state,mtu vrouter-name nic ip vlan nic-state mtu ------------ --------- --------------- ---- --------- ---- Core-3 eth2.4000 10.1.3.2/30 4000 up 9216 Core-3 eth1.4092 10.2.3.2/30 4092 up 9216


CLI (network-admin@Core-3) > vrouter-ospf-interface-showvrouter-name nic l3-port area mtu state hello-due neighbor ------------ --------- ------- ------- ---- -------------- --------- -------- Core-3 eth1.4092 9 0.0.0.0 9216 point-to-point 0.357s 1 Core-3 eth2.4000 0.0.0.0 9216 point-to-point 0.463s 1


CLI (network-admin@Core-3) > vrouter-ospf-neighbor-show format neighbor,pri,State,address,interface,vlan

vrouter-name neighbor pri State address interface vlan ------------ --------- --- ------------ -------- ------------------ ---- Core-3 10.10.1.1 1 Full/DROther 10.1.3.1 eth2.4000:10.1.3.2 4000 Core-3 10.10.2.1 1 Full/DROther 10.2.3.1 eth1.4092:10.2.3.2


CLI (network-admin@Core-3) > vrouter-routes-show vrouter-name network type interface next-hop distance metric vrf ------------ -------------- --------- --------- -------- -------- ------ --- Core-3 10.1.2.0/30 ospf eth2.4000 10.1.3.1 110 20 Core-3 10.1.2.0/30


CLI (network-admin@Core-2) > vrouter-interface-show format nic,ip,vlan,nic-state,mtu vrouter-name nic ip vlan nic-state mtu ------------ --------- ---------------- ---- --------- ---- Core-2 eth0.4092 10.1.2.2/30 4092 up 9216 Core-2 eth1.4091 10.2.3.1/30 4091 up 9216


CLI (network-admin@Core-2) > vrouter-ospf-interface-show vrouter-name nic l3-port area mtu state hello-due neighbor ------------ --------- ------- ------- ---- -------------- --------- -------- Core-2 eth0.4092 61 0.0.0.0 9216 point-to-point 0.068s 1 Core-2 eth1.4091 57 0.0.0.0 9216 point-to-point 0.068s 1



vrouter-name neighbor pri State address interface ------------ --------- --- ------------ -------- ------------------ Core-2 10.10.1.1 1 Full/DROther 10.1.2.1 eth0.4092:10.1.2.2 Core-2 10.10.3.1 1 Full/DROther 10.2.3.2 eth1.4091:10.2.3.1


CLI (network-admin@Core-2) > vrouter-routes-show vrouter-name network type interface next-hop distance metric vrf ------------ -------------- --------- --------- -------- -------- ------ --- Core-2 10.1.2.0/30 connected eth0.4092 Core-2 10.1.3.0/30

Validation Steps:Use below command to verify Interface state, ospf neighbor and routes reachability between Core-1, Core-2 (witness) and Site2:Core-3



Core-1 10.2.3.0/30 ospf eth0.4092 10.1.2.2 110 20 Core-1 10.2.3.0/30 ospf eth2.4000 10.1.3.2 110 20 Core-1 10.13.11.0/24 ospf eth0.4092 10.1.2.2 110 20 Core-1 10.13.12.0/24 connected eth1.1312 Core-1 10.14.12.0/24 connected eth1.1412 Core-1 10.20.1.0/24 connected eth1.4050 Core-1 10.20.2.0/24 ospf eth0.4092 10.1.2.2 110 20 Core-1 10.20.3.0/24 ospf eth2.4000 10.1.3.2 110 20 Core-1 40.40.1.0/29 connected eth0.4040 Core-1 40.40.2.0/29 ospf eth0.4092 10.1.2.2 110 20 Core-1 40.40.3.0/29 ospf eth2.4000 10.1.3.2 110 20 Core-1 172.11.11.0/24 connected eth3.1111 Core-1 172.11.12.0/24 ospf eth0.4092 10.1.2.2 110 20

CLI (network-admin@Core-1) >

ospf eth1.4092 10.2.3.1 110 20 Core-3 10.1.3.0/30 connected eth2.4000 Core-3 10.2.3.0/30 connected eth1.4092 Core-3 10.13.11.0/24 ospf eth1.4092 10.2.3.1 110 20 Core-3 10.13.12.0/24 connected eth0.1312 Core-3 10.14.12.0/24 connected eth0.1412 Core-3 10.20.1.0/24 ospf eth2.4000 10.1.3.1 110 20 Core-3 10.20.2.0/24 ospf eth1.4092 10.2.3.1 110 20 Core-3 10.20.3.0/24 connected eth1.4050 Core-3 40.40.1.0/29 ospf eth2.4000 10.1.3.1 110 20 Core-3 40.40.2.0/29 ospf eth1.4092 10.2.3.1 110 20 Core-3 40.40.3.0/29 connected eth0.4040 Core-3 172.11.11.0/24 ospf eth2.4000 10.1.3.1 110 20 Core-3 172.11.12.0/24 ospf eth1.4092 10.2.3.1 110 20 CLI (network-admin@Core-3) >

ospf eth0.4092 10.1.2.1 110 20 Core-2 10.1.3.0/30 ospf eth1.4091 10.2.3.2 110 20 Core-2 10.2.3.0/30 connected eth1.4091 Core-2 10.13.11.0/24 connected eth0.1311 Core-2 10.13.12.0/24 ospf eth0.4092 10.1.2.1 110 20 Core-2 10.13.12.0/24 ospf eth1.4091 10.2.3.2 110 20 Core-2 10.14.12.0/24 ospf eth0.4092 10.1.2.1 110 20 Core-2 10.14.12.0/24 ospf eth1.4091 10.2.3.2 110 20 Core-2 10.20.1.0/24 ospf eth0.4092 10.1.2.1 110 20 Core-2 10.20.2.0/24 connected eth0.4050 Core-2 10.20.3.0/24 ospf eth1.4091 10.2.3.2 110 20 Core-2 40.40.1.0/29 ospf eth0.4092 10.1.2.1 110 20 Core-2 40.40.2.0/29 connected eth0.4040 Core-2 40.40.3.0/29 ospf eth1.4091 10.2.3.2 110 20 Core-2 172.11.11.0/24 ospf eth0.4092 10.1.2.1 110 20 Core-2 172.11.12.0/24 connected eth4.1112 CLI (network-admin@Core-2)

Site-1 Core-1

!Create fabric

CLI (network-admin@Core-1) > fabric-create name PN-DCI vlan 4050 fabric-network in-band control-network in-band fabric-advertisement-network inband-only


Step 5 : Create the Fabric and define all the in-band subnets of the nodes that will be part of the fabric

A�er completing the initial setup, the OSPF underlay, and the in-band properties, it is now time to create a new fabric instance to subsequently add all other switches.

Choose any switch to create the fabric instance and use the optional password option, so that other switches can join the fabric only by providing the assigned password.

Use the following sample configuration for Core-1

Site-2 Core-3

!Add members to the fabric

CLI (network-admin@Leaf-2) > fabric-join switch-ip 10.20.1.1 vlan 4050

Joined fabric PN-DCI, Restarting nvOS…Please enter username and password: Username (network-admin): Password:

Site-3 Core-2 (witness)

!Add members to the fabric

CLI (network-admin@Leaf-3) > fabric-join switch-ip 10.20.1.1 vlan 4050

Joined fabric PN-DCI, Restarting nvOS…Please enter username and password: Username (network-admin): Password:

Step 6 : Configure the rest of the core switches across all sites to join the existing Fabric

Use below command to join fabric on Core-2 (witness) and Core-3

Step 7 : Configure the VXLAN Tunnel End-Points (VTEPs) Each Core is configured with a VTEP. VxLAN uses VxLAN tunnel end-point (VTEP) to map tenants to segments and performs VXLAN encapsulation and de-capsulation on all packets. By creating the VTEP objects above, the Fabric will dynamically create a mesh of interconnecting tunnels among them without any additional operational e�ort.


Validation Steps:Use below command to verify fabric-node-show on any Core-1, Core-2 (witness) or Core-3

Core-1

!fabric-node-show

CLI (network-admin@Core-1) > fabric-node-showname fab-name mgmt-ip in-band-ip in-band-vlan-type fab-tid out-port version state firmware-upgrade device-state ------ -------- ---------------- ------------ ----------------- ------- -------- ---------------- ------ Core-1 PN-DCI 172.17.180.14/24 10.20.1.1/24 public 13 6.0.3-6000317001 online not-required ok Core-3 PN-DCI 172.17.180.15/24 10.20.3.1/24 public 13 89 6.0.3-6000317001 online not-required ok Core-2 PN-DCI 10.20.2.1/24 public 13 61 6.0.3-6000317001 online not-required ok CLI (network-admin@Core-1) >

Core-1

!Create VTEP-VLAN

CLI (network-admin@Core-1) > vlan-create id 4040 scope fabric description VTEP-VLAN ports none


NOTE: A�er fabric-join, it will restart nvOS and the switch join the fabric.

NOTE: Pluribus logical VTEP is not described in this guide, however Pluribus recommends using cluster for VTEP-HA (high-availability). More information can be found here.

NOTE: Single command can be also used to create the VLAN with fabric-scope, and configure SVI individually using switch-local.

Validation Steps:Use below command to verify vlan-show <id> on any Core-1

Core-1

!Vlan-show <id> CLI (network-admin@Core-1) > vlan-show id 4040switch id type auto-vxlan replicators scope description active stats ports untagged-ports active-edge-ports------ ---- ------ ---------- ----------- ----- ----------- ------ ----- ----- -------------- -----------------Core-1 4040 public no none fabric VTEP-VLAN yes yes 0 none 0Core-3 4040 public no none fabric VTEP-VLAN yes yes 0 none 0Core-2 4040 public no none fabric VTEP-VLAN yes yes 0 none 0

Validation Steps:Use below command to verify vtep-show on any Core-1, Core-2 (witness) or Core-3


!Run commands locally on a switch by using theCLI (network-admin@Core-1) > switch-local

!Create VTEP-VLAN-SVICLI (network-admin@Core-1)*> vrouter-interface-add vrouter-name Core-1 vlan 4040 ip 40.40.1.1/29 mtu 9216

!Advertise VTEP-IP in OSPFCLI (network-admin@Core-1)*> vrouter-ospf-add vrouter-name Core-1 network 40.40.1.0/29 ospf-area 0

!Enable VTEPCLI (network-admin@Core-1)*> vtep-create name VTEP-Core-1 location Core-1 vrouter-name Core-1 ip 40.40.1.1


!Create VTEP-VLAN-SVICLI (network-admin@Core-3)* > vrouter-interface-add vrout-er-name Core-3 vlan 4040 ip 40.40.3.1/29 mtu 9216




!Create VTEP-VLAN-SVICLI (network-admin@Core-2)* > vrouter-interface-add vrouter-name Core-2 vlan 4040 ip 40.40.2.1/29 mtu 9216




!Vtep-show

CLI (network-admin@Core-1*) > vtep-showscope name location vrouter-name ip virtual-ip mac-learning ------ ----------- -------- ------------ --------- ---------- ------------ fabric VTEP-Core-1 Core-1 Core-1 40.40.1.1 :: on fabric VTEP-Core-3 Core-3 Core-3 40.40.3.1 :: on fabric VTEP-Core-2 Core-2 Core-2 40.40.2.1 :: on

!Vtep-show


!Vtep-show



NOTE: Switch <switch-name> command can be also used to login into another switch in the fabric, but in this configuration will login individually and use the <switch-local> to configure locally.


CLI (network-admin@Core-2) > vrouter-interface-show format nic,ip,vlan,nic-state,mtu vrouter-name nic ip vlan nic-state mtu ------------ --------- ---------------- ---- --------- ---- Core-2 eth0.4092 10.1.2.2/30 4092 up 9216 Core-2 eth1.4091 10.2.3.1/30 4091 up 9216


CLI (network-admin@Core-2) > vrouter-ospf-interface-show vrouter-name nic l3-port area mtu state hello-due neighbor ------------ --------- ------- ------- ---- -------------- --------- -------- Core-2 eth0.4092 61 0.0.0.0 9216 point-to-point 0.068s 1 Core-2 eth1.4091 57 0.0.0.0 9216 point-to-point 0.068s 1



vrouter-name neighbor pri State address interface ------------ --------- --- ------------ -------- ------------------ Core-2 10.10.1.1 1 Full/DROther 10.1.2.1 eth0.4092:10.1.2.2 Core-2 10.10.3.1 1 Full/DROther 10.2.3.2 eth1.4091:10.2.3.1


CLI (network-admin@Core-2) > vrouter-routes-show vrouter-name network type interface next-hop distance metric vrf ------------ -------------- --------- --------- -------- -------- ------ --- Core-2 10.1.2.0/30 connected eth0.4092 Core-2 10.1.3.0/30



!Tunnel-showCLI (network-admin@Core-1*) > tunnel-show format name type,local-ip,remote-ip, state,name type local-ip remote-ip state ------------------------------- ----- --------- --------- ----- auto-tunnel- 40.40.1.1_40.40.3.1 vxlan 40.40.1.1 40.40.3.1 ok auto-tunnel- 40.40.1.1_40.40.2.1 vxlan 40.40.1.1 40.40.2.1 ok





!Enable VxLAN-loopback-trunkCLI (network-admin@Core-1)* > trunk-modify name vxlan-loopback-trunk ports 69 (Using for Riot).






!Trunk-showCLI (network-admin@Core-1*) > trunk-show name vxlan-loopback-trunk format trunk-id,switch,name,ports,vxlan-termination,statustrunk-id switch name ports vxlan-termination status -------- ------ -------------------- ----- ----------------- ---------------- 397 Core-1 vxlan-loopback-trunk 69 off up,stp-edge-port



Step 8 : [Optional] Reserve a port for VXLAN loopback / packet recirculation This step is required only if the ACF switches are not part of the S52xx or S41xx family.

It is necessary to dedicate a front-panel port for handling broadcast and unknown multicast (BUM) tra�ic that needs to be sent across the mesh of VXLAN tunnels. In fact, an internal physical loopback facility is required to recirculate this tra�ic in order to perform Head End Replication (HER) and send a unicast packet to each remote VTEP associated to the same Layer 2 extended segment or VNI.

NOTE: Certain hardware platforms have internal ports available for this task, so this step is not necessary with those devices.


Validation Steps:Use below command to verify trunk-show on any Core-1, Core-2 (witness) or Core-3


Step 9 : Configure stretched VLANs with VxLAN (VNI) & Auto-VxLAN VxLAN overlay networks, each stretched VLAN is mapped to a VNI number representing a VxLAN segment.

Create a VLAN either by specifying a VLAN ID and auto-vlxan identifier for the tunnel with no ports tagging,

NOTE: VxLAN (VNI) ID will auto map with VTEP.

Use the following commands with fabric scope for configuring the stretched VLANs with associated anycast gateway addresses and the associated VNIs for Core-1, Core-2 (witness) and Core-3

Figure 3 Stretched VLANs (layer-2) details


Configuring the Overlay network involves the configuration of (at least) the following base features:Detailed Overlay (Day-1/2) configurations steps:

Configure stretched VLANs with VxLAN (VNI) & Auto-VxLAN

Configure routing and stretched subnets across sites

Step 9Step 10




!Assign the VLANs to ports connected to Dell switchesCLI (network-admin@Core-1)* > vlan-port-add vlan-id 1312 ports 33,41 CLI (network-admin@Core-1)* > vlan-port-add vlan-id 1412 ports 33,41CLI (network-admin@Core-1)* > vlan-port-add vlan-id 3939 ports 33,41CLI (network-admin@Core-1)* > vlan-port-add vlan-id 1111 ports 33,41CLI (network-admin@Core-1)* > vlan-port-add vlan-id 30 ports 33,41CLI (network-admin@Core-1)* > vlan-port-add vlan-id 40 ports 33,41


!Assign the VLANs to ports connected to Dell switchesCLI (network-admin@Core-3)* > vlan-port-add vlan-id 1312 ports 113,121 CLI (network-admin@Core-3)* > vlan-port-add vlan-id 1412 ports 113,121CLI (network-admin@Core-3)* > vlan-port-add vlan-id 3939 ports 113,121CLI (network-admin@Core-3)* > vlan-port-add vlan-id 1111 ports 113,121CLI (network-admin@Core-3)* > vlan-port-add vlan-id 30 ports 113,121CLI (network-admin@Core-1)* > vlan-port-add vlan-id 40 ports 113,121


!Assign the VLANs to ports connected to Dell switchesCLI (network-admin@Core-2)* > vlan-port-add vlan-id 1311 ports 1

CLI (network-admin@Core-2)* > vlan-port-add vlan-id 1112 ports 2

The following commands are used with switch-local to assign VLANs to associated ports for Core-1, Core-2 (witness) and Core-3

Core-1

!Create stretched vSAN VLAN

CLI (network-admin@Core-1) > vlan-create id 1312 scope fabric auto-vxlan vxlan 1001312 ports none

!Create stretched vMotion VLAN


!Create stretched VxRail Node Discovery VLAN


!Create stretched Mgmt VLAN


!Create stretched Witness Mgmt VLAN


!Create stretched Eng & Mkt VLAN

CLI (network-admin@Core-1) > vlan-create id 30 scope fabric auto-vxlan vxlan 1000030 ports none CLI (network-admin@Core-1) > vlan-create id 40 scope fabric auto-vxlan vxlan 1000040 ports none



!Create Layer-3 GW interfaceCLI (network-admin@Core-1)* > vrouter-interface-add vrouter-name Core-1 nic eth1.1312 ip 10.13.12.254/24 vlan 1312 mtu 9216

CLI (network-admin@Core-1)* > vrouter-interface-add vrouter-name Core-1 nic eth1.1412 ip 10.14.12.254/24 vlan 1412 mtu 9216





CLI (network-admin@Core-2)* > vrouter-interface-add vrouter-name Core-2 nic eth0.1311 ip 10.13.11.254/24 vlan 1311 vlan-type public if data mtu 9216 mtu 9216

Core-1

!Vlan-show <id> CLI (network-admin@Core-1) > vlan-show id 1312,1412,3939,1111,1112,30,40 format switch,id,vxlan,scope,description,ports,switch id vxlan scope description ports------ ---- ------- ------ ----------- --------------------Core-1 30 1000030 fabric vlan-30 0,33,41,69,272,397Core-3 30 1000030 fabric vlan-30 0,69,113,121,272,397Core-2 30 1000030 fabric vlan-30 0Core-1 40 1000040 fabric vlan-40 0,33,41,69,272,397Core-3 40 1000040 fabric vlan-40 0,69,113,121,272,397Core-2 40 1000040 fabric vlan-40 0Core-1 1111 1001111 fabric vlan-1111 0,33,41,69,272,397Core-3 1111 1001111 fabric vlan-1111 0,69,113,121,272,397Core-2 1111 1001111 fabric vlan-1111 0,69,397Core-1 1112 1001112 fabric vlan-1112 0,69,397Core-3 1112 1001112 fabric vlan-1112 0,69,397Core-2 1112 1001112 fabric vlan-1112 0,2,69,397Core-1 1312 1001312 fabric vlan-1312 0,33,41,69,272,397Core-3 1312 1001312 fabric vlan-1312 0,69,113,121,272,397Core-2 1312 1001312 fabric vlan-1312 0,69,397Core-1 1412 1001412 fabric vlan-1412 0,33,41,272Core-3 1412 1001412 fabric vlan-1412 0,113,121,272Core-2 1412 1001412 fabric vlan-1412 0Core-1 3939 1003939 fabric vlan-3939 0,33,41,272Core-3 3939 1003939 fabric vlan-3939 0,113,121,272Core-2 3939 1003939 fabric vlan-3939 0CLI (network-admin@Core-1) >

!VTEP-VxLAN-show CLI (network-admin@Core-1) > vtep-vxlan-showname vxlan isolated----------- ------- --------VTEP-Core-1 1000030 noVTEP-Core-1 1000040 noVTEP-Core-1 1001311 noVTEP-Core-1 1001111 noVTEP-Core-1 1001312 noVTEP-Core-1 1003939 noVTEP-Core-1 1001412 noVTEP-Core-1 1001112 noVTEP-Core-3 1000030 noVTEP-Core-3 1000040 noVTEP-Core-3 1001311 noVTEP-Core-3 1001111 noVTEP-Core-3 1001312 noVTEP-Core-3 1003939 noVTEP-Core-3 1001412 noVTEP-Core-3 1001112 noVTEP-Core-2 1000030 noVTEP-Core-2 1000040 noVTEP-Core-2 1001311 noVTEP-Core-2 1001111 noVTEP-Core-2 1001312 noVTEP-Core-2 1003939 noVTEP-Core-2 1001412 noVTEP-Core-2 1001112 noCLI (network-admin@Core-1) >


Validation Steps:Use the command below to verify vlan-show & vtep-vxlan-show from any Core-1, Core-2 (witness) or Core-3

Site-1 Core-1

!Create VRF fabric-wide using fabric scope CLI (network-admin@Core-1)> vrf-create name VRF-1 scope fabric

!Create Subnet fabric-wide using fabric scope CLI (network-admin@Core-1)> subnet-create name subnet-vxlan-1001111 scope fabric vxlan 1001111 network 172.11.11.0/24 anycast-gw-ip 172.11.11.254 vrf VRF-1 CLI (network-admin@Core-1)> subnet-create name subnet-vxlan-1001112 scope fabric vxlan 1001112 network 172.11.12.0/24 anycast-gw-ip 172.11.12.254 vrf VRF-1 CLI (network-admin@Core-1)> subnet-create name subnet-vxlan-1001311 scope fabric vxlan 1001311 network 10.13.11.0/24 anycast-gw-ip 10.13.11.254 vrf VRF-1 CLI (network-admin@Core-1)> subnet-create name subnet-vxlan-1001312 scope fabric vxlan 1001312 network 10.13.12.0/24 anycast-gw-ip 10.13.12.254 vrf VRF-1 CLI (network-admin@Core-1)> subnet-create name subnet-vxlan-1000030 scope fabric vxlan 1000030 network 30.30.30.0/24 anycast-gw-ip 30.30.30.254 vrf VRF-1 CLI (network-admin@Core-1)> subnet-create name subnet-vxlan-1000040 scope fabric vxlan 1000040 network 40.40.40.0/24 anycast-gw-ip 40.40.40.254 vrf VRF-1

Use the following sample configuration using fabric scope form any Core-1, Core-2(witness) and Core-3


Validation Steps:Use the command below to verify vrf-show and subnet-show from any Core-1, Core-2(witness) or Core-3

Step 9 : Routing between Stretched VLAN across sites vSAN VLANs between site 1 and site 2 are stretched, however vSAN VLAN on the witness site is routed to site 1 and site 2.

L3VNI (VRF) is used to route between L2VNIs (VLAN-1111, 1112, 1311, 1312, 30 & 40). The ingress VTEP will perform IP destination routing lookup. The encapsulated VXLAN tra�ic is sent between VTEPs where it is associated with L3VNI the VRF-1 in which VLAN-1111, 1112, 1311, 1312, 30 & 40 reside.

Figure 4 Layer-3 (L3VNI) details


To provide inter-vlan routing, Pluribus supports the recirculation technique with the dedicated for any fabric packets that need to be routed between two hosts in di�erent VNI-mapped VLANs.

Use the following sample configuration using switch-local form Core-1, Core-2(witness) and Core-3

Core-1

!vrf-show CLI (network-admin@Core-1) > vrf-showswitch name vnet scope anycast-mac vrf-gw vrf-gw2 vrf-gw-ip6 vrf-gw2-ip6 active hw-router-mac hw-vrid flags enable description------ ----- ---- ------ ----------------- ------ ------- ---------- ----------- ------ ----------------- ------- ------ ------ -----------Core-1 VRF-1 0:0 fabric 64:0e:94:40:00:02 :: :: :: :: yes 66:0e:94:df:08:33 1 subnet yesCore-2 VRF-1 0:0 fabric 64:0e:94:40:00:02 :: :: :: :: yes 66:0e:94:f8:ea:e3 1 subnet yesCore-3 VRF-1 0:0 fabric 64:0e:94:40:00:02 :: :: :: :: yes 66:0e:94:de:fd:75 1 subnet yesCLI (network-admin@Core-1) >

!subnet-show

CLI (network-admin@Core-1) > subnet-showswitch name scope vlan vxlan vrf network anycast-gw-ip packet-relay forward-proto state enable------ -------------------- ------ ---- ------- ----- -------------- ------------- ------------ ------------- ------------------------ ------Core-1 subnet-vxlan-1000030 fabric 30 1000030 VRF-1 30.30.30.0/24 30.30.30.254 disable dhcp ok yesCore-3 subnet-vxlan-1000030 fabric 30 1000030 VRF-1 30.30.30.0/24 30.30.30.254 disable dhcp ok yesCore-2 subnet-vxlan-1000030 fabric 30 1000030 VRF-1 30.30.30.0/24 30.30.30.254 disable dhcp ok yesCore-1 subnet-vxlan-1000040 fabric 40 1000040 VRF-1 40.40.40.0/24 40.40.40.254 disable dhcp ok yesCore-3 subnet-vxlan-1000040 fabric 40 1000040 VRF-1 40.40.40.0/24 40.40.40.254 disable dhcp ok yesCore-2 subnet-vxlan-1000040 fabric 40 1000040 VRF-1 40.40.40.0/24 40.40.40.254 disable dhcp ok yesCore-1 subnet-vxlan-1001111 fabric 1111 1001111 VRF-1 172.11.11.0/24 172.11.11.254 disable dhcp vrouter interface exists yesCore-3 subnet-vxlan-1001111 fabric 1111 1001111 VRF-1 172.11.11.0/24 172.11.11.254 disable dhcp ok yesCore-2 subnet-vxlan-1001111 fabric 1111 1001111 VRF-1 172.11.11.0/24 172.11.11.254 disable dhcp ok yesCore-1 subnet-vxlan-1001112 fabric 1112 1001112 VRF-1 172.11.12.0/24 172.11.12.254 disable dhcp ok yesCore-3 subnet-vxlan-1001112 fabric 1112 1001112 VRF-1 172.11.12.0/24 172.11.12.254 disable dhcp ok yesCore-2 subnet-vxlan-1001112 fabric 1112 1001112 VRF-1 172.11.12.0/24 172.11.12.254 disable dhcp vrouter interface exists yesCore-1 subnet-vxlan-1001311 fabric 1311 1001311 VRF-1 10.13.11.0/24 10.13.11.254 disable dhcp ok yesCore-3 subnet-vxlan-1001311 fabric 1311 1001311 VRF-1 10.13.11.0/24 10.13.11.254 disable dhcp ok yesCore-2 subnet-vxlan-1001311 fabric 1311 1001311 VRF-1 10.13.11.0/24 10.13.11.254 disable dhcp vrouter interface exists yesCore-1 subnet-vxlan-1001312 fabric 1312 1001312 VRF-1 10.13.12.0/24 10.13.12.254 disable dhcp vrouter interface exists yesCore-3 subnet-vxlan-1001312 fabric 1312 1001312 VRF-1 10.13.12.0/24 10.13.12.254 disable dhcp vrouter interface exists yesCore-2 subnet-vxlan-1001312 fabric 1312 1001312 VRF-1 10.13.12.0/24 10.13.12.254 disable dhcp ok yesCLI (network-admin@Core-1) >

Core-1

!trunk-show

CLI (network-admin@Core-1) > trunk-show format switch,name,portsswitch name ports------ -------------------- -------Core-1 PO-101 33,41Core-1 vxlan-loopback-trunk 69Core-3 PO-102 113,121Core-3 vxlan-loopback-trunk 69Core-2 vxlan-loopback-trunk 69CLI (network-admin@Core-1) >


Pluribus Networks’ UNUM is a unified fabric management, automation and analytics platform that enables administrators to significantly reduce fabric deployment times, execute everyday fabric management lifecycle tasks and conduct e�ective tra�ic analysis, all from the centralized graphical interface.

A single instance of Pluribus UNUM can seamlessly scale to visualize multiple sites data center interconnected in a single unified topology view.

Pluribus UNUM – Unified Automation, Management and Analytics

Validation Steps:Use below command to verify VxLAN-loopback-trunk from any Core-1, Core-2(witness) or Core-3


!Run commands locally on a switch by using the

CLI (network-admin@Core-1) > switch-local

!Add VLAN to VxLAN-loopback- trunk

CLI (network-admin@Core-1*) > vlan-port-add vlan-id 1111 ports 397,69








!Add VLAN to VxLAN-loopback -trunk









!Add VLAN to VxLAN-loopback- trunk





UNUM provides day-0 automation capabilities with respect to underlay fabric deployments and playbooks for Layer-2/3 and Overlay configuration.

Depending on the actual topology connections between devices are represented in the topology map in that case UNUM may display a topology as shown in the following example.


How UNUM simplifies stretched VLANs configuration across Multi-sites In this example, we are going to extend VLAN-30 (Eng) across site-A to site-B over VxLAN.

Select the Fabric from the le� side navigation bar to create VLAN across all switch in fabric or select an individual Switch from the fabric to create only the VLAN for that switch.

Figure 5 UNUM Deploy, Manage, Visualize Multi-Sites (DC-I) from single from pane of glass

NOTE: UNUM requires all hosts, switches, and servers to run LLDP if they are to appear in the Topology.


Click on Create a VLAN by id

Enter the parameter settings for: • Switch – select any switch in fabric • Scope – select fabric scope to extend VLAN across multi-sites • ID – configure VLAN ID • Description – Eng VLAN • Ports – None

Check the Stats – VLAN statistics box to receive VLAN statistical information.

Step 1: Click on Manager → Layer 2 → Manage VLAN


NOTE: We can add VLAN to port now, but in this example we will cover in next-step.

• VXLAN – VXLAN 1000030 mapped to VLAN for tunnel • Select Auto-VXLAN-Mode – VXLAN encapsulation mode

Additional fields are configured by clicking on the icon for Additional Fields. Enter the parameter settings for:


Click Save to continue

Step 2: Click on VLAN Ports to add VLAN to port

Then select the desired switch (Core-1) from the fabric and the dashboard updates automatically with VLAN PORT settings.

The dashboard displays a list of existing VLAN Ports and a drop-down selection list. The dashboard displays a list of existing VLAN Ports entries. Parameters include: ports and untagged-ports.


Select the VLAN using the up / down arrows in each of the columns as required.

Interactive Port Selector The Interactive Port Selector widget displays a grid accurately representing switch port availability and shows the number of physical ports on a given switch. Gray depicts down ports and white, up ports. Selected ports appear in blue.

Click the Edit icon to display and select a range of ports. Click the desired ports and click Apply to continue.


Follow the above steps by choosing selecting Site-B Core-3 switch and add the physical ports.

Click on Apply and Save to continue.

More details can find on www.pluribusnetworks.com/UNUM

Pluribus Networks, Inc. www.pluribusnetworks.com

Networking from Dell Technologieswww.DellTechnologies.com/Networking

Copyright © 2021 Pluribus Networks, Inc. All Rights Reserved. Netvisor is a registered trademark, and The Pluribus Networks logo, Pluribus Networks, Freedom, Adaptive Cloud Fabric, Insight Analytics, UNUM and VirtualWire are trademarks of Pluribus Networks, Inc. All other brands and product names are registered and unregistered trademarks of their respective owners. Pluribus Networks reserve the right to make changes to its technical information and specifications at any time, without notice. This informational document may describe features that may not be currently available or may be part of a future release or may require separate licensed so�ware to function as described. This document describes functionality that is dependent on the underlying open networking hardware device. Functionality will vary by physical device, and not all capabilities described herein will work on every device. Adaptive Cloud Fabric Overview. DE/PN-DG-DCI-062221

The following resources provide support and additional information: • The Dell Demo Center platform provides several SmartFabric OS10 virtual labs. These vLabs provide sample OS10 based configurations that help the user get familiar with Dell EMC SmartFabric OS10 CLI and features. • The Dell Technologies online Support page, which focuses on meeting customer needs with proven services and support. • Dell Community, which is an online technical community where IT professionals have access to numerous resources for Dell EMC so�ware, hardware, and services. • Dell Technologies online support Knowledge Base, which provides storage solutions technical documents and expertise that helps to ensure success on Dell EMC Storage platforms. • The Networking Solutions Info Hub, which provides access to networking guides. • Pluribus Networks technical documentation for Netvisor ONE, the Adaptive Cloud Fabric and UNUM Management Platform. • The Pluribus partner portal. • Additional Pluribus support inquiries can be sent to [email protected]

Support and references

de-pn-dg dell emc data center interconnect fabric for os10

Documents