39282588 nexus 7000 new innovations for data center interconnects lab

© 2010 Cisco Systems, Inc. All rights reserved 1 of 38

Nexus 7000 New Innovations for Data Center Interconnects LABDCT-2004

Technical Marketing

Data Center Business Unit

Version 2.1

Cisco Live 2010 – Networkers Technical Program


Nexus 7000

The Cisco Nexus 7000 Series is a modular data center class series of switching systems

designed for highly scalable end-to-end 10 Gigabit Ethernet networks. The Cisco Nexus 7000

Series is purpose built for the data center and has many unique features and capabilities

designed specifically for such mission critical place in the network.

Cisco NX-OS, a state-of-the-art operating system, powers the Cisco Nexus 7000 Platform.

Cisco NX-OS is built with modularity, resiliency, and serviceability at its foundation. Drawing

on its Cisco IOS and Cisco SAN-OS heritage, Cisco NX-OS helps ensure continuous

availability and sets the standard for mission-critical data center environments.

Lab Objectives

This instructor-led hands-on lab will introduce the participants to the OTV (Overlay Transport

Virtualization) solution for the Nexus 7000. This innovative feature set simplifies Datacenter

Interconnect designs, allowing multisite Data Center communication and transparent Layer 2



extension across multiple Data Center sites. OTV accomplishes this without the overhead

introduced by MPLS or VPLS. By the end of the laboratory session the participant should be

able to understand basic and advanced OTV functionality and configuration with the Nexus

7000.

General Disclaimer

The content of this Lab is based on a feature set not presently released by Cisco. Feature content, configuration commands and terminology are subjected to change at any point in time until official code availability. The Lab Hands-on session will be delivered using pre-release Early Field Trial code.



Lab Procedure

The Lab consists of 8 PODs. Each single POD represents a typical but simplified Nexus 7000

Data Center site where Nexus 7000 is used as an edge device attached to a layer 3 Core

cloud. The core consists of a pair of Catalyst 6500s. Catalyst 6500s are not the typical DC

core platform and are here only used for convenience to model a simplified L3 core network.

A Nexus 5000 with an attached ESX server represent the access layer.

The aggregation layer (on which all the configuration for this lab is performed) is built on

Nexus 7000 10-slot chassis with dual supervisors, one 48-port GE Copper card (model N7K-

M148GT-12) and one 32-port 10GE fiber card (model N7K-M132XP-12) each. Nexus 7000

devices run a PRE-Release Early Field Trial version of the NX-OS 5.0(2).

One student is assigned to each Pod. A student will be able to configure his own Nexus

7000 aggregation device.

During the Lab procedure the students will go through the following steps:

System Verification: Management VRF, Basic Connectivity, CLI Familiarization,

Base configuration: Spanning Tree, LACP and OSPF.

Configuring OTV to establish adjacencies with 2 Head-End remote sites.

Verifying OTV environment.

Testing the multisite connectivity and remote MAC learning.



Lab Topology and Access

In this multi-site Data-Center environment Each student will be able to configure a single

Nexus 7000 device in a pre-assigned site. The goal of the lab is to establish L2 connectivity

with remote end sites over a generic IP core leveraging the Nexus 7000 Overlay Transport

technology.

Figure 1 –Multi-Site Data-Center Topology (Overview of all the PODs)

Each site has independent connectivity through the IP Core infrastructure, as described in

Figure 2

.



Figure 2 - Single Site connecting to the Core and the Head-Ends (Single POD Detail)



Each POD has its own set of pre-assigned interfaces and IP addresses. The diagrams below

represent topologies for odd pods (Pod1, Pod3, Pod5, Pod7) and even pod (Pod2, Pod4,

Pod6, Pod8).

Figure 3 Topology for the odd Pods (1,3,5,7)

Site X Site Y

Site-X MAC: 0050.5622.2222 Site-X IP: 10.100.0.2

Site-Y MAC: 0050.5633.3333 Site-Y IP: 10.100.0.3

POD-SiteMAC: 0050.5611.1111

POD-Site IP: 10.100.0.1

2/9

2/2 2/1

1/1

Edge and Aggregation

Nexus 7000

Student Pod

VMware

ESX

1/13

Access

Nexus 5000

1/3

1/14

2/48 2/48 2/47

e1/1-8 e1/13-20

10.x.1.2

10.x.1.1 10.x.2.1

2/5 2/5

10.y.1.1 10.y.1.2

10.y.14.1

10.y.14.2 10.y.11.2

10.y.12.1

10.y.13.2

10.y.13.1

10.y.12.2

x is the POD number

y could be [12|34|56|78] Loopback 0 10.99.100.2/32

Loopback 0 10.99.100.1/32

Loopback 0 10.99.y.1/32

Loopback 0 10.99.x.1/32

Loopback 0 10.99.y.2/32

2/47

10.y.11.1

Generic Layer 3

Core

Remote End-Site

Nexus 7000

10.x.2.2



Figure 4 Topology for the even Pods (2,4,6,8)

2/25

2/4

1/2

Edge and Aggregation

Nexus 7000

Student Pod

VMware

ESX

1/25

Access

Nexus 5000

1/3

2/3

1/26

2/48 2/48 2/47

e1/1-8 e1/13-20

10.x.1.2 10.x.2.2

10.x.1.1 10.x.2.1

2/5 2/5

10.y.1.1 10.y.1.2

10.y.14.1

10.y.14.2 10.y.11.2

10.y.12.1

10.y.13.2

10.y.13.1

10.y.12.2

x is the POD number

y could be [12|34|56|78] Loopback 0 10.99.100.2/32

Loopback 0 10.99.100.1/32

Loopback 0 10.99.y.1/32

Loopback 0 10.99.x.1/32

Loopback 0 10.99.y.2/32

2/47

10.y.11.1

Generic Layer 3

Core

Remote End-Site

Nexus 7000

Site X Site Y Site-X MAC: 0050.5622.2222

Site-X IP: 10.100.0.2


POD-SiteMAC: 0050.5611.1111

POD-Site IP: 10.100.0.1



In the present LAB we leverage the Virtual Device Context feature to consolidate multiple

nodes and reduce the required equipment.

Based on your POD number, identify which of the topologies (odd or even) you will be

working with and which IP addresses and interfaces you will be using. All access to

your POD devices is via the ESX VMware server that is available via the Microsoft Remote

Desktop (Microsoft Remote Desktop client can be found under

start>accessories>communication). Credential to be used for the Remote desktop are defined

in Table 1.

PLEASE NOTE: Interfaces and IP addresses referenced throughout various steps of the guide will change based on your target topology (Odd or Even POD # Topology). CLI commands have x, y, z parameters indicating such variable configuration. Please refer to Figure 3 and Figure 4 to identify the right interfaces and IP addresses The Output of the show commands in this guide refers to POD 1 and so may slightly vary based on the POD # you are operating with.

Figure 5 - multi-VDC POD deployment

Pod 1 Pod 2

N7K-Edge



In order to connect to your Nexus device:

1. Open the Microsoft Remote Desktop Client on your workstation and point your

machine to the Pod‟s VM instance as shown in Table 1.

POD Information Remote Desktop VM IP address Login/Password

POD1 128.107.222.201 Student1/otv1-s1

POD2 128.107.222.202 Student1/otv2-s1

POD3 128.107.222.203 Student1/otv3-s1

POD4 128.107.222.204 Student1/otv4-s1

POD5 128.107.222.205 Student1/otv5-s1

POD6 128.107.222.206 Student1/otv6-s1

POD7 128.107.222.207 Student1/otv7-s1

POD8 128.107.222.208 Student1/otv8-s1

Table 1 - POD Access Details

2. For your convenience you will find puTTY connections for all relevant devices on the

Desktop. Double click on the connection of the Nexus 7000. Click YES to proceed

if you get an SSH warning message.



Figure 6 - Remote Shared Desktop Environment with puTTY consoles

Step 1 System Verification

PLEASE NOTE: This section is not required to complete the OTV configuration. You can

skip this step if you are already familiar with the Nexus 7000 hardware and software

infrastructure. In this case jump to Step 2 (CLI Familiarization paragraph).

During the entire duration of this lab we will be just logging into the management interface via

ssh. However it is good to keep in mind that the Nexus 7000 requires console access to

perform the initial configuration of the system. After performing the initial configuration, the

system can be completely managed from the management interface.

Let’s start by checking the system and its configuration.

N7K-1-pod1-S1# show module

Mod Ports Module-Type Model Status

--- ----- -------------------------------- ------------------ ------------

1 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 ok

2 32 10 Gbps Ethernet Module N7K-M132XP-12 ok

5 0 Supervisor module-1X N7K-SUP1 ha-standby

6 0 Supervisor module-1X N7K-SUP1 active *

Mod Sw Hw

--- -------------- ------

1 5.0(0.312) 1.0

2 5.0(0.312) 1.3



5 5.0(0.312) 1.1

6 5.0(0.312) 1.1

Mod MAC-Address(es) Serial-Num

--- -------------------------------------- ----------

1 00-1b-54-c2-f8-2c to 00-1b-54-c2-f8-60 JAF1223AGAK

2 00-22-55-77-f6-20 to 00-22-55-77-f6-44 JAB123500AZ

5 00-22-55-77-f8-f8 to 00-22-55-77-f9-00 JAB123501Z7

6 00-22-55-77-f8-a8 to 00-22-55-77-f8-b0 JAB123501ZR

Mod Online Diag Status

--- ------------------

1 Pass

2 Pass

5 Pass

6 Pass

Xbar Ports Module-Type Model Status

--- ----- -------------------------------- ------------------ ------------

1 0 Fabric Module 1 N7K-C7010-FAB-1 ok



Xbar Sw Hw

--- -------------- ------

1 NA 1.0

2 NA 1.0

3 NA 1.0

Xbar MAC-Address(es) Serial-Num

--- -------------------------------------- ----------

1 NA JAB1229003X

2 NA JAB1234003C

3 NA JAB1234003U

* this terminal session

Let’s check now the software the system is running.

N7K-1-pod1-S1# show version

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.

The copyrights to certain works contained in this software are

owned by other third parties and used and distributed under

license. Certain components of this software are licensed under

the GNU General Public License (GPL) version 2.0 or the GNU

Lesser General Public License (LGPL) Version 2.1. A copy of each

such license is available at

http://www.opensource.org/licenses/gpl-2.0.php and

http://www.opensource.org/licenses/lgpl-2.1.php

Software

BIOS: version 3.17.0

loader: version N/A



kickstart: version 5.0(2)

system: version 5.0(2)

BIOS compile time: 03/23/08

kickstart image file is: bootflash:/n7000-s1-kickstart.5.0.2.S10.gbin

kickstart compile time: 12/25/2020 12:00:00 [02/25/2010 03:44:41]

system image file is: bootflash:/n7000-s1-dk9.5.0.2.S10.gbin

system compile time: 2/7/2010 3:00:00 [02/25/2010 04:34:08]

Hardware

cisco Nexus7000 C7010 (10 Slot) Chassis ("Supervisor module-1X")

Intel(R) Xeon(R) CPU with 4129620 kB of memory.

Processor Board ID JAB123501Z7

Device name: N7K-1

bootflash: 2000880 kB

slot0: 0 kB (expansion flash)

Kernel uptime is 0 day(s), 23 hour(s), 39 minute(s), 59 second(s)

Last reset at 185087 usecs after Tue Dec 2 04:59:22 2008

Reason: Reset Requested by CLI command reload

System version: 4.1(1.66)

Service:

plugin

Core Plugin, Ethernet Plugin

N7K-1-pod1-S1#

1. NX-OS is composed by two images: a kickstart image that contains the Linux Kernel and a system image that contains the NX-OS software components. They both show up in the configuration.

2. In a future release we will be adding other plug-ins, like the “Storage” plug-in for FCoE.

Let’s now take a look at the running configuration.

N7K-1-pod1-S1# show running-config

version 5.0(2) <omitted config> vrf context management vlan 1 <omitted interface config> interface Ethernet2/1 interface Ethernet2/2 <omitted interface config>

interface Ethernet2/16

Active Plug-in

These are the interfaces available to your Pod (Virtual Device Context)

Storage Devices

CPU

Images Location

NX-OS Version



interface mgmt0 ip address 192.168.1.17/16

3. This is the configuration of the first Pod. As explained earlier each Pod runs within a Virtual Device Context (VDC). By using the VDC feature, we can segment the physical Nexus 7000 into multiple logical switches, each of which runs in a separate memory space and only has visibility into the hardware resources that it owns, providing total isolation between the VDCs.

One of the features of “show running-config” in NX-OS consists in the ability to not only

look at the running-config but to also at the default values, which do not appear in the base

config. The keyword to use is “all”.

N7K-1-pod1-S1# show running-config all | section mgmt0

interface mgmt0

no description

speed auto

duplex auto

snmp trap link-status

no shutdown

cdp enable

ip redirects

ip address 192.168.1.17/16

ip port-unreachable

ipv6 redirects

ip arp gratuitous update

ip arp gratuitous request

Management VRF and Basic Connectivity

The management interface is always part of the management VRF. The management

interface “mgmt0” is the only interface allowed to be part of this VRF.

The Management VRF provides total isolation of management traffic from the rest of the

traffic flowing through the box.

In this step we will:

- Verify that only the mgmt0 interface is part of the management VRF

Management Interface Config



- Verify that no other interface can be part of the management VRF

- Verify that the default gateway is reachable only using the management VRF

N7K-1-pod1-S1# show vrf

VRF-Name VRF-ID State Reason

default 1 Up --

management 2 Up --

N7K-1-pod1-S1# show vrf interface

Interface VRF-Name VRF-ID

Ethernet1/1 default 1





<omitted output>

mgmt0 management 2

N7K-1-pod1-S1# show vrf management interface

Interface VRF-Name VRF-ID

mgmt0 management 2

4. The management VRF interface is part of the default configuration and the management interface “mgmt0” is the only interface that can be made member of this VRF. Let’s verify it.

N7K-1-pod1-S1# conf t

N7K-1-pod1-S1(config)# interface ethernet 2/y

N7K-1-pod1-S1(config-if)# vrf member management

% VRF management is reserved only for mgmt0

N7K-1-pod1-S1(config-if)# show int mgmt0

mgmt0 is up

DCE oper mode is auto

Hardware: GigabitEthernet, address: 0022.5577.f8a8 (bia 0022.5577.f8a8)

Internet Address is 192.168.1.17/16

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA

full-duplex, 1000 Mb/s

Auto-Negotiation is turned on

EtherType is 0x0000

1 minute input rate 72 bits/sec, 0 packets/sec

1 minute output rate 24 bits/sec, 0 packets/sec

Rx

353 input packets 0 unicast packets 265 multicast packets

88 broadcast packets 51632 bytes

Tx

92 output packets 0 unicast packets 91 multicast packets

FastEthernet? GigabitEthernet?... no,

just “ethernet” interfaces

Use Ethernet 2/10 for the Odd POD #s (i.e. POD #1, 3, 5, 7) and Ethernet 2/26 for Even POD #s (i.e. POD 2, 4, 6, 8 )



1 broadcast packets 21554 bytes

Try to reach the out-of-band management network’s default gateway with a ping.

N7K-1-pod1-S1(config-if)# ping 192.168.0.1

PING 192.168.0.1 (192.168.0.1): 56 data bytes ping: sendto 192.168.0.1 64 chars, No route to host Request 0 timed out ping: sendto 192.168.0.1 64 chars, No route to host

Request 1 timed out ping: sendto 192.168.0.1 64 chars, No route to host Request 2 timed out ping: sendto 192.168.0.1 64 chars, No route to host Request 3 timed out ping: sendto 192.168.0.1 64 chars, No route to host Request 4 timed out --- 192.168.0.1 ping statistics --- 5 packets transmitted, 0 packets received, 100.00% packet loss N7K-1-pod1-S1(config-if)#

5. The ping fails because we are trying to reach a system on the out-of-band management network without specifying the correct VRF.

N7K-1-pod1-S1# ping 192.168.0.1 vrf management

PING 192.168.0.1 (192.168.0.1): 56 data bytes

Request 0 timed out

64 bytes from 192.168.0.1: icmp_seq=1 ttl=63 time=0.593 ms




--- 192.168.0.1 ping statistics ---

5 packets transmitted, 4 packets received, 20.00% packet loss

round-trip min/avg/max = 0.585/0.674/1.005 ms

Linux-like output



Step 2 CLI Familiarization NX-OS CLI is very IOS-like as you will notice when configuring the system. Also NX-OS

implements a hierarchy independent CLI, so that any command can be issued from any CLI

context.

PLEASE NOTE: This section is not required to complete the OTV configuration so you can

skip this step if you are already familiar with the Nexus 7000 CLI capabilities. In this case

jump to Step 3 (Spanning Tree configuration paragraph).

In this step we will:

- Verify the CLI hierarchy independence by issuing a ping from different CLI contexts

- Verify the CLI piping functionality


N7K-1-pod1-S1(config)# ping ?

*** No matches in current mode, matching in (exec) mode ***

<CR>

A.B.C.D or Hostname IP address of remote system

WORD Enter Hostname

multicast Multicast ping

N7K-1-pod1-S1(config)# ping 192.168.0.1 vrf management

PING 192.168.0.1 (192.168.0.1): 56 data bytes 64 bytes from 192.168.0.1: icmp_seq=0 ttl=63 time=4.257 ms 64 bytes from 192.168.0.1: icmp_seq=1 ttl=63 time=0.714 ms 64 bytes from 192.168.0.1: icmp_seq=2 ttl=63 time=0.562 ms 64 bytes from 192.168.0.1: icmp_seq=3 ttl=63 time=0.581 ms 64 bytes from 192.168.0.1: icmp_seq=4 ttl=63 time=0.568 ms --- 192.168.0.1 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.562/1.336/4.257 ms

N7K-1-pod1-S1(config)# int e2/y

N7K-1-pod1-S1(config-if)# ping ?

*** No matches in current mode, matching in (exec) mode ***

<CR>

A.B.C.D or Hostname IP address of remote system

WORD Enter Hostname

multicast Multicast ping

N7K-1-pod1-S1(config-if)# ping 192.168.0.1 vrf management

PING 192.168.0.1 (192.168.0.1): 56 data bytes






--- 192.168.0.1 ping statistics ---

Hierarchy

Independent CLI

CLI Hierarchy

Independent

Use Ethernet 2/10 for the Odd POD #s (i.e. POD #1, 3, 5, 7) and Ethernet 2/26

for Even POD #s (i.e. POD 2, 4, 6, 8 )



5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 0.586/1.251/3.768 ms

6. You can use the up-arrow and get the command history from the exec mode

7. Any command can be issued from anywhere within the configuration

Multiple piping options are available lots of them derived from the Linux world.

N7K-1-pod1-S1# show running-config | ?

cut Print selected parts of lines.

diff Show difference between current and previous invocation

(creates temp files: remove them with 'diff-clean' command

and dont use it on commands with big outputs, like 'show

tech'!)

egrep Egrep - print lines matching a pattern

grep Grep - print lines matching a pattern

head Display first lines

human Output in human format (if permanently set to xml, else it

will turn on xml for next command)

last Display last lines

less Filter for paging

no-more Turn-off pagination for command output

perl Use perl script to filter output

section Show lines that include the pattern as well as the

subsequent lines that are more indented than matching line

sed Stream Editor

sort Stream Sorter

sscp Stream SCP (secure copy)

tr Translate, squeeze, and/or delete characters

uniq Discard all but one of successive identical lines

vsh The shell that understands cli command

wc Count words, lines, characters

xml Output in xml format (according to .xsd definitions)

begin Begin with the line that matches

count Count number of lines

end End with the line that matches

exclude Exclude lines that match

include Include lines that match

N7K-1-pod1-S1# sh running-config | grep ?

WORD Search for the expression

count Print a total count of matching lines only

ignore-case Ignore case difference when comparing strings

invert-match Print only lines that contain no matches for <expr>

line-exp Print only lines where the match is a whole line

line-number Print each match preceded by its line number

next Print <num> lines of context after every matching line

prev Print <num> lines of context before every matching line

word-exp Print only lines where the match is a complete word

Enhanced CLI Piping



The following command will grab the instance of a line with “mgmt0” and print the following 3

lines after that match.

N7K-1-pod1-S1# sh running-config | grep next 3 mgmt0

interface mgmt0

no snmp trap link-status

ip address 192.168.1.17/16


N7K-1-pod1-S1(config)# int mgmt 0

N7K-1-pod1-S1(config-if)# [TAB]

cdp end ipv6 push this

control exit no shutdown vrf

description ip pop snmp where

8. The [TAB] completes the CLI command and shows the available keywords.

If you want to know the CLI context you are in use the “where” command.

N7K-1-pod1-S1(config-if)# where

conf; interface mgmt0 admin@N7K-1-pod1-S1%default

Step 3 Spanning Tree

It is time to bring up the interfaces and configure the Spanning Tree Protocol. Rapid

Spanning Tree Protocol (RSTP) is standardized in 802.1d (now, IEEE 802.1D-2004).

Cisco's implementation of RSTP in both NX-OS and IOS provides a separate spanning tree

instance for each active VLAN, which permits greater flexibility of Layer 2 topologies in

conjunction with IEEE 802.1Q trunking. This implementation is also referred to as Rapid Per-

VLAN Spanning Tree (Rapid-PVST). Rapid-PVST is the default spanning tree mode for

NX-OS, so it does not need to be explicitly enabled.

Best practices dictate deterministic placement of the spanning tree root in the network.

Particularly a network administrator should ensure that a root switch does not inadvertently

end up on a small switch in the access layer creating a sub-optimal topology more prone to

failures.

Let’s first configure the VLANs in each data-center site

9. Each site will have 2 different VLANs, one local to the site and one to be extended on the overlay to the remote data-center sites. VLANs are x0 and x00 where x identifies the POD# (i.e. 10 and 100 for POD 1, 20 and 200 for POD 2 and so on).


Enter configuration commands, one per line. End with CNTL/Z.



N7K-1-pod1-S1(config)# vlan x0

N7K-1-pod1-S1(config-vlan)# no shut

N7K-1-pod1-S1(config-vlan)# vlan x00

N7K-1-pod1-S1(config-vlan)# no shut

N7K-1-pod1-S1(config)# spanning-tree vlan x0, x00 priority 4096

Now let’s bring up the interfaces facing the Access Layer (i.e. facing the Nexus 5000)

N7K-1-pod1-S1(config-if-range)# int ethernet 2/y

N7K-1-pod1-S1(config-if-range)# switchport

N7K-1-pod1-S1(config-if-range)# switchport mode trunk

N7K-1-pod1-S1(config-if-range)# switchport trunk allowed vlan x0, x00

N7K-1-pod1-S1(config-if-range)# no shutdown

Now let’s create these VLANs on the attached Nexus 5000 access device. Double-click on

the puTTY console shortcut for the Nexus 5000 on your shared desktop. Check the Network

Diagram and based on your POD find out which VLAN you need to create and add those on

the interfaces connecting to your Nexus 7000 (i.e. 10 and 100 for POD 1, 20 and 200 for

POD 2 and so on).

N5K-1# conf t N5K-1(config)# vlan x0, x00

N5K-1(config-vlan)# no shut

Now configure interfaces toward the Nexus 7000

N5K-1(config)# int ethernet 1/z

N5K-1(config-if)# switchport

N5K-1(config-if)# switchport mode trunk

N5K-1(config-if)# switchport trunk allowed vlan x0, x00

N5K-1(config-if)# no shutdown

And enable VLAN to be extended on the interface toward the End Host/Server. When you

configure the port as STP type edge (i.e. equivalent of portfast) an appropriate informational

warning message is printed.

Make Sure you extend the right VLAN to the HOST (VLAN 100 for Pod 1, VLAN 200 for

POD2, VLAN 300 for POD3 and so on)

N5K-1(config)# int ethernet 1/3

N5K-1(config-if)# switchport

N5K-1(config-if)# switchport mode access

N5K-1(config-if)# switchport access vlan x00

N5K-1(config-if)# spanning-tree port type edge

N5K-1(config-if)# no shut

Warning: Edge port type (portfast) should only be enabled on ports connected

to a single host. Connecting hubs, concentrators, switches, bridges, etc...

to this interface when edge port type (portfast) is enabled, can cause

temporary bridging loops. Use with CAUTION

Interface numbers will change for odd and even POD #s. Check your Network topology to target the right interfaces

Interface numbers will change for odd and even POD #s.

Check your Network topology to target the

right interfaces

VLAN #s will change based on the POD# you

are operating with.



Check the spanning-tree from both the Nexus 7000 …

N7K-1-pod1-S1# show spanning-tree vlan x00

VLAN0100

Spanning tree enabled protocol rstp

Root ID Priority 4196

Address 0022.5579.c442

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 4196 (priority 4096 sys-id-ext 100)

Address 0022.5579.c442


Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------

Eth2/9 Desg FWD 2 128.265 P2p

N7K-1-pod1-S1(config-if-range)# show spanning-tree vlan x0

VLAN0010



Address 0022.5579.c442

This bridge is the root



Address 0022.5579.c442



---------------- ---- --- --------- -------- ------------------------------

Eth2/9 Desg FWD 2 128.265 P2p

… and the and the Nexus 5000

N5K-1# show spanning-tree vlan x00

VLAN0100



Address 0022.5579.c442

Cost 2

Port 129 (Ethernet1/1)



Address 000d.eca4.04fc



---------------- ---- --- --------- -------- -------------------------------

Eth1/1 Root FWD 2 128.129 P2p

Eth1/3 Desg FWD 2 128.131 Edge P2p



Step 4 Configuring OTV to connect edge devices to remote end-sites

OTV provides Layer 2 connectivity between remote network sites. OTV uses MAC address-

based routing and IP-encapsulated forwarding across a Layer 3 network to provide support

for applications that require Layer 2 adjacency, such as clusters and vmotion. You deploy

OTV on the edge devices in each site. OTV requires no other changes to the sites or the core

network. OTV avoids the addition of multiple routing tables to every device in the network that

other methods, such as Multiprotocol Label Switching (MPLS), require.

Figure 7 - OTV Packet Flow

The following terminology is used for OTV throughout this document:

Site: A Layer 2 network that may be single-homed or multihomed to the core network and the OTV overlay network. Layer 2 connectivity between sites is provided by edge devices that operate in an overlay network. Layer 2 sites are physically separated from each other by the core IP network.

Core Network: The customer backbone network that connects Layer 2 sites over IP. This network can be customer managed, provided by a service provider, or a mix of both. OTV is transparent to the core network because OTV flows are treated as regular IP flows. Edge Device: A Layer 2 switch that performs OTV functions. An edge device performs typical Layer 2 learning and forwarding on the site-facing interfaces (internal interfaces) and performs IP-based virtualization on the core-facing interfaces. The edge device can be collocated in a device that performs Layer 3 routing on other



ports. OTV functionality only occurs in an edge device.

Internal Interface: The Layer 2 interface on the edge device that connects to site-based switches or site-based routers. The internal interface is a Layer 2 interface regardless of whether the internal interface connects to a switch or a router.

Join Interface: The interface facing the core network. The name implies that the edge device joins an overlay network through this interface. The IP address of this interface is used to advertise reachability of a MAC address present in this site.

Figure 8 - OTV Terminology (1 of 2)

MAC Routing: MAC routing associates the destination MAC address of the Layer 2 traffic with an edge device IP address. The MAC to IP association is advertised to the edge devices through an overlay routing protocol. In MAC routing, MAC addresses are reachable through an IP next hop. Layer 2 traffic destined to a MAC address will be encapsulated in an IP packet based on the MAC to IP mapping in the MAC routing table.

Overlay Interface: A logical multi-access multicast-capable interface. The overlay interface encapsulates Layer 2 frames in IP unicast or multicast headers. The overlay interface is connected to the core via one or more physical interfaces. You assign IP addresses from the core network address space to the physical interfaces that are associated with the overlay interface.

Overlay Network: A logical network that interconnects remote sites for MAC routing of Layer 2 traffic. The overlay network uses either multicast routing in the core network or an overlay server to build an OTV routing information base (ORIB). The ORIB associates destination MAC addresses with remote edge device IP addresses.

Authoritative Edge Device: An edge device that forwards Layer 2 frames into and out of a site over the overlay interface. For the first release of OTV, there is only one



authoritative edge device for all MAC unicast and multicast addresses per VLAN. Each VLAN can be assigned to a different authoritative edge device.

Figure 9- OTV Terminology (2 of 2)

In this section you will:

Select the Join interface and establish OSPF connectivity with the Core.

Enable OTV

Configure the Overlay interface

Join the Data-Center site to the Core via the join interface. Extend a VLAN across the overlay to connect the local site with the remote sites.

10. In this first step we will identify the interconnection to the core and configure OSPF for L3 connectivity. This interface will be assigned as the Join interface of the OTV Edge device

Let’s now select the join interface on the Nexus 7000 edge device. Look at the topology diagram and based on your POD topology pick one of the 2 uplink interfaces connected to the Core. First, un-shut both connections to the core (i.e. selecting a range of interfaces):

N7K-1-pod1-S1(config)# int e 1/y-z

N7K-1-pod1-S1(config-if-range)# no shut

N7K-1-pod1-S1# sh cdp neighbors

C6K-2 Eth1/13 139 R S I WS-C6503 Gig2/1

C6K-1 Eth1/14 133 R S I WS-C6503 Gig2/2

N5K-1(FLC12220548) Eth2/9 161 S I s N5K-C5020P-BF Eth1/1

Interface numbers will change for odd and even POD #s. Check your Network

topology to target the right interfaces and determine „y‟ and „z‟ values.



11. NX-OS is a fully modular operating system; most software modules don’t run unless the correspondent service is enabled. We refer to these features that need to be specifically enabled as “conditional services”. Once the service is enabled, the CLI

becomes visible and the feature can be used and configured.

Now let’s configure Layer 3 and OSPF Routing

N7K-1-pod1-S1(config)# conf t

N7K-1-pod1-S1(config)# feature ospf

N7K-1-pod1-S1(config)# router ospf 1

N7K-1-pod1-S1(config)# interface loopback0

N7K-1-pod1-S1(config-if)# ip address 10.99.x.1/32

N7K-1-pod1-S1(config-if)# ip router ospf 1 area 0.0.0.0

N7K-1-pod1-S1(config)# interface Ethernet1/<first_uplink>

N7K-1-pod1-S1(config-if)# ip address 10.x.y.1/30

N7K-1-pod1-S1(config-if)# ip ospf network point-to-point


N7K-1-pod1-S1(config-if)# ip igmp version 3

N7K-1-pod1-S1(config-if)# no shutdown

N7K-1-pod1-S1(config)# interface Ethernet1/<second_uplink>

N7K-1-pod1-S1(config-if)# ip address 10.x.z.1/30

N7K-1-pod1-S1(config-if)# ip ospf network point-to-point


N7K-1-pod1-S1(config-if)# ip igmp version 3

N7K-1-pod1-S1(config-if)# no shutdown

Let’s check OSPF neighbors to see if the OSPF connectivity was successfully established. N7K-1-pod1-S1# sh ip ospf neighbors

OSPF Process ID 1 VRF default

Total number of neighbors: 2

Neighbor ID Pri State Up Time Address Interface

10.99.12.2 1 FULL/ - 00:18:14 10.1.2.2 Eth1/13

10.99.12.1 1 FULL/ - 00:18:13 10.1.1.2 Eth1/14

12. In this second step we configure the OTV Overlay interface and join the Overlay transport through the IP core.

Let’s first enable the OTV feature set. Note: it may take few seconds to enable the required set of protocols, but this is a one-time operation. N7K-1-pod1-S1(config)# conf t

N7K-1-pod1-S1(config)# feature otv

Enable OSPF as conditional service

Interface numbers may change for odd and even

POD #s. Check your Network topology to target

right interfaces (i.e. Ethernet 1/13-14 or Ethernet

1/25-26)

Check the Network diagram to assign the right IP addresses to your POD. „X‟ indicated here matches the

POD# (i.e. „1‟ for POD1, „2‟ for POD2 and so on). „y‟ and „z‟ will differ based on the interface IDs (check

the right topology diagram)



We will use the pre-configured VLANs <X>00 as VLAN to extend and VLAN <X>0 as site VLAN. X is again the POD# (VLAN 100 and VLAN 10 for POD 1, VLAN 200 and VLAN 20 for POD 2 and so on). The site VLAN is the one used to communicate with other edge devices in the local site. In this case it will be not used as we only have a single edge device in the site, however for completeness we will configure it. N7K-1-pod1-S1(config)# conf t

N7K-1-pod1-S1(config)# otv site-vlan x0

Now let’s specify for POD X the Overlay configuration, X is the POD# (Overlay 1 for POD 1, Overlay 2 for POD 2 and so on).

N7K-1-pod1-S1(config)# interface Overlay <X>

N7K-1-pod1-S1(config-if-overlay)# otv control-group 239.<X>.1.1

N7K-1-pod1-S1(config-if-overlay)# otv data-group 239.<X>.2.0/28

Where X is again the POD# (1 for POD 1, 2 for POD 2 and so on). The group address is used for control plane related operations. Each edge device joins the group and sends control/protocol related packets to this group. This is used for discovery of other edge-devices. The data-group-range specifies a multicast group range that is used for multi-destination traffic (doesn't hit CPU of remote edge devices).

Now join the site you are working on with the next hop core device selecting the join interface. This command assigns an L3 interface as the core-facing interface for OTV. This interface is used for overlay operations such as discovering remote edge-devices, providing the source address for OTV encapsulated packets and the destination address for unicast traffic sent by remote edge-devices. Each POD has two ECMP interface connections to the core and one of these should be used as join interface. After you enter a command an informational message reminds you that IGMPv3 is required to be configured on the join interface. Message can be just ignored if IGMPv3 was already configured as instructed earlier in the guide.

N7K-1-pod1-S1(config-if-overlay)# otv join-interface Ethernet1/y

OTV needs join interfaces to be configured for IGMP version 3

Last let’s pick a VLAN to be extended. Multiple VLANs and a VLAN range could be extended however just for a practical demonstration we will extend a single VLAN across the core to reach the two remote Head-End Data-Centers.

N7K-1-pod1-S1(config-if-overlay)# otv extend-vlan ?

, Multi range separator

- Range separator

<1-3967,4048-4093> VLAN ID 1-4094 or range(s): 1-5, 10 or 2-5,7-19

VLANs will change based on the POD #s.

Join Interface (uplink interface to the Core) may change for odd and even POD #s. Check your Network topology to target right

interfaces (i.e. Ethernet 1/13-14 or Ethernet 1/25-26)



N7K-1-pod1-S1(config-if-overlay)# otv extend-vlan x00

N7K-1-pod1-S1(config-if-overlay)# no shutdown

Now let’s check the OTV configuration just completed:

N7K-1-pod1-S1# show running-config otv

!Command: show running-config otv

!Time: Tue Jan 19 14:34:06 2010

version 5.0(2)

feature otv

otv site-vlan 10

interface Overlay1

otv join-interface Ethernet1/13

otv control-group 239.1.1.1

otv data-group 239.1.2.0/28

otv extend-vlan 100

You have now completed OTV configuration in your POD !!



Step 5 OTV verification and Monitoring Commands

In this step we will monitor and troubleshoot the OTV configuration. First of all let’s display the OTV overlay status and parameters for your POD (local site): N7K-1-pod1-S1# show otv overlay <Pod_Number>

OTV Overlay Information

Overlay Interface Overlay1

VPN Name : Overlay1

VPN ID : 192

State : UP

IPv4 multicast group : Overlay1-239.1.1.1

IPv6 multicast group : [None]

Mcast data group range(s): 239.1.2.0/28

External interface(s) : Ethernet1/13

External IPv4 address : 10.1.2.1

External IPv6 address : 0::

Encapsulation format : GRE/IPv4

Site-vlan : 10

Capability : Multicast-Reachable

Is Adjacency Server : NO

Adj Server Configured : NO

Prim/Sec Adj Svr(s) : [None] / [None]

Also let’s verify what VLANs are being extended:

N7K-1-pod1-S1# show otv vlan

OTV VLAN Configuration Information

VLAN-ID VlanState Switchport/ External Overlay

Forward Count Interface Group

100 UP 1/1 Ethernet1/13 Overlay1-239.1.1.1

Now let‟s check how many OTV edge devices are present on the site. Because this is a single-homed site, one node will be listed through this command. The „*‟ symbol next to the MAC address indicates the local node.

N7K-1-pod1-S1# show otv site all

OTV Overlay Information

Site-VLAN : 10

Site Adjacency database

Overlay: Overlay1-239.1.1.1, Adjacencies: 1

System-ID Priority Ordinal

* 0022.5579.c442 0 0

The authoritative device is the OTV node elected to forward traffic to/from the L3 core. Only one authoritative device will be elected in a site. The show command below returns a non empty output only on the authoritative OTV data-center edge node.

N7K-1-pod1-S1# show otv vlan authoritative

OTV VLAN Configuration Information

VLAN-ID VlanState Switchport/ External Overlay

Forward Count Interface Group

100 UP 1/1 Ethernet1/13 Overlay1-239.1.1.1



The MAC address table will report mac addresses of end-hosts and devices learnt on the VLAN. If no traffic was ever sent across the overlay only the local router mac will be populated in the table. N7K-1-pod1-S1# show mac address-table

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports

---------+-----------------+--------+---------+------+----+-----------------

G - 0022.5579.c442 static - F F sup-eth1(R)

The MAC address in the table is actually the local router MAC, let’s verify

this using a physical routed interface already in use (i.e. an uplink

interface to the core)

N7K-1-pod1-S1# show interface ethernet 1/y mac-address

----------------------------------------------------------------------------

Interface Mac-Address Burn-in Mac-Address

----------------------------------------------------------------------------

Ethernet1/13 0022.5579.c442 001b.54c2.f838

In OTV we also cache ARP resolution for mac addresses not local to the site and learnt via the overlay. If no traffic was ever sent across the overlay no ARP would have been resolved, and so no entries are cached by the OTV process. N7K-1-pod1-S1# show otv arp-nd-cache

OTV ARP/ND L3->L2 Address Mapping Cache

Let’s now verify the neighbor routers visible via the OTV overlay. We should see two adjacencies here representing the two Nexus 7000 edge devices on the remote sites.

N7K-1-pod1-S1# show otv adjacency

Overlay Adjacency database

Overlay-Interface Overlay1 :

System-ID Dest Addr Adj-State TM_State Up Time Adj-State

0022.5579.1dc2 10.12.11.2 default default 01:57:41 UP

0022.5579.1dc3 10.12.12.2 default default 01:57:46 UP

The IS-IS protocol used underneath by OTV to advertise MAC addresses over the IP cloud allows to resolve the hostname and gives us a better indication of the neighbor devices:

N7K-1-pod1-S1# show otv isis adjacency

OTV-IS-IS process: default VPN: Overlay1

OTV-IS-IS adjacency database:

System ID SNPA Level State Hold Time Interface

Site-X 0022.5579.1dc2 1 UP 00:00:25 Overlay1

Site-Y 0022.5579.1dc3 1 UP 00:00:23 Overlay1

We can optionally check the OTV overlay information on one of the remote sites. Let’s use the puTTY shortcuts to reach the remote site 7k on Site-X or Site-Y via SSH (see Figure 10).

Interface numbers will change for odd and even

POD #s. Check your Network topology to target

the right interfaces



You will see multiple Overlays and sites depending on how many other students successfully completed this lab in other PODs. Site-X# show otv isis adjacency





N7K-1-pod1-S1 0022.5579.c442 1 UP 00:00:07 Overlay1









N7K-4-pod3-S2 0022.5579.be42 1 UP 00:00:08 Overlay3





N7K-4-pod4-S2 0022.5579.be43 1 UP 00:00:09 Overlay4




Figure 10 - SSH puTTY shortcuts to 7K on site-X or site-Y











N7K-7-pod7-S1 001b.54c2.b1c2 1 UP 00:00:33 Overlay7






Because we are only interested to our Overlay, we can also be more specific.

Site-X# show otv isis adjacency <Overlay_id>


OTV-IS-IS adjacency database for Overlay1:



N7K-1-pod1-S1 0022.5579.c442 1 UP 00:00:07 Overlay1



Step 6 Testing Multisite connectivity and mac learning

Let’s now verify connectivity between a Local site (student POD) and the remote sites (Site-X and Site-Y Head-Ends). Every site has a server (actually a VM deployed on an ESX server) which can be used to verify connectivity across the OTV cloud.

Figure 11 - POD and Remote Site Address Connectivity

The local VM is the device which hosts the Remote Desktop session that we are currently using to configure the devices. From the current VM let’s open a windows command prompt and from there try to ping the remote IP addresses 10.100.0.2 (Remote Site-X) and 10.100.0.3 (Remote Site-Y) as illustrated at Figure 15 and Figure 14 to verify connectivity toward the remote end sites. Note: the first ping of a new flow maybe lost till learning of the remote MAC happens. Subsequent pings will always be successful.

Site-X MAC: 0050.5622.2222 Site-X IP: 10.100.0.2


POD-Site-MAC: 0050.5611.1111 POD-Site-IP: 10.100.0.1

VM where each student operates the Remote Desktop Connection



Figure 12- Connecting to the server on the Remote Site

.

Figure 13 - Logging on the Remote Site



Figure 15 - Windows Command prompt on the local POD

Figure 14 - Connectivity Tests to remote Site-X (10.100.0.2 / 0050.5622.2222)



Last, check on the local Nexus 7000 that addresses of the remote VM servers were learned on the local site and that ARP Table entries, mapping remote IPs and MACs, were cached successfully. N7K-1-pod1-S1# show mac address-table

Legend:




---------+-----------------+--------+---------+------+----+----------------

G - 0022.5579.d2c2 static - F F sup-eth1(R)

* 100 0050.5611.1111 dynamic 0 F F Eth2/9

O 100 0050.5622.2222 dynamic 0 F F Overlay1

N7K-1-pod1-S1# show otv arp-nd-cache



VLAN/MAC Address Uptime Layer-3 Address Exp Time Left

0100-0050.5622.2222 00:00:35 10.100.0.2 00:19:24

Optionally repeat the connectivity test for Remote Site Y

Figure 16 - Connectivity Tests to remote Site-X (10.100.0.3 / 0050.5633.3333)



N7K-1-pod1-S1# show mac address-table

Legend:




---------+-----------------+--------+---------+------+----+----------------

G - 0022.5579.d2c2 static - F F sup-eth1(R)

* 100 0050.5611.1111 dynamic 240 F F Eth2/9



N7K-1-pod1-S1# show otv arp-nd-cache



VLAN/MAC Address Uptime Layer-3 Address Exp Time Left

0100-0050.5622.2222 00:00:36 10.100.0.2 00:19:23

0100-0050.5633.3333 00:19:30 10.100.0.3 00:00:29

You can optionally access the remote servers by simply clicking on the remote Server Site-X and Site-Y puTTY icons (see Figure 12 and Figure 13) to gain console access to both the end stations and perform bidirectional connectivity tests through the Network Overlay. When Logging in into the remote server VMs, use the same login and password used to access you POD and documented in Table 1 - POD Access Details

Congratulations!!! The lab is now complete!

Please LOG OFF from the Windows Machines (Click “Start” on the

bottom left corner and “Log Off” right above), do NOT just close the

Windows Remote Desktop window.



Recommended Reading

Introductory overview on OTV:

http://www.cisco.com/en/US/prod/switches/ps9441/nexus7000_promo.html

Cisco Nexus 7000 Series Switches:

www.cisco.com/en/US/products/ps9402/index.html

Cisco NX-OS Feature Navigator:

www.cisco.com/go/nxosnav

Cisco NX-OS Home Page:

www.cisco.com/go/nxos

Complete Your Online Session Evaluation

Cisco values your input. Give us your feedback! We read and carefully consider your scores and comments, and incorporate them into the content program year after year

Go to the Internet stations located throughout the Convention Center to complete your session evaluations

Thank you!




39282588 nexus 7000 new innovations for data center interconnects lab

Documents