ipexpert’s lab preparation detailed solution...

IPexpert’s Lab Preparation Detailed Solution Guide for the Cisco® CCIE™ Data Center v1.0 Lab Exam

Volume 1

Authored by: Rick Mur - CCIE3 #21946 (R&S / SP / Storage), JNCIE-SP #851

CCIE Data Center Lab Preparation Detailed Solution Guide

Copyright © by IPexpert. All rights reserved. 1

IPexpert’s Lab Preparation Detailed Solution Guide for

Cisco’s CCIE Data Center Lab Before We Begin This product is part of the IPexpert suite of materials that provide CCIE candidates and network engineers with a comprehensive training program. For information about the full solution, contact an IPexpert Training Advisor today. Telephone: +1.810.326.1444 Email: [email protected] Congratulations! You now possess one of the ULTIMATE CCIETM Lab preparation and network operation resources available today! This resource was produced by senior engineers, technical instructors, and author boasting decades of internetworling experience. Although there is no way to 100% guarantee success rate on the CCIE™ Data Center Lab exam, we feel VERY confident that your chances of passing the Lab will improve dramatically after completing this industry-recognized Workbook! Technical Support from IPexpert, and your CCIE community!



IPexpert is proud to lead the industry with multiple support options at your disposal free of charge. Our online communities have attracted a membership of over 20,000 of your peers from around the world! At blog.ipexpert.com, you can keep up to date with everything IPexpert does and read the latest in technical articles from world-renowned IPexpert instructors. At OnlineStudyList.com, you may subscribe to multiple “SPAM-free,” moderated CCIE-focused email lists.

Feedback Do you have a suggestion or other feedback regarding this book or other IPexpert products? At IPexpert, we look to you – our valued clients – for the real world, frontline evaluation that we believe is necessary so that we may always improve. Please send an email with your thoughts to [email protected] or call 1.866.225.8064 (international callers dial +1.810.326.1444). In addition, for those using this book as CCIETM preparation, when you pass the CCIETM Lab exam, we want to hear about it! Email your CCIETM number to [email protected] and let us know how IPexpert helped you succeed. We would like to send you a gift of thanks and congratulations.

Additional CCIETM Preparation Material IPexpert, Inc. is committed to developing the most effective Cisco CCIETM R&S, Security, Voice, Wireless and Data Center Lab certification preparation tools available. Our team of certified networking professionals develops the most up-to-date and comprehensive materials for networking certification, including self-paced workbooks, online Cisco hardware rental, classroom training, online (distance learning) instructor-led training, audio products, and video training materials. Unlike other certification-training providers, we employ the most experienced and accomplished teams of experts to create, maintain, and constantly update our products. At IPexpert, we are focus on making your CCIETM Lab preparation more effective.

Issues with this Book This book is carefully edited to ensure the accuracy of all content. Should you find any error whatsoever, please email a page reference and detailed comment to [email protected]. Your email will be responded to promptly.



IPEXPERT END-USER LICENSE AGREEMENT END USER LICENSE FOR ONE (1) PERSON ONLY

IF YOU DO NOT AGREE WITH THESE TERMS AND CONDITIONS,

DO NOT OPEN OR USE THE TRAINING MATERIALS.

This is a legally binding agreement between you and IPEXPERT, the “Licensor,” from whom you have licensed the IPEXPERT training materials (the “Training Materials”). By using the Training Materials, you agree to be bound by the terms of this License, except to the extent these terms have been modified by a written agreement (the “Governing Agreement”) signed by you (or the party that has licensed the Training Materials for your use) and an executive officer of Licensor. If you do not agree to the License terms, the Licensor is unwilling to license the Training Materials to you. In this event, you may not use the Training Materials, and you should promptly contact the Licensor for return instructions. The Training Materials shall be used by only ONE (1) INDIVIDUAL who shall be the sole individual authorized to use the Training Materials throughout the term of this License.

Copyright and Proprietary Rights The Training Materials are the property of IPEXPERT, Inc. ("IPEXPERT") and are protected by United States and International copyright laws. All copyright, trademark, and other proprietary rights in the Training Materials and in the Training Materials, text, graphics, design elements, audio, and all other materials originated by IPEXPERT at its site, in its workbooks, scenarios and courses (the "IPEXPERT Information") are reserved to IPEXPERT.

The Training Materials cannot be used by or transferred to any other person. You may not rent, lease, loan, barter, sell or time-share the Training Materials or accompanying documentation. You may not reverse engineer, decompile, or disassemble the Training Materials. You may not modify, or create derivative works based upon the Training Materials in whole or in part. You may not reproduce, store, upload, post, transmit, download or distribute in any form or by any means, electronic, mechanical, recording or otherwise any part of the Training Materials and IPEXPERT Information other than printing out or downloading portions of the text and images for your own personal, non-commercial use without the prior written permission of IPEXPERT.

You shall observe copyright and other restrictions imposed by IPEXPERT. You may not use the Training Materials or IPEXPERT Information in any manner that infringes the rights of any person or entity.



Exclusions of Warranties THE TRAINING MATERIALS AND DOCUMENTATION ARE PROVIDED “AS IS.” LICENSOR HEREBY DISCLAIMS ALL OTHER WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. SOME STATES DO NOT ALLOW THE LIMITATION OF INCIDENTAL DAMAGES OR LIMITATIONS ON HOW LONG AN IMPLIED WARRANTY LASTS, SO THE ABOVE LIMITATIONS OR EXCLUSIONS MAY NOT APPLY TO YOU. This agreement gives you specific legal rights, and you may have other rights that vary from state to state.

Choice of Law and Jurisdiction This Agreement shall be governed by and construed in accordance with the laws of the State of Michigan, without reference to any conflict of law principles. You agree that any litigation or other proceeding between you and Licensor in connection with the Training Materials shall be brought in the Michigan state or courts located in Port Huron, Michigan, and you consent to the jurisdiction of such courts to decide the matter. The parties agree that the United Nations Convention on Contracts for the International Sale of Goods shall not apply to this License. If any provision of this Agreement is held invalid, the remainder of this License shall continue in full force and effect.

Limitation of Claims and Liability ANY ACTION ON ANY CLAIM AGAINST IPEXPERT MUST BE BROUGHT BY THE USER WITHIN ONE (1) YEAR FOLLOWING THE DATE THE CLAIM FIRST ACCRUED, OR SHALL BE DEEMED WAIVED. IN NO EVENT WILL THE LICENSOR’S LIABILITY UNDER, ARISING OUT OF, OR RELATING TO THIS AGREEMENT EXCEED THE AMOUNT PAID TO LICENSOR FOR THE TRAINING MATERIALS. LICENSOR SHALL NOT BE LIABLE FOR ANY SPECIAL, INCIDENTAL, INDIRECT, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, REGARDLESS OF WHETHER LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. WITHOUT LIMITING THE FOREGOING, LICENSOR WILL NOT BE LIABLE FOR LOST PROFITS, LOSS OF DATA, OR COSTS OF COVER.



Entire Agreement This is the entire agreement between the parties and may not be modified except in writing signed by both parties.

U.S. Government - Restricted Rights

The Training Materials and accompanying documentation are “commercial computer Training Materials” and “commercial computer Training Materials documentation,” respectively, pursuant to DFAR Section 227.7202 and FAR Section 12.212, as applicable. Any use, modification, reproduction release, performance, display, or disclosure of the Training Materials and accompanying documentation by the U.S. Government shall be governed solely by the terms of this Agreement and shall be prohibited except to the extent expressly permitted by the terms of this Agreement.

IF YOU DO NOT AGREE WITH THE ABOVE TERMS AND CONDITIONS, DO NOT OPEN OR USE THE TRAINING MATERIALS AND CONTACT LICENSOR FOR INSTRUCTIONS ON RETURN OF THE TRAINING MATERIAL



Contents IPexpert’s ...................................................................................................................................................... 1

Lab Preparation Detailed Solution Guide for Cisco’s CCIE Data Center Lab............................................. 1 Before We Begin ....................................................................................................................................... 1 Feedback ................................................................................................................................................... 2 Additional CCIETM Preparation Material ................................................................................................... 2 Issues with this Book ................................................................................................................................ 2 IPEXPERT END-USER LICENSE AGREEMENT .............................................................................................. 3 Copyright and Proprietary Rights ............................................................................................................. 3 Exclusions of Warranties .......................................................................................................................... 4 Choice of Law and Jurisdiction ................................................................................................................. 4 Limitation of Claims and Liability.............................................................................................................. 4 Entire Agreement ..................................................................................................................................... 5 U.S. Government - Restricted Rights ........................................................................................................ 5 Default Lab Topology ................................................................................................................................ 9 Default passwords and IP addresses ........................................................................................................ 9

Chapter 1: Introduction to CCIE Data Center .............................................................................................. 10 Who Should Read this Book? ................................................................................................................. 11 How to Use this Book ............................................................................................................................. 11 An Introduction to CCIE Data Center ...................................................................................................... 11 Availability .............................................................................................................................................. 12 Written exam .......................................................................................................................................... 12 The current published reading list:......................................................................................................... 12 Lab exam ................................................................................................................................................. 13 Software Versions ................................................................................................................................... 13 CCIE Storage? .......................................................................................................................................... 13 What about P and A tracks? ................................................................................................................... 13 Troubleshooting ..................................................................................................................................... 14 An Introduction to the Proctor Labs CCIE Data Center hardware rack .................................................. 14 Software Versions ................................................................................................................................... 17

Chapter 2: Data Center Networking Layer 2 Infrastructure ....................................................................... 18 (NX-OS) ........................................................................................................................................................ 18

Generic Lab topology .............................................................................................................................. 19 Introduction ............................................................................................................................................ 20 General Rules .......................................................................................................................................... 20 Solutions ................................................................................................................................................. 21

Task 1: General set-up ........................................................................................................................ 21 Task 2: Implement VLANs ................................................................................................................... 26 Task 3: Implement Private-VLANs ....................................................................................................... 33 Task 4: Implement Rapid Spanning-Tree protocol ............................................................................. 41 Task 5: Implement Multiple Spanning-Tree protocol ......................................................................... 48 Task 6: Spanning-Tree and UDLD features ......................................................................................... 56 Task 7: Fabric Extenders ..................................................................................................................... 58 Task 8: Misc features .......................................................................................................................... 61

Chapter 3: Data Center Networking Layer 3 Infrastructure (NX-OS) .......................................................... 64 General Rules .......................................................................................................................................... 65 Solutions ................................................................................................................................................. 66



Task 1: Layer 3 topology set-up .......................................................................................................... 66 Task 2: Static routing........................................................................................................................... 75 Task 3: EIGRP ....................................................................................................................................... 81 Task 4: OSPF ........................................................................................................................................ 94 Task 5: Redistribution, BFD and ECMP ............................................................................................. 116 Task 6: Layer 3 switching features .................................................................................................... 124 Task 7: FabricPath and OTV .............................................................................................................. 127

Chapter 4: Data Center Networking High Availability (NX-OS) ................................................................. 139 General Rules ........................................................................................................................................ 140 Solutions ............................................................................................................................................... 141 Task 1: Topology set-up ........................................................................................................................ 141

Task 2: Port-Channels ....................................................................................................................... 143 Task 3: Virtual Port-Channels (vPCs) ................................................................................................. 166 Task 4: Graceful Restart / Non-Stop Forwarding .............................................................................. 184 Task 5: HSRP ...................................................................................................................................... 188 Task 6: VRRP ...................................................................................................................................... 198 Task 7: GLBP ...................................................................................................................................... 202 Task 8: Virtual Port-Channels (vPCs) and FabricPath ........................................................................ 206

Chapter 5: Data Center Storage Networking ............................................................................................ 210 General Rules ........................................................................................................................................ 211 Solutions ............................................................................................................................................... 212

Task 1: Initial set-up .......................................................................................................................... 212 Task 2: VSANs .................................................................................................................................... 227 Task 3: Zoning ................................................................................................................................... 244 Task 4: FC Domain ............................................................................................................................. 254 Task 5: Fibre Channel Security Features ........................................................................................... 260 Task 6: Advanced Features ............................................................................................................... 271

Chapter 6: Data Center Storage Networking Extension ........................................................................... 276 General Rules ........................................................................................................................................ 277 Solutions ............................................................................................................................................... 278

Task 1: Initial set-up .......................................................................................................................... 278 Task 2: FCIP ....................................................................................................................................... 282 Task 3: FCIP Security ......................................................................................................................... 297 Task 4: SAN Extension Tuner ............................................................................................................ 301 Task 5: iSCSI....................................................................................................................................... 304 Task 6: iSLB ........................................................................................................................................ 323

Chapter 7: Data Center Unified Fabric ...................................................................................................... 340 General Rules ........................................................................................................................................ 341 Solutions ............................................................................................................................................... 342

Task 1: Native Fibre Channel on Nexus ............................................................................................. 342 Task 2: Fibre Channel over Ethernet (FCoE) ..................................................................................... 353 Task 3: Multi hop FCoE ..................................................................................................................... 358 Task 4: FCoE Quality of Service (QoS) ............................................................................................... 371 Task 5: N-Port Virtualization (NPV) and N-Port ID Virtualization (NPIV) .......................................... 381 Task 6: FCoE NPV .............................................................................................................................. 391

Chapter 8: Security Features ..................................................................................................................... 393 General Rules ........................................................................................................................................ 394 Solutions ............................................................................................................................................... 395



Task 1: Port Security ......................................................................................................................... 395 Task 2: DHCP Snooping, DAI, IP Source Guard .................................................................................. 412 Task 3: Access Control Lists ............................................................................................................... 419 Task 4: AAA services.......................................................................................................................... 435 Task 5: 802.1X ................................................................................................................................... 448 Task 6: Cisco TrustSec ....................................................................................................................... 451

Chapter 9: Management Features ............................................................................................................ 455 General Rules ........................................................................................................................................ 456 Solutions ............................................................................................................................................... 457

Task 1: Role Based Access Control (RBAC) ........................................................................................ 457 Task 2: Traffic monitoring ................................................................................................................. 470 Task 3: NetFlow ................................................................................................................................. 474 Task 4: Management protocols ........................................................................................................ 479 Task 5: Device management ............................................................................................................. 493 Task 6: Smart Call Home and GOLD .................................................................................................. 503



Default Lab Topology

Default passwords and IP addresses

• Default management username / password: admin / IPexpert123 • Other passwords: ipexpert • Management IP addressing: 172.16.100.0/24 • Management Default Gateway: 172.16.100.254



Chapter 1: Introduction to CCIE Data Center

Chapter 1: Introduction to CCIE Data Center introduces the team of authors, consultants, and editors that completed this book and describes the book’s purpose. This chapter also provides suggestions for the usage of this written work.



Who Should Read this Book? This workbooks primary audience is for those CCIE candidates that are searching for the most comprehensive and error-free materials available covering the CCIE Data Center practical lab exam. These students should possess a home rack of equipment for CCIE-level command-line practice, they should possess an equipment emulator (for certain parts of the topology), or they should rent equipment from a company like www.proctorlabs.com. The authors and technical editors exhaustively tested all of the demonstrations found throughout the technology tasks, troubleshooting- and full-scale lab exercises against all practice rack options described earlier. Where issues arise with popular equipment emulators, the text makes note. This book is the most remarkably thorough and technically accurate book written on the CCIE Data Center lab exam to date.

How to Use this Book This book breaks all specific CCIE Data Center technologies down on a chapter-by-chapter basis for a complete and thorough review of this broad set of topics. Each chapter is broken down is various tasks regarding the subject. Following this, the Detailed Solutions Guide provided with this workbook provides an intense examination of the operation of the tasks, including key aspects of troubleshooting for the specific technology. After this, the book presents some of the most common issues that can result with a particular technology-set, and most importantly, details the simple troubleshooting tools and steps that succeed for remediation.

The final chapters conclude the book with sample lab scenarios that provide a full scale lab exam as you will see it when you take the actual test. The Detailed Solutions Guide then provides a well-designed approach for troubleshooting each major task and offers detailed explanations. The text provides reference guides for the most popular and powerful show and debug commands for a specific technology.

Each chapter uses specific initial configurations on the specific chapter. Readers may download initial configurations, or install them in a simple Graphical User Interface (GUI) on www.proctorlabs.com.

Students are encouraged to follow along on a rack of equipment for every section of every chapter. This really enhances and strengthens the learning process.

An Introduction to CCIE Data Center Since the release of the Nexus platform there has been talk about when these platforms were to be introduced in a CCIE track. With the introduction of UCS in 2009 this became an even higher request especially since UCS really took off in sales.

http://www.proctorlabs.com/

http://www.proctorlabs.com/



The scope of the exam is pretty much based on the usual suspects, so in summary you should be aware of the:

• UCS B-series blade systems • UCS C-series rackmount systems connected to UCS Manager via FEX • Virtual Interface Cards (virtualized NICs and HBAs) in all servers • Nexus 7000 with all features like VDC, OTV, FabricPath, etc. • Nexus 5500 with all features like FCoE, FEX • Nexus 2000 connected to either the 5k or the 7k • Nexus 1000V distributed virtual switch in ESX

o There is no mention of any VMware product in the blueprint, so expect ESX and vCenter to be pre-installed on the UCS blades and FC boot to pre-configured disks

• MDS 9222i for connecting FC storage to UCS • ACE appliance • DCNM management software

Availability The live exam is available from September 1st.

Currently there are no dates when the lab is available.

Written exam The written exam has an extensive blueprint published to Cisco Learning Network (CLN) including a reading list.

The current published reading list:

Data Center Fundamentals (ISBN-10: 1-58705-023-4)

NX-OS and Cisco Nexus Switching (ISBN-10: 1-58705-892-8)

Cisco Unified Computing System (UCS) (ISBN-10: 1-58714-193-0)

I/O Consolidation in the Data Center (ISBN-10: 1-58705-888-X)

http://www.ciscopress.com/bookstore/product.asp?isbn=1587050234



http://www.ciscopress.com/bookstore/product.asp?isbn=158705888X



Storage Networking Fundamentals (ISBN-10: 1-58705-162-1)

Please find the extensive blueprint published by Cisco on the bottom of this blog post.

Lab exam

There is not much information available regarding the lab exam. Availability is not mentioned. There is however information regarding the hardware list and this is an immense list of expensive hardware you require:

Software Versions

• NXOS v6.0(2) on Nexus 7000 Switches • NXOS v5.1(3) on Nexus 5000 Switches • NXOS v4.2(1) on Nexus 1000V • NXOS v5.2(2) on MDS 9222i Switches • UCS Software release 2.0(1x) for UCS-6248 Fabric Interconnect and all UCS systems • Software Release A5(1.0) on ACE4710 • Cisco Data Center Manager software v5.2(2)

CCIE Storage? There are currently no plans for replacing CCIE Storage for CCIE Datacenter. Because of this, there will not be a large focus on MDS/FC configuration as there is another track for that.

What about P and A tracks? A CCNA Data Center and CCNP Data Center will be released soon!




Troubleshooting Troubleshooting will be a big part of the exam, which is also pretty clear in the blueprint. There is no confirmation yet how this will be introduced, either using tickets in the CCIE R&S or just by pre-configuration on the lab. I can imagine that they pre-configured a broken Nexus 1000V on an ESX installation on one of the JBODs. More information on how this troubleshooting is done will be available during other Q&A sessions. The implication is that it might be trouble tickets like the CCIE R&S.

An Introduction to the Proctor Labs CCIE Data Center hardware rack The IPexpert CCIE Data Center rack will support 100% of the features that are tested on the lab! We have based the topology to be close as possible on the CCIE Data Center rack layout, but have ensured that all features and functionality is there.

Our CCIE Data Center rack layout is based on the very limited information that has been made available by Cisco. IPexpert has been in close contact with the people involved in creating this lab exam, and therefore the layout of the rack is based on some early examples and the published components and software version blueprint.

As you will see the topology is very much based on a common datacenter design and has more 'static' layout than other CCIE tracks.

The blueprint specified the following components to be in the lab:

First is the NX-OS Networking equipment.

• Nexus7009 (with licensing) o (1) Sup o (1) 32 Port 10Gb (F1 Module) o (1) 32 Port 10Gb (M1 Module)

• Nexus5548 • Nexus2232

The Nexus 7000 will be configured with VDC's to simulate various different topologies and create multiple 'core switch' layers within the network.



Nexus 5548 will be used as a 'distribution' layer within the datacenter network. The Nexus 2k's can be configured as FEX for the Nexus 7000; Nexus 5000 and the Fabric Interconnects of the UCS system to connect the UCS C-series rack mount servers. The VDC's are a major component in the network as the number of devices is limited and the connectivity is very much based on a best practice design.

The below drawing illustrates an example topology from our new CCIE Data Center lab preparation workbook which is currently under development.

All these interconnections and switches are based within a single physical chassis with complete separation of the control and data plane protocols!

Second is the storage networking (SAN) equipment:

• Dual attached JBODs = Fibre Channel disks • MDS 9222i (dual fabric)

The MDS switches used in the lab are capable of a ton of features. The blueprint however only describes certain fibre-channel features which are considered 'basic' features like zoning, VSANs, oversubscription and ISLs. The other major topic on the blueprint is Fibre Channel Expansion over FCIP and iSCSI. These features are the IP features supported by the MDS platform. The 1G Ethernet connections are connected to the Nexus switches for testing the expansion features. Through that connection it's possible to connect the MDS switches across another connection than Fibre Channel. As the CCIE Storage track is



not being replaced by the CCIE Data Center the focus on Storage Networking (SAN) features is not that big. The major topics are more in the features that aren't tested in any other CCIE track.

The JBODs mentioned in this list represent just plain simple hard-disks that are connected via Fibre Channel. They are used later as shared storage for the UCS system.

The third major component within the hardware blueprint is the Unified Computing System (UCS).

• UCS-6248 Fabric Interconnects • UCS-5108 Blade Chassis

o B200 M2 Blade Servers o Palo/VIC mezzanine card o Menlo/Emulex mezzanine card

• UCS C200 Series Server = Connected to Fabric Interconnects o VIC card for C-series

This is based on the C-series rackmount servers, connected to the Fabric Interconnects so the C-series can also be managed from the central UCS manager the same as the Blade chassis is managed.

The blades are equipped with different NICs. This also means a little different configuration. The VIC cards are the most interesting ones as they can virtualize NICs to present to the OS.

Ones inside the blades there is a pre-installed VMware ESX(i) environment with a Nexus 1000v distributed virtual switch. As this is a Cisco lab exam, you are not required to know anything about VMware. Of course you will need to be able to install this environment in your possible own lab, but when you step into the lab you will face a pre-installed VMware and 1000V. After that, the switch is not configured and you are required to configure it.

The final topic on the blueprint is called ANS (Application Networking Services). This means an ACE appliance is in your lab that you will need to configure. There is not much very interesting going on there and you will not see a lot of points on that appliance. You will need to know the topics as described on the lab blueprint and our workbook will focus a whole section on these specific topics.

The last components are used for management. You will not be configuring these devices, but just using them from your student workstation to access the network.



• Cisco Catalyst Switch 3750 = management ethernet connections • Cisco 2511 Terminal Server = console lines

What is not mentioned on the hardware blueprint list is that you will also need to be able to configure (or set-up) the DCNM software as is being given by Cisco when you purchase enough Nexus equipment. Again this is not extremely difficult, but you need to be aware of the basic configuration items related to this software.

Software Versions

• NXOS v6.0(2) on Nexus 7000 Switches • NXOS v5.1(3) on Nexus 5000 Switches • NXOS v4.2(1) on Nexus 1000v • NXOS v5.2(2) on MDS 9222i Switches • UCS Software release 2.0(1x) for UCS-6248 Fabric Interconnect and UCS system • Software Release A5(1.0) for ACE 4710 • Cisco Data Center Manager software v5.2(2)

Above you'll find a reference overview of the used software versions. The exact versions are still unknown where we might be using newer software versions as our IPexpert lab will be using quite new hardware for virtualization purposes. Within the Nexus 7000 we will be using the new Supervisor 2E, meaning that we are able to build 8 VDC's and 1 management VDC meaning we have enough flexibility for some challenging topologies!

The next chapter of this workbook, Chapter 2: Data Center Networking Layer 2 Infrastructure (NX-OS) begins with the initial topic on the CCIE Data Center Blueprint regarding layer 2 switching, VLANs, Private-VLANs, Spanning-Tree and other layer 2 features on the NX-OS platform.



Chapter 2: Data Center Networking Layer 2 Infrastructure

(NX-OS)

Chapter 2: Data Center Networking Layer 2 Infrastructure (NX-OS) is intended to let you be familiar with the NX-OS CLI on the Nexus switches and afterwards configure Layer 2 Ethernet features on the physical Nexus switches within the topology as shown at the beginning of this workbook. We highly recommend to create your own diagram at the beginning of each lab so you are able to draw on your own diagram, making it much easier when you step into the real lab. Our devices start with a blank configuration, which will not be the case when you are in the real lab. Then devices are staged with configuration containing usernames/passwords, management IP addressing, core IP addressing and (possible) errors.



Generic Lab topology



Introduction This first lab is intended to let you be familiar with the NX-OS CLI on the Nexus switches and afterwards configure Layer 2 Ethernet features on the physical Nexus switches within the topology as shown at the beginning of this workbook.

We highly recommend to create your own diagram at the beginning of each lab so you are able to draw on your own diagram, making it much easier when you step into the real lab.

Our devices start with a blank configuration, which will not be the case when you are in the real lab. Then devices are staged with configuration containing usernames/passwords, management IP addressing, core IP addressing and (possible) errors.

General Rules • Try to diagram out the task. Draw your own connections the way you like it

• Create a checklist to aid as you work thru the lab

• Take a very close read of the tasks to ensure you don’t miss any points during grading!

• Take your time. This is not a Mock Lab, so no time constraints are in place for finishing this particular chapter

Estimated Time to Complete: 3 hours



Solutions

Task 1: General set-up The write command is no longer available within NX-OS, though you need to use write erase to erase the configuration from all boxes. When rebooting from empty configuration you are asked to configure base defaults to manage the system.

Nexus1# write erase Warning: This command will erase the startup-configuration. Do you wish to proceed anyway? (y/n) [n] y Nexus1# reload This command will reboot the system. (y/n)? [n] y 2012 Aug 26 19:04:24 Nexus1 %$ VDC-1 %$ %PLATFORM-2-PFM_SYSTEM_RESET: Manual system restart from Command Line Interface

writing reset reason 9, NX7 SUP Ver 3.22.0 Serial Port Parameters from CMOS PMCON_1: 0x200 PMCON_2: 0x0 PMCON_3: 0x3a PM1_STS: 0x101 Performing Memory Detection and Testing Total mem found : 4096 MB Performing memory test... Passed. NumCpus = 2. Status 61: PCI DEVICES Enumeration Started Status 62: PCI DEVICES Enumeration Ended Status 9F: Dispatching Drivers Status 9E: IOFPGA Found Status 9A: Booting From Primary ROM Status 98: Found Cisco IDE Status 98: Found Cisco IDE Status 90: Loading Boot Loader Reset Reason Registers: 0x0 0x10 Filesystem type is ext2fs, partition type 0x83 GNU GRUB version 0.97 Autobooting bootflash:/n7000-s1-kickstart.5.1.5.bin bootflash:/n7000-s1-dk9.5.1

.5.bin... Filesystem type is ext2fs, partition type 0x83 Booting kickstart image: bootflash:/n7000-s1-kickstart.5.1.5.bin.... ...............................................................................

....................Image verification OK INIT: version 2 Checking all filesystems..r.r.r..r done. Loading system software



/bootflash//n7000-s1-dk9.5.1.5.bin read done Uncompressing system image: bootflash:/n7000-s1-dk9.5.1.5.bin Sun Aug 26 19:08:10 UTC 2012

blogger: nothing to do. ..done Sun Aug 26 19:08:13 UTC 2012 Load plugins that defined in image conf: /isan/plugin_img/img.conf Loading plugin 0: core_plugin... num srgs 1 0: swid-core-supdc3, swid-core-supdc3 num srgs 1 0: swid-supdc3-ks, swid-supdc3-ks INIT: Entering runlevel: 3 2012 Aug 26 19:08:38 %$ VDC-1 %$ %DAEMON-2-SYSTEM_MSG: <<%PLATFORM-2-PS_PWR_INPUT_MISSING>> Power supply 1 present but all AC/DC inputs are not connected, power redundancy might be affected - platform[2674]

2012 Aug 26 19:08:38 %$ VDC-1 %$ %DAEMON-2-SYSTEM_MSG: <<%PLATFORM-2-PS_ABSENT>> Power supply 2 is absent/shutdown, ps-redundancy might be affected - platform[2674]



2012 Aug 26 19:08:52 %$ VDC-1 %$ %CMPPROXY-2-LOG_CMP_UP: Connectivity Management processor(on module 9) is now UP

2012 Aug 26 19:08:58 %$ VDC-1 %$ %IDEHSD-2-MOUNT: logflash: online 2012 Aug 26 19:09:09 %$ VDC-1 %$ %COPP-2-COPP_NO_PINST: Control-plane is unprotected.

2012 Aug 26 19:09:12 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 1 has come online

---- System Admin Account Setup ---- Do you want to enforce secure password standard (yes/no) [y]: 2012 Aug 26 19:09:14 switch %$ VDC-1 %$ %PLATFORM-2-PS_OK: Power supply 1 ok (Serial number DTM1437011H)

2012 Aug 26 19:09:14 switch %$ VDC-1 %$ %PLATFORM-2-PS_FANOK: Fan in Power supply 1 ok

2012 Aug 26 19:09:15 switch %$ VDC-1 %$ %PLATFORM-2-XBAR_DETECT: Xbar 1 detected (Serial number JAF1524AKTF)

2012 Aug 26 19:09:23 switch %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 3 detected (Serial number JAF1448BPGN) Module-Type 10 Gbps Ethernet Module Model N7K-M132XP-12

2012 Aug 26 19:09:23 switch %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 3 powered up (Serial number JAF1448BPGN)

2012 Aug 26 19:09:25 switch %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 5 detected (Serial number JAF1448BBDK) Module-Type 10/100/1000 Mbps Ethernet Module Model N7K-M148GT-11

2012 Aug 26 19:09:25 switch %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 5 powered up (Serial number JAF1448BBDK)



2012 Aug 26 19:09:38 switch %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 1 (Fan1(sys_fan1) fan) ok

2012 Aug 26 19:09:38 switch %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 2 (Fan2(sys_fan2) fan) ok

2012 Aug 26 19:11:51 switch %$ VDC-1 %$ %XBAR-2-XBAR_INSUFFICIENT_XBAR_BANDWIDTH: Module in slot 3 has insufficient xbar-bandwidth.

Enter the password for "admin": IPexpert123 Confirm the password for "admin": IPexpert123 ---- Basic System Configuration Dialog VDC: 1 ---- This setup utility will guide you through the basic configuration of the system. Setup configures only enough connectivity for management of the system. Please register Cisco Nexus7000 Family devices promptly with your supplier. Failure to register may affect response times for initial service calls. Nexus7000 devices must be registered to receive entitled support services. Press Enter at anytime to skip a dialog. Use ctrl-c at anytime to skip the remaining dialogs. Would you like to enter the basic configuration dialog (yes/no): no User Access Verification switch login: admin Password: Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php switch#

Hostnames are configured using the switchname or hostname command. Initially the hostname command was not available when NX-OS was first forked from SAN-OS. Since this caused a lot of confusion the hostname command has been ported back into NX-OS. When configured, the one that is last used will be shown in the configuration. Pay attention to this as this might be a requirement during the lab.

switch# conf t Enter configuration commands, one per line. End with CNTL/Z.

http://www.cisco.com/tac

http://www.opensource.org/licenses/gpl-2.0.php

http://www.opensource.org/licenses/lgpl-2.1.php



switch(config)# hostname SW1-1 SW1-1(config)# sh run | in SW1 hostname SW1-1 vdc SW1-1 id 1 SW1-1(config)# switchname SW1-1 SW1-1(config)# sh run | in SW1 switchname SW1-1 vdc SW1-1 id 1 SW1-1(config)# ip domain-name ipexpert.com SW1-1(config)# no ip domain-lookup SW1-1(config)#

Configuring the domain-name and DNS lookups is equal as to how it’s done within IOS.

To ensure both SSH and Telnet connections are allowed towards the management (mgmt0) interface of the Nexus you should enable Telnet using the feature command. SSH is the default mechanism to manage the Nexus switch and therefore the Telnet feature needs to be re-enabled. By default the switch rejects any attempts of telnet.

To save your configuration using the wr command. The only option is to create a CLI alias, which can be executed from anywhere in the CLI. The write command is reserved for the erasing of the configuration as we did at the beginning of the lab, so we need to use an abbreviation for that (what most engineers use in real-life as well).

SW1-1(config)# sh run int mgmt0 !Command: show running-config interface mgmt0 !Time: Sun Aug 26 19:24:59 2012 version 5.1(5) interface mgmt0 ip address 10.160.48.241/24 SW1-1(config)# feature telnet SW1-1(config)# cli alias name write copy run start Invalid alias name. Reserved word. SW1-1(config)# cli alias name wr copy run start SW1-1(config)# wr [########################################] 100% Copy complete, now saving to disk (please wait)... SW1-1(config)#

Note: The management IP address wasn’t erased during the erase of the configuration. This doesn’t mean you can just erase your configuration and reload from the Ethernet management connection, because you are required to set a password for the admin user. Meaning you can’t log-in using the SSH connection that is by default enabled.

By default the SSH log-in is enabled.



IPexpert:~ rick$ ssh [email protected] The authenticity of host '10.160.48.241 (10.160.48.241)' can't be established.

RSA key fingerprint is 95:23:75:04:1c:2e:59:f1:be:a1:e5:f7:ff:62:6d:80. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.160.48.241' (RSA) to the list of known hosts.

User Access Verification Password: Last login: Sun Aug 26 19:13:54 2012 Bad terminal type: "xterm-256color". Will assume vt100. Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php SW1-1# exit Connection to 10.160.48.241 closed. RetinaRick:~ rick$ telnet 10.160.48.241 Trying 10.160.48.241... telnet: connect to address 10.160.48.241: Connection refused telnet: Unable to connect to remote host

Only after enabling the Telnet feature on the CLI you are able to establish a Telnet connection to the device.

IPexpert:~ rick$ telnet 10.160.48.241 Trying 10.160.48.241... Connected to 10.160.48.241. Escape character is '^]'. Password: telnet> q Connection closed. IPexpert:~ rick$

Configure the other tasks by just navigating through the configuration. You will never remember all options available within the OS, but by just navigating using keywords you should be seeing a lot of hints pointing to the right solution according to what was asked in the task.

SW1-1(config)# cdp format device-id ? mac-address Mac-address of the Chassis serial-number Chassis Serial Number/OUI system-name System name/Fully Qualified Domain Name (Default) SW1-1(config)# cdp format device-id serial-number SW1-1(config)# cdp advertise v2 SW1-1(config)# int mgmt0






SW1-1(config-if)# no cdp enable SW1-1(config-if)# sh run int mgmt0 !Command: show running-config interface mgmt0 !Time: Sun Aug 26 19:56:49 2012 version 5.1(5) interface mgmt0 no cdp enable ip address 10.160.48.241/24 SW1-1(config-if)# rate-limit cpu direction ? both Set max input and output packet rate input Set max input packet rate output Set max output packet rate SW1-1(config-if)# rate-limit cpu direction both pps 999 action log SW1-1(config-if)# sh run int mgmt0 !Command: show running-config interface mgmt0 !Time: Sun Aug 26 19:57:20 2012 version 5.1(5) interface mgmt0 no cdp enable ip address 10.160.48.241/24 rate-limit cpu direction both pps 999 action log

Task 2: Implement VLANs The first task tells you to configure all inter switch links (as seen on the topology drawing) to be configured as layer 2 trunks with 802.1Q encapsulation. Besides that you should only allow a certain number of vlans, meaning the allowed vlans command should be used.

SW1-1(config)# default interface e3/1 SW1-1(config)# int e3/1 SW1-1(config-if)# sh run int e3/1 !Command: show running-config interface Ethernet3/1 !Time: Thu Sep 6 20:24:28 2012 version 5.1(5) interface Ethernet3/1 SW1-1(config-if)# switchport SW1-1(config-if)# sw mode trunk SW1-1(config-if)# sw trunk allowed vlan 100-499 This will cause VLANS to be overwritten. Continue anyway? [yes] yes



Because this command caused a lot of outages in the world, Cisco implemented a security to ask you if you are really sure that you want this command to be entered.

SW1-1(config-if)# sh run int e3/1 !Command: show running-config interface Ethernet3/1 !Time: Thu Sep 6 20:24:39 2012 version 5.1(5) interface Ethernet3/1 switchport switchport mode trunk switchport trunk allowed vlan 100-499

In Classic IOS you only saw the shutdown command under the interface. Within NX-OS the opposite is true and you only see the no shutdown. Don’t think the port is enabled when you see this configuration. You really need to see the no shutdown command under the interface before it really works!

SW1-1(config-if)# no shut SW1-1(config-if)# sh run int e3/1 !Command: show running-config interface Ethernet3/1 !Time: Thu Sep 6 20:24:47 2012 version 5.1(5) interface Ethernet3/1 switchport switchport mode trunk switchport trunk allowed vlan 100-499 no shutdown SW1-1(config-if)#

To remove a specific range (or one VLAN) from the allowed range, you don’t need to enter the whole range again as you would need to in older IOS releases. NX-OS supports the remove command and an except command.

Be aware when using the except command, because this will remove all VLANs from the trunk without warning you! Besides it’s not what the task asked you. You should remove the VLAN from the allowed range without specifying it. We do this with the remove command.

SW1-1(config-if)# switchport trunk allowed vlan except 333 SW1-1(config-if)# sh run int e3/1 !Command: show running-config interface Ethernet3/1 !Time: Thu Sep 6 20:29:42 2012



version 5.1(5) interface Ethernet3/1 switchport switchport mode trunk switchport trunk allowed vlan 1-332,334-3967,4048-4093 no shutdown SW1-1(config-if)# sw trunk allowed vlan 100-499 This will cause VLANS to be overwritten. Continue anyway? [yes] SW1-1(config-if)# sw trunk allowed vlan remove 333 SW1-1(config-if)# sh run int e3/1 !Command: show running-config interface Ethernet3/1 !Time: Thu Sep 6 20:30:05 2012 version 5.1(5) interface Ethernet3/1 switchport switchport mode trunk switchport trunk allowed vlan 100-332,334-499 no shutdown SW1-1(config-if)#

Of course the proctor will never be able to verify if you used this single command, but this workbook prepares you for the lab and every possible question. So this task shows you the possibilities of the OS.

The next tasks will be about configuring the VTP protocol. This protocol ensures all VLANs are distributed throughout the network. As with almost all features within NX-OS you need to enable it first before configuring it. The tasks in the workbook will push you to configure about every feature VTP has and the following configuration will show you all those steps.

If you check the Cisco Documentation about Layer 2 Switching you will see similar configuration examples which is also what you will see when you sit the lab. Most questions can be related to the Documentation as that was used during the creation of the lab itself.

SW1-1(config)# feature vtp Once enabled the VTP command becomes available. When you check the options you have, you will see about every task can be answered with these commands.

SW1-1(config)# vtp ? domain Set the name of the VTP administrative domain file Set the name of the VTP file name mode Configure VTP device mode password Set the password for the VTP administrative domain pruning Set the adminstrative domain to permit pruning



version Set the adminstrative domain to VTP version SW1-1(config)# vtp domain IPexpert SW1-1(config)# vtp pruning

VTP pruning is the option to allow VLANs to be removed from switches where they are not required. This limits the broadcast domains created (automatically) using VTP. This is also the main reason why VTP is not recommended to be used in large datacenters.

SW1-1(config)# vtp version ? <1-2> Set the adminstrative domain to VTP version

The task asks you to configure the latest version. Which is the highest number?

SW1-1(config)# vtp version 2 SW1-1(config)# vtp file ? bootflash: URI for vlan.dat usb1: URI for vlan.dat usb2: URI for vlan.dat

Vlan.dat has historically been the VLAN database. Today it’s the VTP database which is separately stored on the flash. When you erase the configuration of a NX-OS device, the VTP information will remain on the flash and will be configured again when re-enabled.

SW1-1(config)# vtp file ipexpert.dat SW1-1(config)# vtp mode server

The mode we are looking for in the task is that SW1 is the VTP server for the network, while SW2 and SW3 are the clients. This way SW2 and SW3 are not able to create vlans. When you try to do this, you will get an error message.

SW1-1(config)# vtp password ipexpert

Verify that everything is properly configured

SW1-1(config)# sh vtp status VTP Status Information ---------------------- VTP Version : 2 (capable) Configuration Revision : 4 Maximum VLANs supported locally : 1005 Number of existing VLANs : 9 VTP Operating Mode : Server VTP Domain Name : IPexpert VTP Pruning Mode : Enabled (Operationally Enabled) VTP V2 Mode : Enabled VTP Traps Generation : Disabled



MD5 Digest : 0xF3 0xA6 0x62 0x7A 0xB1 0x85 0x77 0x40 Configuration last modified by 0.0.0.0 at 8-26-12 21:14:51 Local updater ID is 0.0.0.0 VTP version running : 2

To verify the VTP password a different command is required

SW1-1(config)# show vtp password VTP password: ipexpert

After creating the VLANs in the next task you should verify on all switches if VTP pushed the information to those switches. After configuring the names of the VLANs, verify again, as VTP should be pushing that information towards all switches as well.

SW1-1(config)# vlan 101-104 SW1-1(config-vlan)# exit SW1-1(config)# vlan 101 SW1-1(config-vlan)# name IPexpertVLAN101 SW1-1(config-vlan)# vlan 102 SW1-1(config-vlan)# name IPexpertVLAN102 SW1-1(config-vlan)# vlan 103 SW1-1(config-vlan)# name IPexpertVLAN103 SW1-1(config-vlan)# vlan 104 SW1-1(config-vlan)# name IPexpertVLAN104 SW1-1(config-vlan)# sh vlan VLAN Name Status Ports ---- -------------------------------- --------- -------------------------------

1 default active 101 IPexpertVLAN101 active Eth3/1 102 IPexpertVLAN102 active Eth3/1 103 IPexpertVLAN103 active Eth3/1 104 VLAN0104 active Eth3/1 1002 fddi-default suspended 1003 token-ring-default suspended 1004 fddinet-default suspended 1005 trnet-default suspended VLAN Type Vlan-mode ---- ----- ---------- 1 enet CE 101 enet CE 102 enet CE 103 enet CE 104 enet CE 1002 enet CE 1003 enet CE 1004 enet CE 1005 enet CE Remote SPAN VLANs



-------------------------------------------------------------------------------

Primary Secondary Type Ports ------- --------- --------------- -------------------------------------------

As you can see the name of VLAN 104 hasn’t been changed yet. This is caused by the fact that VLANs are still located in some sort of database within the configuration. You first need to exit out of the VLAN configuration before the changes are applied

SW1-1(config-vlan)# exit SW1-1(config)# sh vlan VLAN Name Status Ports ---- -------------------------------- --------- -------------------------------

1 default active 101 IPexpertVLAN101 active Eth3/1 102 IPexpertVLAN102 active Eth3/1 103 IPexpertVLAN103 active Eth3/1 104 IPexpertVLAN104 active Eth3/1 1002 fddi-default suspended 1003 token-ring-default suspended 1004 fddinet-default suspended 1005 trnet-default suspended VLAN Type Vlan-mode ---- ----- ---------- 1 enet CE 101 enet CE 102 enet CE 103 enet CE 104 enet CE 1002 enet CE 1003 enet CE 1004 enet CE 1005 enet CE Remote SPAN VLANs -------------------------------------------------------------------------------

Primary Secondary Type Ports ------- --------- --------------- -------------------------------------------

SW1-1(config)#



Now verify the VLANs on all 3 switches to ensure that everything has been synchronized using VTP.

The last option to configure in this task is a special case. You really need to pay attention to the fact what is happening in the show command.

Pay close attention that in the IGMP snooping show command VLAN 105 is shown, while in the show vlan, you don’t see this 105. You probably need to check the Cisco Documentation for this feature, but it’s meant for ‘pre-staging’ VLANs without creating them first.

This means it’s possible to enable IGMP snooping on a VLAN before it’s created and applying this configuration immediately when deployed.

It’s a corner-case feature, but Cisco is known to use those particular features in 1 or 2 point tasks within the CCIE lab. So this could be easy points to get when you know what they are looking for, or you could easily loose points when you don’t and also have no clue where to look for. Again looking at Cisco Documentation helps a lot in this case and it should be easy to spot this feature in the layer 2 documentation guide.

SW1-1(config)# vlan configuration 105 SW1-1(config-vlan-config)# ? ip Configure IP features no Negate a command or set its defaults service-policy Configure service policy for an interface end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1-1(config-vlan-config)# ip igmp snooping

Another feature to configure is a service-policy in this section. With this policy you can apply QoS markings or Policing at VLAN level instead of port-level. Be aware that you will mark and police ALL traffic entering and exciting the VLAN on this switch.

SW1-1(config-vlan-config)# service-policy type qos

After enabling the IGMP snooping this shows up in the show output of IGMP snooping just like the task required you to. Still the VLAN hasn’t been created yet.

SW1-1(config)# exit SW1-1# wr [########################################] 100% Copy complete, now saving to disk (please wait)... SW1-1#



Save your configuration before moving on with the next task. Frequently saving you configuration might save you a lot of trouble when you are configuring features that might cause the box to crash. Or you reload the box without saving and you lose precious points!

Task 3: Implement Private-VLANs The assignments in this task are created in such a way that you are building a private VLAN based network. The assignments are becoming quite clear when you read the full task before starting the configuration as the assignments individually can be a little difficult to understand, but the entire picture is much clearer.

Private VLANs are set-up in such a way that traffic is protected between certain ports. There is a Primary VLAN which only contains the so-called Promiscuous ports. These ports can receive all traffic from all ports that are associated with this primary VLAN. This means that the ‘client’ ports are in other VLANs than the promiscuous port.

The secondary VLANs can be create in 2 ways. The “Isolated” and “Community” VLANs are these 2 types of secondary VLANs. The Isolated version means that all ports that are configured in these VLANs cannot communicate to each other. As pointed out in the drawing below. Within the layer 2 domain of this particular VLAN ports will never communicate with each other, but can only talk to the promiscuous port in the primary VLAN which they are associated with. The second type is the Community VLANs. All interfaces in these VLANs can communicate to each other within the same Community VLAN. This means that hosts in different community VLANs cannot communicate to each other. Again these community VLANs are able to talk to the promiscuous port in the primary VLAN which they are associated with.

The drawing below illustrates the possible (and impossible) traffic flows.



The associations / mappings between primary and secondary VLANs need to be manually configured on the switch ports. The final step is that it’s also possible to trunk Private VLANs between different switches. This means that a promiscuous port doesn’t need to be on the same switch as the host ports. Therefore a whole isolated section of your network can be created on different switches.

The promiscuous port doesn’t have to be a fixed Ethernet port. It’s also possible to create a VLAN interface (SVI) which is the promiscuous port for that particular Primary VLAN.

The first assignment point you to create a promiscuous port for an external firewall which will receive all traffic connected to Primary VLAN 200.

As with all features within the NX-OS platform the private VLAN feature also needs to be activated first, before any commands are available within the CLI.

SW1-1(config)# feature private-vlan

Then the VLAN needs to be created and the VLAN needs to be made primary. Which we do under the VLAN configuration.



SW1-1(config)# vlan 200 SW1-1(config-vlan)# private-vlan ? association Configure association between private VLANs community Configure the VLAN as community private VLAN isolated Configure the VLAN as isolated private VLAN primary Configure the VLAN as primary private VLAN SW1-1(config-vlan)# private-vlan primary SW1-1(config-vlan)# sh vlan private-vlan Primary Secondary Type Ports ------- --------- --------------- -------------------------------------------

Still we don’t see anything, but this is due to the fact that we need to exit out of the VLAN configuration before it’s applied to the switch.

SW1-1(config-vlan)# exit ERROR: Private VLANs can only be configured in VTP transparent/off modes

We receive an error that VTP is not possible to carry private VLAN information. This is true for the supported versions of VTP on the NX-OS platform. Older Catalyst switches running IOS or even CatOS also support VTP version 3, which is capable of carrying the private VLAN configuration. This code is currently not implemented in NX-OS.

SW1-1(config)# vtp mode transparent SW1-1(config)# vlan 200 SW1-1(config-vlan)# private-vlan primary SW1-1(config-vlan)# exit SW1-1(config-if)# int e3/19 SW1-1(config-if)# switchport SW1-1(config-if)# sw mode private-vlan promiscuous

When we change VTP to transparent mode we do see that the changes are applied and that we enabled a primary VLAN. We don’t have to create any mappings yet as that will come in the following assignments.

SW1-1(config)# sh vlan private-vlan Primary Secondary Type Ports ------- --------- --------------- -------------------------------------------

200 primary E3/19

The next assignment is a combination of enabling isolated ports on the same switch. Before they are able to communicate to the promiscuous port and the firewall some mappings need to be created.



SW1-1(config)# vlan 201 SW1-1(config-vlan)# private-vlan isolated SW1-1(config)# vlan 200 SW1-1(config-vlan)# private-vlan association 201 SW1-1(config-vlan)# exit SW1-1(config)# sh vlan private-vlan Primary Secondary Type Ports ------- --------- --------------- -------------------------------------------

200 201 isolated SW1-1(config)# SW1-1(config-if)# int e3/19 SW1-1(config-if)# sw private-vlan ? association Private vlan trunk association host-association Set the private VLAN host association mapping Set the private VLAN trunk promiscuous mapping trunk Set the private vlan trunking configuration SW1-1(config-if)# sw private-vlan mapping ? <1-3967,4048-4093> Primary private VLAN trunk Private-vlan trunk promiscuous SW1-1(config-if)# sw private-vlan mapping 200 ? <1-3967,4048-4093> Secondary VLAN IDs add Add a VLAN to private VLAN list remove Remove a VLAN from private VLAN list SW1-1(config-if)# sw private-vlan mapping 200 201

After the creation of the new isolated VLAN a mapping needs to be made between the primary and secondary VLANs. This is done under the primary VLAN configuration and under the promiscuous port.

This model of having to create the mapping on multiple places is sometimes a bit difficult to remember every spot where the mapping needs to be done, but in real life you will see you get a lot of flexibility with this model and you are able to create almost any topology with this flexibility.

Now the final step is to create the same mapping on the isolated host ports.

SW1-1(config)# int ethernet 3/20-21 SW1-1(config-if-range)# switchport mode private-vlan ? host Port mode pvlan host promiscuous Port mode pvlan promiscuous trunk Private-vlan trunk promiscuous SW1-1(config-if-range)# switchport mode private-vlan host ERROR: Ethernet3/20-21: requested config change not allowed



Don’t forget to enable the port on the Nexus 7000 as switchport, as by default the ports are in non switching mode.

SW1-1(config-if-range)# int e3/20 SW1-1(config-if)# switchport SW1-1(config-if)# switchport mode private-vlan host SW1-1(config-if)# switchport private-vlan host-association ? <1-3967,4048-4093> Primary VLAN ID SW1-1(config-if)# switchport private-vlan host-association 200 ? <1-3967,4048-4093> Secondary VLAN ID SW1-1(config-if)# switchport private-vlan host-association 200 201 SW1-1(config-if)# SW1-1(config-if)# int e3/21 SW1-1(config-if)# switchport SW1-1(config-if)# switchport mode private-vlan host SW1-1(config-if)# sw priv host-association 200 201 SW1-1(config-if)# exit SW1-1(config)# SW1-1(config)# sh vlan private-vlan Primary Secondary Type Ports ------- --------- --------------- -------------------------------------------

200 201 isolated Eth3/19, Eth3/20, Eth3/21

After the mapping has been created on all 3 ports (promiscuous and host ports) you will see this mapping in the show vlan. Here all 3 ports are shown, but this doesn’t mean that ports can communicate to each other. The type of ‘isolate’ means that this is not possible for the host ports and only for the promiscuous port.

The following 2 assignments point you in the direction of community VLANs. These VLANs are able to communicate to each other, but not between different community VLANs and of course they are able to communicate towards the promiscuous port.

Remember that all traffic on this promiscuous port is untagged. So it’s not possible to distinguish traffic between different secondary VLANs on the firewall (virtually connected to port Ethernet 3/19).

SW1-1(config)# vlan 202 SW1-1(config-vlan)# private-vlan community SW1-1(config-vlan)# vlan 203 SW1-1(config-vlan)# private-vlan community SW1-1(config-vlan)# vlan 200 SW1-1(config-vlan)# private-vlan association add 202-203 SW1-1(config-vlan)# sh vlan private Primary Secondary Type Ports ------- --------- --------------- -------------------------------------------

200 201 isolated Eth3/19, Eth3/20, Eth3/21 202 community 203 community



When the new community VLANs are added, again the mapping needs to be created between the primary and the secondary VLAN.

The following is that the interfaces need to be assigned to the correct community.

SW1-1(config-vlan)# int e3/22-23 SW1-1(config-if-range)# switch SW1-1(config-if-range)# switchport mode private-vlan host SW1-1(config-if-range)# switchport private host 200 202

Next are the second set of ports that are not allowed to communicate to hosts in VLAN 202

SW1-1(config-if-range)# int e3/24-25 SW1-1(config-if-range)# switch SW1-1(config-if-range)# switchport mode private-vlan host SW1-1(config-if-range)# switchport private host 200 203 SW1-1(config-if-range)# sh vlan private Primary Secondary Type Ports ------- --------- --------------- -------------------------------------------

200 201 isolated Eth3/19, Eth3/20, Eth3/21 200 202 community Eth3/22, Eth3/23 200 203 community Eth3/24, Eth3/25

And finally don’t forget to add the community VLANs to the promiscuous port, otherwise they won’t be able to communicate with the firewall!

Don’t forget the ‘add’ keyword in the command; otherwise you will overwrite all current enabled VLANs just like it is possible with the trunk allowed vlan command!

SW1-1(config-if-range)# int e3/19 SW1-1(config-if)# sw private mapping 200 add 202-203 SW1-1(config-if)# sh vlan private Primary Secondary Type Ports ------- --------- --------------- -------------------------------------------

200 201 isolated Eth3/19, Eth3/20, Eth3/21 200 202 community Eth3/19, Eth3/22, Eth3/23 200 203 community Eth3/19, Eth3/24, Eth3/25 SW1-1(config-if)#

The next task requires you to create a different primary VLAN. This primary VLAN will not be configured towards a physical interface, but will be configured under a layer 3 interface on the core switch (SW1-1). This is a virtual promiscuous port. Configuration however is basically the same as a physical interface!

SW1-1(config)# vlan 204 SW1-1(config-vlan)# private-vlan primary SW1-1(config-vlan)# exit SW1-1(config)# interface vlan 204 ^



Invalid interface format at '^' marker.

On the NX-OS platform it’s not possible to configure VLAN interfaces (SVIs) right away. Again you need to enable the feature first!

SW1-1(config)# feature interface-vlan SW1-1(config)# interface vlan 204 SW1-1(config-if)# ip address 10.1.10.254/24 SW1-1(config-if)# no shut SW1-1(config-if)# private-vlan mapping 205 SW1-1(config-if)# exit

On the SVI the private VLAN mapping to the primary VLAN needs to be configured.

SW1-1(config)# sh int vlan 204 private-vlan mapping Interface Secondary VLAN --------- ----------------------------------------------------------------

For the next task another isolated VLAN needs to be created. This VLAN is associated with the special primary VLAN. After that configuration the mapping is visible on the SVI of VLAN 204.

SW1-1(config)# vlan 205 SW1-1(config-vlan)# private-vlan isolated SW1-1(config-vlan)# vlan 204 SW1-1(config-vlan)# private-vlan association 205 SW1-1(config-vlan)# exit SW1-1(config)# sh int vlan 204 private-vlan mapping Interface Secondary VLAN --------- ---------------------------------------------------------------- vlan204 205

The next task requires the use of so-called private VLAN trunk links as the hosts are not connected directly to the SW1-1, but to SW2. Private-VLAN trunk links can carry traffic for normal VLANs and private VLANs at the same time.

The configuration can be to transfer primary VLANs to multiple switches or to transfer secondary VLANs across multiple switches. In this task we configure the first trunk link to pass only secondary VLANs.

SW1-1

SW1-1(config)# int e3/1 SW1-1(config-if)# sw mode private-vlan trunk ? promiscuous Port mode trunk promiscuous secondary Port mode trunk isolated SW1-1(config-if)# sw mode private-vlan trunk secondary SW1-1(config-if)# sw private-vlan trunk allowed vlan 205 SW1-1(config-if)# sw privat association trunk 204 205



SW2 SW2(config)# feature private-vlan SW2(config)# vtp mode transparant SW2(config)# vlan 205 SW2(config-vlan)# private-vlan isolated SW2(config-vlan)# exit SW2(config)# int e1/1 SW2(config-if)# sw mode private-vlan trunk secondary SW2(config-if)# sw private-vlan trunk allowed vlan 205

When configuring private VLAN trunks you don’t need to change the normal trunk allowed vlan command, as the private VLANs on the private-vlan trunk allowed list are automatically added to the normal VLAN allowed list. On SW2 the same configuration is required to transfer the secondary VLAN.

As SW2 does not know VLAN 204 (the Primary VLAN) it’s not necessary to create the association, because the mapping for this is done on SW1-1.

Next are the hosts that need to be configured in the newly created secondary VLAN on SW2. Remember that VTP is disabled for this use so you need to manually create the private VLAN on SW2. The host association does need to be made to ensure the interface is a member of VLAN 205.

SW2(config)# int e1/21-23 SW2(config-if-range)# switchport SW2(config-if-range)# sw mode private-vlan host SW2(config-if-range)# sw private-vlan host-association 204 205 SW2(config-if-range)# no shut

Final task is that a second trunk link needs to be enabled to extend the current VLANs. You are required to use a different trunk link for this use. There are 2 ways to solve this case. The easiest way is to just extend the secondary VLANs, just like we did in the previous assignment. The other solution is to extend the primary private VLAN to SW2 and create a new VLAN 201 and 202 to connect those special hosts. The problem which is created then is that the community VLAN on SW2 is not the same as the community VLAN on SW1, which means that your connectivity is broken and you are not completing the requirements of this task successfully.

SW1-1

SW1-1(config)# int e3/2 SW1-1(config-if)# sw mode private-vlan trunk secondary SW1-1(config-if)# sw private-vlan trunk allowed vlan 201-202 SW1-1(config-if)# sw privat association trunk 200 201 SW1-1(config-if)# sw privat association trunk 200 202 SW2 SW2(config)# vlan 201 SW2(config-vlan)# private-vlan isolated SW2(config-vlan)# exit



SW2(config)# vlan 202 SW2(config-vlan)# private-vlan community SW2(config-vlan)# exit SW2(config)# int e1/2 SW2(config-if)# sw mode private-vlan trunk secondary SW2(config-if)# sw private-vlan trunk allowed vlan 201-202 SW2(config)# int e1/24-25 SW2(config-if-range)# switchport SW2(config-if-range)# sw mode private-vlan host SW2(config-if-range)# sw private-vlan host-association 200 201 SW2(config-if-range)# no shut SW2(config)# int e1/26 SW2(config-if-range)# switchport SW2(config-if-range)# sw mode private-vlan host SW2(config-if-range)# sw private-vlan host-association 200 202 SW2(config-if-range)# no shut

Now you have successfully completed task 3!

Task 4: Implement Rapid Spanning-Tree protocol The Rapid Per-VLAN Spanning-Tree protocol is by default enabled on NX-OS systems. This implementation is a Cisco proprietary method of working with Rapid Spanning-Tree (IEEE 802.1w). Classical spanning-tree is no longer supported. Still Rapid-PVST+ (Cisco’s implementation) is supporting a backwards compatibility with classic STP.

One important difference with classical STP is that Rapid-PVST+ assumes that network (core) links are configured point-to-point without any hops in between (hubs). This assumption ensures a much quicker convergence time and is interrupt driven instead of timer driven like in classical STP.

The spanning-tree port type is an important command which you should configure wisely! It can help you convergence both for host and network links a lot when this is properly configured. When host ports are configured as Edge ports they are also not able to start a Topology Change (TC) towards the network which initiates spanning-tree convergence. The flap of an edge interface therefore does not start a topology change in the network. Keeping the spanning-tree network much more stable!

The first task tells you to configure all host connecting ports should be configured as ‘edge’ ports so they aren’t generating topology changes. Do this by using a range command will ease up your life and save you precious CCIE lab time.

SW2 SW2(config)# interface e1/10-15 SW2(config-if-range)# spanning-tree port type ? edge Consider the interface as edge port (enable portfast) network Consider the interface as inter-switch link normal Consider the interface as normal spanning tree port



SW2(config-if-range)# spanning-tree port type edge Warning: edge port type (portfast) should only be enabled on ports connected to a single

host. Connecting hubs, concentrators, switches, bridges, etc... to this interface when edge port type (portfast) is enabled, can cause temporary bridging loops.

Use with CAUTION Edge Port Type (Portfast) will be configured in 6 interfaces due to the range command

but will only have effect when the interfaces are in a non-trunking mode. SW2(config-if-range)# SW3 SW3(config)# interface e1/10-15 SW3(config-if-range)# spanning-tree port type edge SW3(config-if-range)#

The command warns you that it will only have effect when the ports are configures in switchport mode access plus that spanning-tree portfast is enabled so interfaces will come up fast when hosts are connected.

The second assignment requires you to configure a root configuration for the spanning-tree instance in VLAN 101. There are 2 ways to do this, but there is no indication which one needs to be used. Then use whatever you think is your favorite way to configure this.

The first way is by configuring manual priority values. Choose low values for this, just as a best practice. There are fixed values that you are able to use for configuring the spanning-tree priority. Don’t mind if you don’t reminder these from the top of your head. When you type it in wrong, the CLI will automatically tell you which values are possible to fill-in.

SW2(config)# spanning-tree vlan 101 priority 4093 ERROR: % Bridge priority must be in increments of 4096 Allowed values are: 0 4096 8192 12288 16384 20480 24576 28672 32768 36864 40960 45056 49152 53248 57344 61440 SW2(config)# spanning-tree vlan 101 priority 4096 SW2(config)#

The next, and currently preferred, way is by enabling the root functionality by using the dedicated commands for this. This means that the switch will configure a dedicated priority for both the root and backup root bridge. The default root priority value is 25476. When there is already a bridge on the network with a lower priority than this (meaning it’s the current spanning-tree root bridge) the switch will choose a priority value of 4096 lower than the current lowest priority value in the network, so when using this command the switch will always become the root bridge. Of course this is not



maintained and when a new switch with a lower value comes online, that will become the root bridge. This is a single configuration task so at the moment you enter the command, the switch will become the root.

For the backup root bridge this fixed value is 28672 without a fallback mechanism to ensure it’s always the backup root. The switch can’t determine the backup root bridge priority value as this is unknown on the local switch.

The next assignment tells you to ensure that the timers of the Rapid-PVST+ process for VLAN 101. You can easily configure lower timers for this, but you will you know what the best timer values for this network are? Well that’s quite difficult. Therefore NX-OS has an automatic way of configuring this by using the special ‘diameter’ command. This diameter command automatically calculates the best timer values based on the maximum amount of hops between 2 hosts of the network. As our network consists of 3 layer 2 switches, so the diameter should be configured on 3.

SW2 SW2(config)# spanning-tree vlan 101 root primary diameter 3 SW3 SW3(config)# spanning-tree vlan 101 root secondary diameter 3

For assignment 4 and 5 of this task it’s required to read these together, otherwise you might be configuring the same thing twice, which will cost you time. The task is that SW1 is the root bridge for VLAN 102. Together with that, any new (unknown) switch may become the root bridge of the network. Of course you will never know what the pre-configuration of that new switch might look like, but we can take some assumptions here and ensure that the priority values are as low as possible. The key point in assignment 5 is that the switch will be configured with default spanning-tree priority values. The default spanning-tree priority value is 32768, which means we need to configure our network for values lower than that and ensuring SW1 will be the primary root bridge for VLAN 102.

SW1 SW1-1(config)# spanning-tree vlan 101 priority 8192 SW2 SW2(config)# spanning-tree vlan 101 priority 16384 SW3 SW3(config)# spanning-tree vlan 101 priority 16384

The next assignment is about steering traffic within the spanning-tree network. This means that we are going to change spanning-tree costs to alter the layer 2 path through the network and influence which link is going to be blocked. By default Rapid-PVST+ on NX-OS is using the so called short path-cost



method. This means that a 16-bit number is used to determine the cost. You can change this so that a 32-bit number is used to calculate the cost.

Then links can be manually changed what cost is used. On trunk links this can be changed per VLAN. By default all ports are set to auto. This means that according to the link bandwidth a specific cost is attached. Depending on if short or long path-cost method is configured.

Bandwidth Short path-cost method Long path-cost method

10 Mbps 100 2,000,000

100 Mbps 19 200,000

1 Gigabit Ethernet 4 20,000

10 Gigabit Ethernet 2 2,000

The table above shows which cost is given to which links dependent on the configured or negotiated bandwidth.

With a 16-bit number values of 1 up to 65,535 can be configured. The assignment requires you to configure numbers of over 100,000. This means that all switches need to be configured with a long path-cost method.

SW1 SW1-1(config)# spanning-tree pathcost method long SW2 SW2(config)# spanning-tree pathcost method long SW3 SW3(config)# spanning-tree pathcost method long

The next task is the true traffic steering / engineering assignment, where you are asked to ensure that SW1 is always using SW3 to reach the root bridge of VLAN 101. We just configured SW2 to be the root bridge of VLAN 101.



There are a number of things to take a look at in this position. First is that spanning-tree will always use the shortest path towards the root bridge as primary path. Meaning that it will elect its root port on that path. Because we have a loop in our network, there will be links in a blocking state. In the current design when we take a look at root ports and costs. The links between SW1 and SW3 will become blocking as the following drawing illustrates.

Now we need to change this behavior as we want all traffic to flow through SW3. This means that the links between SW1 and SW2 need to be on blocking, as we do need a path to the root switch without any loops. There are various ways in which you are able to achieve the same goal, but the easiest way in this case is by raising the cost of the links from SW1 to SW2. All these links are are 10 Gigabit Ethernet connections so the default cost of all links in the network is 2,000. We are required to change the cost with values higher than 100,000. So let’s change them to 4,000,000 to ensure that no matter what kind of connection is made between those switches, the default path cost method will never elect it as a better path than the path between SW1, through SW3 to SW2.



The drawing above illustrates the changes and how traffic will flow towards the root bridge. The links between SW1 and SW2 are now in blocking state to prevent loops in the network and we accomplished the task!

SW1-1(config)# interface Ethernet3/1-4 SW1-1(config-if-range)# spanning-tree cost 4000000

The next assignment requires you that for all inter-switch-links the fourth link is used as the primary link. All other links will become blocking. Cost is not an option, because the cost is based on bandwidth and not on a port level. Spanning-tree knows another config knob to use in the case where there are multiple equal bandwidth links between bridges like in our network. In normal networks you would use port-channeling for this features, but in this case we have multiple ports, of which most will become blocking. Either way we want to ensure that the fourth link will become primary.

This is done using the port-priority feature. As a tie-breaker when default port-priority of 128 is used the port number is used. So in this case the lowest number wins, which is never the fourth link in our case. We want to change the port-priority on the fourth link to a lower value than the default of 128. Be aware that the port-priority can only be configured with increments of 32. Again just like the spanning-tree root priority, the CLI will tell you which values are available for use in this case.

We will use priority 64 (any number lower than 128 is good enough) and place it on all ‘4th’ links between the 3 switches in the topology.

SW1 SW1-1(config)# int e3/4 SW1-1(config-if)# spanning-tree port-priority ? <0-224> Port priority in increments of 32 SW1-1(config-if)# spanning-tree port-priority 100 ERROR: % Port Priority in increments of 32 is required



Allowed values are: 0 32 64 96 128 160 192 224 SW1-1(config-if)# spanning-tree port-priority 64 SW1-1(config-if)# interface e3/8 SW1-1(config-if)# spanning-tree port-priority 64 SW1-1(config-if)# SW2 SW2(config-if)# interface e1/4 SW2(config-if)# spanning-tree port-priority 64 SW2(config-if)# interface e1/8 SW2(config-if)# spanning-tree port-priority 64 SW2(config-if)# SW3 SW3(config-if)# interface e1/4 SW3(config-if)# spanning-tree port-priority 64 SW3(config-if)# interface e1/8 SW3(config-if)# spanning-tree port-priority 64 SW3(config-if)#

The next assignment asks you to configure the lowest convergence times for VLAN 103. Now we can’t use the diameter command as that will not configure the lowest times. In this case the solution is simple. Configure all spanning-tree timers to their lowest value possible.

SW1 SW1-1(config)# spanning-tree vlan 103 ? <CR> , Multi range separator - Range separator forward-time Set the forward delay for the spanning tree hello-time Set the hello interval for the spanning tree max-age Set the max age interval for the spanning tree priority Set the bridge priority for the spanning tree root Configure switch as root SW1-1(config)# spanning-tree vlan 103 hello-time ? <1-10> Number of seconds between generation of config bpdu SW1-1(config)# spanning-tree vlan 103 hello-time 1 SW1-1(config)# spanning-tree vlan 103 forward-time ? <4-30> Number of seconds for the forward delay timer SW1-1(config)# spanning-tree vlan 103 forward-time 4 SW1-1(config)# spanning-tree vlan 103 max-age ? <6-40> Maximum number of seconds the information in a bpdu is valid SW1-1(config)# spanning-tree vlan 103 max-age 6



SW2 SW2(config)# spanning-tree vlan 103 hello-time 1 SW2(config)# spanning-tree vlan 103 forward-time 4 SW2(config)# spanning-tree vlan 103 max-age 6 SW3 SW3(config)# spanning-tree vlan 103 hello-time 1 SW3(config)# spanning-tree vlan 103 forward-time 4 SW3(config)# spanning-tree vlan 103 max-age 6

The next assignment is a little bit cryptical. This is mainly done to test your skills of finding examples in the documentation very fast. When you look up the layer 2 documentation of NX-OS and search for ‘Rapid Connectivity’ there is only 1 hit. This is also the technology making Rapid-PVST+ really rapid. As it will assume a point-to-point link infrastructure instead of a shared infrastructure (multiple layer 2 bridges on a single segment).

The answer is quite simple. Make every inter switch link a point-to-point link.

SW1 SW1-1(config)# int e3/1-8 SW1-1(config-if-range)# spanning-tree link-type point-to-point SW2 SW2(config)# int e1/1-8 SW2(config-if-range)# spanning-tree link-type point-to-point SW3 SW3(config)# int e1/1-8 SW3(config-if-range)# spanning-tree link-type point-to-point

The final assignment requires you to clear all spanning-tree configuration from the switches so you can make a clean slate before moving on to the next section of this chapter

Task 5: Implement Multiple Spanning-Tree protocol The Multiple Spanning-Tree protocol is much alike the Rapid Spanning-Tree protocol as tested in the previous task. The main difference lies in the fact that it’s not VLAN or single instance based, but on multiple instances where you can configure multiple VLANS to belong to a specific instance or MSTI. These MSTIs / instances are independent from each other topology wise and should not interfere with each other once a convergence occurs in the MSTI.

This causes a much higher scale as the control-plane overhead of this protocol is much less than the Rapid-PVST+ protocol used in the previous task. However it has some downsides as that configuration of the VLAN to instance mapping needs to be equal on all switches otherwise you will get serious outages on your network. This requires good thinking about which VLAN is added where.



You will see that most assignments in this task are equal to the Rapid-PVST+ task, but require different thinking, where you need to apply instances instead of VLANs.

First assignment asks you to configure MST (IEEE 802.1s) on all 3 switches. Then you need to configure some parameters and ensure that the MST is working on all switches. What this means is that all configuration like name, revision number and VLAN to instance mapping needs to be equal on all switches in order for MST to work properly. Otherwise you will run into issues in where some traffic might even be blocked.

SW1 SW1-1(config)# spanning-tree mode mst SW1-1(config)# spanning-tree mst configuration SW1-1(config-mst)# ? abort Exit region configuration mode, aborting changes instance Map vlans to an MST instance name Set configuration name no Negate a command or set its defaults private-vlan Set private-vlan synchronization revision Set configuration revision number show Display region configurations end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1-1(config-mst)# name IPexpert SW1-1(config-mst)# revision 5 SW1-1(config-mst)# instance 1 vlan 10-99 SW1-1(config-mst)# instance 2 vlan 100-199 SW1-1(config-mst)# instance 3 vlan 800-1299 SW2 SW2(config)# spanning-tree mode mst SW2(config)# spanning-tree mst configuration SW2(config-mst)# name IPexpert SW2(config-mst)# revision 5 SW2(config-mst)# instance 1 vlan 10-99 SW2(config-mst)# instance 2 vlan 100-199 SW2(config-mst)# instance 3 vlan 800-1299 SW3 SW3(config)# spanning-tree mode mst SW3(config)# spanning-tree mst configuration SW3(config-mst)# name IPexpert SW3(config-mst)# revision 5 SW3(config-mst)# instance 1 vlan 10-99 SW3(config-mst)# instance 2 vlan 100-199 SW3(config-mst)# instance 3 vlan 800-1299



The next assignment requires you to use your imagination or check the documentation really quick. There is a specific functionality under the same ‘spanning-tree mst configuration’ configuration knob where you can configure synchronization of private-vlans. This means that no matter what number the secondary VLANs have, they are always mapped to the instance of the primary VLAN to ensure an equal layer 2 topology for all private VLANs contained under 1 primary VLAN.

SW1 SW1-1(config)# spanning-tree mst configuration SW1-1(config-mst)# private-vlan synchronize SW2 SW2(config)# spanning-tree mst configuration SW2(config-mst)# private-vlan synchronize SW3 SW3(config)# spanning-tree mst configuration SW3(config-mst)# private-vlan synchronize

Verify that all configurations were done properly. Be aware that accept the VLANs that you configured. All other VLANs (even when not created) are still mapped to instance 0. Therefore you should always be aware of that (unconfigured) instance 0 when designing you spanning-tree topology.

SW1-1(config)# sh spanning-tree mst configuration Name [IPexpert] Revision 5 Instances configured 4 Instance Vlans mapped -------- ---------------------------------------------------------------------

0 1-9,200-799,1300-4094 1 10-99 2 100-199 3 800-1299 -------------------------------------------------------------------------------

The next assignment lets you configure SW2 to be the primary root bridge for instance 1. You should use the lowest possible value, which is 0.

SW1-1(config)# spanning-tree mst 0 prio 0 SW1-1(config)# sh span mst ##### MST0 vlans mapped: 1-9,200-799,1300-4094 Bridge address 0050.563b.01b1 priority 32768 (32768 sysid 0) Root this switch for the CIST Regional Root this switch Operational hello time 2 , forward delay 15, max age 20, txholdcount 6 Configured hello time 2 , forward delay 15, max age 20, max hops 20 Interface Role Sts Cost Prio.Nbr Type



---------------- ---- --- --------- -------- --------------------------------

Eth2/1 Desg FWD 20000 128.257 P2p ##### MST1 vlans mapped: 10-99 Bridge address 0050.563b.01b1 priority 1 (0 sysid 1) Root this switch for MST1 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- --------------------------------

Eth2/1 Desg FWD 20000 128.257 P2p ##### MST2 vlans mapped: 100-199 Bridge address 0050.563b.01b1 priority 32770 (32768 sysid 2) Root this switch for MST2 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- --------------------------------

Eth2/1 Desg FWD 20000 128.257 P2p

From SW2 it’s clearly visible that SW1-1 is the root bridge for instance 1.

SW2(config)# sh span mst ##### MST0 vlans mapped: 1-9,200-799,1300-4094 Bridge address 0050.563b.01b9 priority 32768 (32768 sysid 0) Root address 0050.563b.01b1 priority 32768 (32768 sysid 0) port Eth2/1 path cost 0 Regional Root address 0050.563b.01b1 priority 32768 (32768 sysid 0) internal cost 20000 rem hops 19 Operational hello time 2 , forward delay 15, max age 20, txholdcount 6 Configured hello time 2 , forward delay 15, max age 20, max hops 20 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- --------------------------------

Eth2/1 Root FWD 20000 128.257 P2p ##### MST1 vlans mapped: 10-99 Bridge address 0050.563b.01b9 priority 32769 (32768 sysid 1) Root address 0050.563b.01b1 priority 1 (0 sysid 1) port Eth2/1 cost 20000 rem hops 19 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- --------------------------------

Eth2/1 Root FWD 20000 128.257 P2p ##### MST2 vlans mapped: 100-199



Bridge address 0050.563b.01b9 priority 32770 (32768 sysid 2) Root address 0050.563b.01b1 priority 32770 (32768 sysid 2) port Eth2/1 cost 20000 rem hops 19 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- --------------------------------

Eth2/1 Root FWD 20000 128.257 P2p

The next assignment asks you to try and configure SW3 to be the root bridge for the MST instance 1 with use of the special root priority command. Why does this fail? Well that’s because what was explained in the Rapid-PVST+ section. The command will (when it sees a lower priority root bridge currently online) to lower the root priority with 4096 to ensure this switch becomes the root. The problem is that SW2 is already using the lowest possible value. So this is not possible and the command will be rejected.

SW3(config-if)# spanning-tree mst 1 root primary ERROR: Failed to set root bridge for MST 1 % It may be possible to make the bridge root by setting the priority % for some (or all) of these instances to zero. SW3(config)# SW3(config)# sh span mst ##### MST0 vlans mapped: 1-9,200-799,1300-4094 Bridge address 0050.563b.01c1 priority 32768 (32768 sysid 0) Root address 0050.563b.01b1 priority 32768 (32768 sysid 0) port Eth2/2 path cost 0 Regional Root address 0050.563b.01b1 priority 32768 (32768 sysid 0) internal cost 20000 rem hops 19 Operational hello time 2 , forward delay 15, max age 20, txholdcount 6 Configured hello time 2 , forward delay 15, max age 20, max hops 20 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- --------------------------------

Eth2/1 Altn BLK 20000 128.257 P2p Eth2/2 Root FWD 20000 128.258 P2p ##### MST1 vlans mapped: 10-99 Bridge address 0050.563b.01c1 priority 32769 (32768 sysid 1) Root address 0050.563b.01b1 priority 1 (0 sysid 1) port Eth2/2 cost 20000 rem hops 19 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- --------------------------------

Eth2/1 Altn BLK 20000 128.257 P2p Eth2/2 Root FWD 20000 128.258 P2p ##### MST2 vlans mapped: 100-199 Bridge address 0050.563b.01c1 priority 32770 (32768 sysid 2) Root address 0050.563b.01b1 priority 32770 (32768 sysid 2)



port Eth2/2 cost 20000 rem hops 19 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- --------------------------------

Eth2/1 Altn BLK 20000 128.257 P2p Eth2/2 Root FWD 20000 128.258 P2p

Now the next task does not allow configuration on the switch itself which needs to be made backup root bridge. This means we have to use the default configuration of spanning-tree. The default priority value is 32,768. This means that when we increase the priority value of SW1 we are sure that SW3 will be elected the new root bridge once SW2 fails.

SW1-1(config)# spanning-tree mst 1 prio 40960

Again we are asked to configure optimal network timers for this topology. The topology is still similar to the one as used in the Rapid-PVST+ chapter, so we can use the diameter command here again! Pay attention that changing other instances than instance 0 is possible for the timers or diameter command, but this is useless as the timers can only be configured once. Timers are always applied for all instances at the same time. Therefore the diameter command only works when configured for instance 0.

SW2(config)# spanning-tree mst 0 root primary diameter 3 SW2(config-if)# sh span mst detail ##### MST0 vlans mapped: 1-9,200-799,1300-4094 Bridge address 0050.563b.01b1 priority 24576 (24576 sysid 0) Root this switch for the CIST Regional Root this switch Operational hello time 2 , forward delay 9 , max age 12, txholdcount 6 Configured hello time 2 , forward delay 9 , max age 12, max hops 20 <output omitted>

Again we are required to use values over 100,000 when using traffic engineering / steering. So this means that all switches need to be configured with a long path-cost method.

SW1 SW1-1(config)# spanning-tree pathcost method long SW2 SW2(config)# spanning-tree pathcost method long SW3 SW3(config)# spanning-tree pathcost method long



The traffic steering question is similar to the task before, but instead of steering a VLAN, a whole instance can be steered. The command is slightly different. Again we use a cost of 4,000,000.

SW1-1(config)# interface Ethernet3/1-4 SW1-1(config-if-range)# spanning-tree mst 2 cost 4000000

The next assignment requires a little bit of thinking. Like the previous task we were also asked to configure port-priority to determine which local interface between switches will be used as the primary one. Now with MST this is configurable per instance. As we have 4 instances we want to load share among all interfaces. Configuring the same lower port-priority per instance on different interfaces does the trick.

SW1 SW1-1(config)# int e3/1 SW1-1(config-if)# spanning-tree mst 0 port-priority 64 SW1-1(config)# int e3/2 SW1-1(config-if)# spanning-tree mst 1 port-priority 64 SW1-1(config)# int e3/3 SW1-1(config-if)# spanning-tree mst 2 port-priority 64 SW1-1(config)# int e3/4 SW1-1(config-if)# spanning-tree mst 3 port-priority 64 SW1-1(config)# int e3/5 SW1-1(config-if)# spanning-tree mst 0 port-priority 64 SW1-1(config)# int e3/6 SW1-1(config-if)# spanning-tree mst 1 port-priority 64 SW1-1(config)# int e3/7 SW1-1(config-if)# spanning-tree mst 2 port-priority 64 SW1-1(config)# int e3/8 SW1-1(config-if)# spanning-tree mst 3 port-priority 64 SW2 SW2(config)# int e1/1 SW2(config-if)# spanning-tree mst 0 port-priority 64 SW2(config)# int e1/2 SW2(config-if)# spanning-tree mst 1 port-priority 64 SW2(config)# int e1/3 SW2(config-if)# spanning-tree mst 2 port-priority 64 SW2(config)# int e1/4 SW2(config-if)# spanning-tree mst 3 port-priority 64 SW2(config)# int e1/5 SW2(config-if)# spanning-tree mst 0 port-priority 64 SW2(config)# int e1/6 SW2(config-if)# spanning-tree mst 1 port-priority 64 SW2(config)# int e1/7 SW2(config-if)# spanning-tree mst 2 port-priority 64 SW2(config)# int e1/8 SW2(config-if)# spanning-tree mst 3 port-priority 64



SW3 SW3(config)# int e1/1 SW3(config-if)# spanning-tree mst 0 port-priority 64 SW3(config)# int e1/2 SW3(config-if)# spanning-tree mst 1 port-priority 64 SW3(config)# int e1/3 SW3(config-if)# spanning-tree mst 2 port-priority 64 SW3(config)# int e1/4 SW3(config-if)# spanning-tree mst 3 port-priority 64 SW3(config)# int e1/5 SW3(config-if)# spanning-tree mst 0 port-priority 64 SW3(config)# int e1/6 SW3(config-if)# spanning-tree mst 1 port-priority 64 SW3(config)# int e1/7 SW3(config-if)# spanning-tree mst 2 port-priority 64 SW3(config)# int e1/8 SW3(config-if)# spanning-tree mst 3 port-priority 64

The final 2 assignments require a little digging in the documentation again. These are exotic features that are easily found when you have the documentation open.

The first asks you to configure the spanning-tree max-hops feature.

SW1 SW1-1(config)# spanning-tree mst max-hops 10

SW2 SW2(config)# spanning-tree mst max-hops 10 SW3 SW3(config)# spanning-tree mst max-hops 10

The final in this task is to enable a pre-standard version of MST on a specific interface. Something rarely seen in production, but it’s possible in this version of software so it’s better that you are aware that it’s possible.

SW2(config)# interface e1/16 SW2(config)# spanning-tree mst pre-standard

You have successfully completed the MST task!



Task 6: Spanning-Tree and UDLD features Some miscellaneous features remain in the spanning-tree section and on the CCIE Data Center blueprint they specifically mentioned UDLD, which is why this is incorporated in this chapter.

With these kinds of features it’s highly recommended to use the documentation. As most of the times the question, or specific keywords in it, are already mentioned somewhere in the documentation. Then they are easily spotted and configured in a few minutes.

So opening up the Cisco Documentation for layer 2 features is highly recommended as that makes searching a lot faster.

The first assignment points you to the direction of spanning-tree port types. As mentioned in the spanning-tree chapters this is very relevant for your spanning-tree design. When ports are configured for Edge functionality they will never trigger a topology change in the network. Within the chapter you were asked to configure certain ports. It’s also possible to configure all unconfigured ports for a dedicated use.

SW3(config)# spanning-tree port type edge default

Next are the special spanning-tree features. The first asks to ensure a port is shut down when BPDUs are received. This is directly related to the BPDU guard feature.

SW3(config)# interface e1/10 SW3(config)# spanning-tree bpduguard enable

The next feature filters BPDUs from edge interfaces or trunk interfaces. Be aware when you configure this on your production network as this might cause bridging loops within your network!

SW3(config)# interface e1/11 SW3(config)# spanning-tree bpdufilter enable

The next assignment asks you to configure the BPDU filtering on an interface on SW2. The problem is that we can’t configure the command directly under the interface. This means that we will be configuring the feature globally for every interface (unconfigured). One thing is that we will be configuring all interfaces, but including Ethernet 1/10 and that means we accomplished this task.

SW2(config)# spanning-tree port type edge bpdufilter default



The question 5 relates to the root guard feature. Usually configured on edge ports. Meaning that it will protect you against that new root bridges are connected to that particular interface. The port will go in err-disable state when this occurs.

Before the root guard feature works, you first need to re-enable BPDU processing on the interface, otherwise the superior BPDU’s are already caught in the BPDU filter and never processed. The spanning-tree bpdufilter disable command will not work in this case as.

SW2(config)# interface e1/11 SW2(config)# no spanning-tree bpdufilter SW2(config)# spanning-tree guard root

The next task is also a guard feature. Instead of making the port a root port. You can also guard a port for becoming a designated port. Again we first need to disable the BPDU filter feature, before the loop guard feature will work.

SW2(config)# interface e1/12 SW2(config)# no spanning-tree bpdufilter SW2(config)# spanning-tree guard loop

The next feature is a special case feature where you would always need the CLI help or the Cisco Documentation. The command disables the Rapid-PVST+ interoperability when MST is enabled.

SW3(config)# interface e1/13 SW3(config)# spanning-tree mst simulate pvst disable

The final 2 questions in this task point you in the direction UDLD or Unidirectional Link Detection. This is a Cisco proprietary protocol where it becomes possible to detect a single sided link failure, both on fiber optic and copper cables. The drawing below illustrates what is meant with this.



The protocol can be enabled in 2 ways. Normal or Aggressive mode. The aggressive mode has the advantage of faster timers and this ensures that traffic is being discarded.

SW3(config)# int e3/12 SW3(config-if)# udld ? ^ % Invalid command at '^' marker. SW3(config-if)# exit

Even this tiny feature needs to be enabled first!

SW3(config)# feature udld SW3(config)# int e1/12 SW3(config-if)# udld ? aggressive Enable UDLD aggressive mode for interface(s) disable Disable UDLD for fiber interface(s) enable Enable UDLD for non-fiber interface(s) SW3(config-if)# udld aggressive SW3(config-if)#

And you successfully completed this task!

Task 7: Fabric Extenders The Fabric Extender features allows you to install so-called ‘remote line-cards’ to either the Nexus 7000 or the Nexus 5000 series switches. There are various ways to configure them, which will be tested in other labs of our CCIE Data Center workbook set. The configuration however is always the same.

The architecture is so that the FEX (short for Fabric EXtender) is configured through the parent switch which can be either an Nexus 7000 or an Nexus 5000 switch, but not mixed at the same time. It’s possible with Virtual Port Channels (vPC), which is tested in a later chapter, to connect a FEX to 2 parent switches and still manage them as a remote line card. However this configuration is tested in the vPC related chapter.

Current we will be configuring the FEX only to SW2 so we get the following architecture in the network.



The configuration for the fabric extenders is relatively simple. This task requires you to match the output. What we can read from the output and the questions is that we need to configure the FEX with a special name, we need to turn on the beacon LED on the FEX and we need to assign the FEX number of 101 to this particular FEX.

The FEX number is also the linecard number which you will be configuring the host interfaces with. The FEX number (or chassis ID) needs to be between 100 and 199 and can be pre-staged. So when connecting certain FEX switches to the Nexus 7000 or 5000 they will automatically get the right FEX number associated. This is based on the FEX serial number. When no staging is done, the FEX number is associated to the FEX when it’s detected on the interface when it’s enabled.

What’s also shown in the output is that the FEX is connected through port-channel 4. Now port-channeling will be tested in detail in a later chapter. The purpose in this task is to build a port-channel where the FEX is connected to. You can do this by just configuring 1 interface in a port-channel and let the FEX connect to that. Which in that case turns out that the FEX is connected to that port-channel.

The configuration for these tasks are:

SW2(config)# feature fex SW2(config)# fex 101 SW2(config-fex)# beacon SW2(config-fex)# description "IPexpert Fabric Extender 1" SW2(config-fex)# exit SW2(config)# SW2(config)# interface e1/24 SW2(config-if)# channel-group 4 mode on



SW2(config-if)# no shutdown SW2(config-if)# exit SW2(config)# SW2(config)# interface port-channel 4 SW2(config-if)# switchport mode fex-fabric SW2(config-if)# fex associate 101 SW2(config-if)# no shutdown

Task 7 is completed



Task 8: Misc features The final task of this chapter will let you configure some miscellaneous features of the NX-OS platforms. These features are pretty basic features, but will challenge you to read through the whole section first before starting, otherwise you might end up with re-doing some configuration which will cost you time. The feature that is asked for in the final bullet requires you to think in a NX-OS centric way. Meaning you have to use so-called port-profiles to configure these features.

The port-profiles are typically used in the Nexus 1000V where virtual ethernet interfaces are created once VMs come online, so the configuration is always based on a template. Within the other Nexus platforms it’s also possible to configure these templates for any kind of interface, both physical and virtual. Then you apply this template to one or more physical / virtual interfaces of the kind of port-profile that you made.

This can save you a lot of work in a production environment too. The configuration is straight forward.

The configuration tasks are based on miscellaneous features that can be enabled on interfaces. First is flowcontrol, which can be activated both in send and receive direction. The straight/cross cable detection only works on copper interfaces and is called Auto MDIX. Finally the usage statistics on show interface are configured using the load-interval command. Since NX-OS you can configure up to 3 counters to be more granular in the averages that are measured.

SW1-1(config)# port-profile type ? ethernet Ethernet type interface-vlan Interface-vlan type port-channel Port-channel type

As previously mentioned, it’s possible to create different types of port-profiles dependent on the type of interface that you are configuring with it.

SW1-1(config)# port-profile type ethernet IPEXPERT SW1-1(config-port-prof)# switchport SW1-1(config-port-prof)# swi mode trunk SW1-1(config-port-prof)# sw trunk allowed vlan 101-104 SW1-1(config-port-prof)# flowcontrol receive on SW1-1(config-port-prof)# no mdix auto

Next up is configuring the features under the port profile. The port-profile is configured like a regular interface with the same options.

The load-interval is different from IOS. When the command is configured without specifying the counter it’s configuring the normal IOS counter, where only 1 average delay timer can be configured. In NX-OS you can configure up to 3 counters to measure different averages. Counter 1 is the default counter and numbers 2 and 3 are specifically mentioned in the show interface output.

SW1-1(config-port-prof)# load-interval ? <30-300> Load interval delay in seconds



counter Specify counter for this load interval SW1-1(config-port-prof)# load-interval counter ? <1-3> Specify counter for this load interval SW1-1(config-port-prof)# load-interval counter 1 ? <30-300> Load interval delay in seconds SW1-1(config-port-prof)# load-interval counter 1 30 SW1-1(config-port-prof)# load-interval counter 2 60 SW1-1(config-port-prof)# load-interval counter 3 120 SW1-1(config-port-prof)# state enabled SW1-1(config-port-prof)# no shut

One important note for port-profiles is that you should never forget the state enabled command. Without this command the port-profile will never be active or will be enabled, so interfaces will not come up with the right configuration without this config knob.

SW1-1(config-port-prof)# SW1-1(config-port-prof)# sh run port-profile !Command: show running-config port-profile !Time: Tue Sep 11 02:57:34 2012 version 5.1(5) port-profile type ethernet IPEXPERT switchport switchport mode trunk switchport trunk allowed vlan 101-104 flowcontrol receive on no mdix auto load-interval counter 1 30 load-interval counter 2 60 load-interval counter 3 120 no shutdown state enabled SW1-1(config-if-range)# int e5/16-18 SW1-1(config-if-range)# inherit port-profile IPEXPERT SW1-1(config-if-range)# SW1-1(config-if-range)# sh run int e5/16 !Command: show running-config interface Ethernet5/16 !Time: Tue Sep 11 02:58:52 2012 version 5.1(5) interface Ethernet5/16 inherit port-profile IPEXPERT



After applying the port-profile to the interfaces, meaning that the configuration is only visible once in the configuration, the interface configuration only shows the application of the port-profile to the interface.

SW1-1(config-if-range)# sh int e5/16 Ethernet5/16 is down (Link not connected) Hardware: 10/100/1000 Ethernet, address: e05f.b930.3a2b (bia e05f.b930.3a2b)

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA Port mode is trunk auto-duplex, auto-speed Beacon is turned off Auto-Negotiation is turned on Input flow-control is on, output flow-control is off Auto-mdix is turned off Switchport monitor is off EtherType is 0x8100 Last link flapped never Last clearing of "show interface" counters never 30 seconds input rate 0 bits/sec, 0 packets/sec 30 seconds output rate 0 bits/sec, 0 packets/sec Load-Interval #2: 1 minute (60 seconds) input rate 0 bps, 0 pps; output rate 0 bps, 0 pps Load-Interval #3: 2 minute (120 seconds) input rate 0 bps, 0 pps; output rate 0 bps, 0 pps RX 0 unicast packets 0 multicast packets 0 broadcast packets 0 input packets 0 bytes 0 jumbo packets 0 storm suppression packets 0 runts 0 giants 0 CRC 0 no buffer 0 input error 0 short frame 0 overrun 0 underrun 0 ignored 0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop 0 input with dribble 0 input discard 0 Rx pause TX 0 unicast packets 0 multicast packets 0 broadcast packets 0 output packets 0 bytes 0 jumbo packets 0 output error 0 collision 0 deferred 0 late collision 0 lost carrier 0 no carrier 0 babble 0 output discard 0 Tx pause 0 interface resets

When verifying the interface. All configuration from the port profile is visible, like the Auto MDIX, the 3 load interval counters and the layer 2 trunk mode.

Now you have successfully completed task 8 and chapter 1 of this workbook. Take a little break before continuing with configuring the layer 3 routing features!



Chapter 3: Data Center Networking Layer 3 Infrastructure (NX-OS)

Data Center Networking Layer 3 Infrastructure is intended to let you be familiar with the NX-OS Layer 3 features on the Nexus platforms to create a basic routed network. We highly recommend creating your own diagram at the beginning of each lab so you are able to draw on your own diagram, making it much easier when you step into the real lab. The lab is divided in two pieces. During the first tasks you will be configuring a dynamically routed layer 3 network using EIGRP and OSPF protocols. The second part of this chapter is based on the Cisco proprietary technologies FabricPath and OTV. Multiple topology drawings are available for this chapter.



Solutions

Task 1: Layer 3 topology set-up The first task is based on drawing 1 as illustrated below. The assignments in this task require you to configure the network in a very self-supporting way. You get to choose which layer 2 trunks you are going to configure and you have the responsibility to configure the VLANs in a proper way. Are you going to do it using VTP or not? It’s all up to you, what you think is best to solve this task. During the real exam, you will find more of these questions where you are free to choose your kind of solution, as long as it complies with the rules set in the task.

During the initial configuration there are VDC’s configured on the Nexus 7000 switch (SW1-1). This configuration is then used throughout the first couple tasks of this chapter. You can log-in to the VDCs using the command: switchto vdc <vdcname>. To switch between VDCs use the same command. To go back to the default VDC you can use: switchback.

SW1-1# switch switchback switchto SW1-1# switchto ? vdc Manage Virtual Device Context SW1-1# switchto vdc ? SW1-1 VDC number 1 SW1-2 VDC number 2 SW1-3 VDC number 3 SW1-4 VDC number 4



SW1-1# switchback ? <CR>

There is not really a best way to configure this network. As long as it works and within the requirements of the chapter the solution is good. The following configuration is considered best practice in this case.

The first assignment is to ensure that all relevant switches carry VLANs that are required for those switches. The topology is designed to be built using Switched Virtual Interfaces (SVIs) also called VLAN interfaces. The SVI’s are created as Layer 3 end of the layer 2 domain and are very common in Datacenter environments. We are using the SVI’s in this topology as it saves a lot of physical cables and ensures a very flexible environment where we can add and remove certain hosts from the interfaces.

Along with the creation of the VLANs and the IP addressing you also need to be sure that the VLANs are transported between the switches that are used as end-point of the interface.

Within the VDC configuration you are able to see the physical interfaces that are connected between the different logical switches. On those interfaces a trunk connection carrying the specific VLANs should be configured. Of course you could enable all VLANs on all interfaces, but this is not helping the topology as spanning-tree will kick in and block at least one connection for each VLAN somewhere in the topology where (without any other spanning-tree configuration) you aren’t sure about which link will be blocked for this.

Now we decide to only configure all VLANs that also have an SVI configured on the same switch. Along with that only a single or a few VLAN(s) is/are allowed on the trunk link between the 2 switches that are configured. Thus we are sure that never a loop exists in the layer 2 network. Which in this case means that spanning-tree will never block any link in our network and we are running a perfect layer 3 network.

The IP addressing is stated in the drawing and we use the VLAN number as subnet ID and the address noted with the switches (according to hostname) as the host portion of the address. All subnets are /24 or 255.255.255.0.

Next is to turn this theory into configuration:

SW1-1

SW1-1# conf t Enter configuration commands, one per line. End with CNTL/Z. SW1-1(config)# feature interface-vlan

As we start with a new configuration we need to enable the SVI feature again.

SW1-1(config)# vlan 112,211,113,122,221 SW1-1(config-vlan)# exit

We need to create the layer 2 VLANs that are used on this particular switch.

SW1-1(config)# interface vlan 113



SW1-1(config-if)# ip address 198.18.113.11 255.255.255.0 SW1-1(config-if)# no shutdown SW1-1(config-if)# int vlan 112 SW1-1(config-if)# ip address 198.18.112.11/24 SW1-1(config-if)# no shutdown SW1-1(config-if)# int vlan 211 SW1-1(config-if)# ip add 198.18.211.11/24 SW1-1(config-if)# no shut SW1-1(config-if)# int vlan 122 SW1-1(config-if)# ip add 198.18.122.11/24 SW1-1(config-if)# no shut SW1-1(config-if)# int vlan 221 SW1-1(config-if)# ip add 198.18.221.11/24 SW1-1(config-if)# no shut

Connection to SW1-2

SW1-1(config-if)# int Ethernet1/11 SW1-1(config-if)# switchport SW1-1(config-if)# switchport mode trunk SW1-1(config-if)# switchport trunk allowed vlan 112,211 SW1-1(config-if)# no shutdown

Connection to SW1-3

SW1-1(config-if)# int Ethernet1/13 SW1-1(config-if)# switchport SW1-1(config-if)# switchport mode trunk SW1-1(config-if)# switchport trunk allowed vlan 113 SW1-1(config-if)# no shutdown

Connection to SW2


Now to verify everything is configured properly a show ip interface brief is enough to show you all the details that we just configured.

SW1-1(config-if)# sh ip int brief IP Interface Status for VRF "default"(1) Interface IP Address Interface Status Vlan112 198.18.112.11 protocol-down/link-down/admin-up Vlan113 198.18.113.11 protocol-down/link-down/admin-up Vlan122 198.18.122.11 protocol-down/link-down/admin-up Vlan211 198.18.211.11 protocol-down/link-down/admin-up Vlan221 198.18.221.11 protocol-down/link-down/admin-up SW1-1(config-if)#



The first switch is configured. Now finishing up by configuring all other layer 2 and layer 3 switches in the network. Both Nexus 5500 switches contain the Layer 3 module so have the possibility to create a full layer 3 topology.

SW1-2

SW1-2# conf t SW1-2(config)# feature interface-vlan SW1-2(config)# vlan 112,211,124,123,132 SW1-2(config-vlan)# exit SW1-2(config)# int vlan 124 SW1-2(config-if)# ip add 198.18.124.12/24 SW1-2(config-if)# no shut SW1-2(config-if)# int vlan 112 SW1-2(config-if)# ip add 198.18.112.12/24 SW1-2(config-if)# no shut SW1-2(config-if)# int vlan 211 SW1-2(config-if)# ip add 198.18.211.12/24 SW1-2(config-if)# no shut SW1-2(config-if)# int vlan 123 SW1-2(config-if)# ip add 198.18.123.12/24 SW1-2(config-if)# no shut SW1-2(config-if)# int vlan 132 SW1-2(config-if)# ip add 198.18.132.12/24 SW1-2(config-if)# no shut

Connection to SW1-1


Connection to SW1-4


Connection to SW3




Verification

SW1-2(config-if)# sh ip int brief IP Interface Status for VRF "default"(1) Interface IP Address Interface Status Vlan112 198.18.112.12 protocol-down/link-down/admin-up Vlan124 198.18.124.12 protocol-down/link-down/admin-up Vlan123 198.18.123.12 protocol-down/link-down/admin-up Vlan211 198.18.211.12 protocol-down/link-down/admin-up Vlan132 198.18.132.12 protocol-down/link-down/admin-up SW1-2(config-if)#

SW1-3

SW1-3# conf t SW1-3(config)# feature interface-vlan SW1-3(config)# vlan 113,134 SW1-3(config-vlan)# exit SW1-3(config)# int vlan 113 SW1-3(config-if)# ip add 198.18.113.13/24 SW1-3(config-if)# no shut SW1-3(config-if)# int vlan 134 SW1-3(config-if)# ip add 198.18.134.13/24

Connection to SW1-1


Connection to SW1-4


Verification

SW1-3(config-if)# sh ip int brief IP Interface Status for VRF "default"(1) Interface IP Address Interface Status Vlan113 198.18.113.13 protocol-down/link-down/admin-up Vlan134 198.18.134.13 protocol-down/link-down/admin-up SW1-3(config-if)#



SW1-4

SW1-4# conf t SW1-4(config)# feature interface-vlan SW1-4(config)# vlan 124,134 SW1-4(config-vlan)# exit SW1-4(config)# int vlan 124 SW1-4(config-if)# ip add 198.18.124.14/24 SW1-4(config-if)# no shut SW1-4(config-if)# int vlan 134 SW1-4(config-if)# ip add 198.18.134.14/24

Connection to SW1-3


Connection to SW1-4


Verification

SW1-4(config-if)# sh ip int brief IP Interface Status for VRF "default"(1) Interface IP Address Interface Status Vlan124 198.18.124.14 protocol-down/link-down/admin-up Vlan134 198.18.134.14 protocol-down/link-down/admin-up SW1-4(config-if)#

SW2

SW2# conf t SW2(config)# feature interface-vlan SW2(config)# vlan 122,221 SW2(config-vlan)# exit SW2(config)# int vlan 122 SW2(config-if)# ip add 198.18.122.2/24 SW2(config-if)# no shut SW2(config-if)# int vlan 221 SW2(config-if)# ip add 198.18.221.2/24



Connection to SW1-1

SW2(config-if)# int Ethernet1/1 SW2(config-if)# switchport SW2(config-if)# switchport mode trunk SW2(config-if)# switchport trunk allowed vlan 122,221 SW2(config-if)# no shutdown

Connection to SW3

SW2(config-if)# int Ethernet1/5 SW2(config-if)# no switchport SW2(config-if)# no shutdown SW2(config-if)# int Ethernet1/5.100 SW2(config-if)# encapsulation dot1q 100 SW2(config-if)# ip add 198.18.100.2/24 SW2(config-if)# no shutdown

This will be our only Layer 3 interface in the network without SVI. Still a VLAN is configured for this link. This is configured using a sub-interface on the physical interface. We first make the interface a layer 3 port and then create the sub-interface while specifying the VLAN number using the encapsulation command.

Verification

SW2(config-if)# sh ip int brief IP Interface Status for VRF "default"(1) Interface IP Address Interface Status Vlan122 198.18.122.2 protocol-down/link-down/admin-up Vlan221 198.18.221.2 protocol-down/link-down/admin-up Ethernet1/5.100 198.18.100.2 protocol-down/link-down/admin-up SW2(config-if)#

SW3

SW3# conf t SW3(config)# feature interface-vlan SW3(config)# vlan 123,132 SW3(config-vlan)# exit SW3(config)# int vlan 123 SW3(config-if)# ip add 198.18.123.3/24 SW3(config-if)# no shut SW3(config-if)# int vlan 132 SW3(config-if)# ip add 198.18.132.3/24



Connection to SW1-1

SW3(config-if)# int Ethernet1/1 SW3(config-if)# switchport SW3(config-if)# switchport mode trunk SW3(config-if)# switchport trunk allowed vlan 123,132 SW3(config-if)# no shutdown

Connection to SW3

SW3(config-if)# int Ethernet1/5 SW3(config-if)# no switchport SW3(config-if)# no shutdown SW3(config-if)# int Ethernet1/5.100 SW3(config-if)# encapsulation dot1q 100 SW3(config-if)# ip add 198.18.100.3/24 SW3(config-if)# no shutdown

Verification

SW3(config-if)# sh ip int brief IP Interface Status for VRF "default"(1) Interface IP Address Interface Status Vlan123 198.18.123.3 protocol-down/link-down/admin-up Vlan132 198.18.132.3 protocol-down/link-down/admin-up Ethernet1/5.100 198.18.100.3 protocol-down/link-down/admin-up SW3(config-if)#

The final assignment is to create Loopback interfaces on the routers in the network so they can be advertised. Loopback interfaces are typically used for managing a device in the network. Now with NX-OS this is usually done using the mgmt0 interface on the devices. The best practice is indeed to use this mgmt0 interface, instead you have a strict requirement to have a redundant management interface. The mgmt0 interface is always non-redundant, so when this dedicated piece of hardware fails, you can’t manage the switch.

It’s highly recommended to do software upgrades through the mgmt0 interface as this interface has a much higher connection which is not rate-limited to the control-plane. When copying software images through an in-band management connection (over a Loopback, Layer 3 interface or SVI) you will see much lower transfer rates as this connection to the control-plane is rate-limited.

In this lab the Loopback interfaces will be used for various purposes and are a good test to see if all routers are in the routed network. Don’t think you already passed a ‘full reachability’ statement yet, because you could be missing some crucial links in the routing protocol which means you didn’t pass the ‘full reachability’ and loose precious points.

Now to finish the task, we create all Loopback interfaces on the switches.

SW1-1

SW1-1(config-if)# int Loopback 0



SW1-1(config-if)# ip add 198.18.0.11/32

For once, it’s not necessary to enable this functionality within NX-OS!

After the creation of the Loopback interface we can verify it’s working properly with a show interface or again a show ip interface brief.

SW1-1(config-if)# do sh int lo0 loopback0 is up Hardware: Loopback Internet Address is 198.18.0.1/32 MTU 1500 bytes, BW 8000000 Kbit, DLY 5000 usec reliability 255/255, txload 1/255, rxload 1/255 Encapsulation LOOPBACK 0 packets input 0 bytes 0 multicast frames 0 compressed 0 input errors 0 frame 0 overrun 0 fifo 0 packets output 0 bytes 0 underruns 0 output errors 0 collisions 0 fifo 0 out_carrier_errors SW1-1(config-if)#

SW1-2

SW1-2(config-if)# int Loopback 0 SW1-2(config-if)# ip add 198.18.0.12/32

SW1-3


SW1-4


SW2

SW2(config-if)# int Loopback 0 SW2(config-if)# ip add 198.18.0.2/32

SW3

SW3(config-if)# int Loopback 0 SW3(config-if)# ip add 198.18.0.3/32



Task 2: Static routing This task is to introduce the static routing features of NX-OS.

The initial assignment asks you to ensure that SW1-3 can ping the loopback address of SW1-4. We can configure a static route for this. A static route is created using a few keywords:

• IP Prefix

• Network Mask in dotted decimal or / notation

• Next-hop interface

• Next-hop IP address

• Properties

The network mask can be noted in classical dotted decimal notation (255.255.255.0) like in Classical IOS or could be using the / notation (x.x.x.x/24) which means the same, but is much easier to configure and easier to see how large the subnet mask is.

Next can either be directly the next-hop IP address where the router will select the next-hop interface by itself by looking into the routing table for a connected prefix for this next-hop. On certain platforms this next-hop can also be ‘recursive’, meaning it can point to another routing entry which in that case points to another routing-entry, until it finds a directly connected prefix which is an interface connected to the router. It depends very much on the platform how many levels of recursion are supported. Best practice is to create static routes to point directly to a connected interface.

Instead of a next-hop IP address you can also specify a next-hop interface. Which usually points to a point-to-point (serial or tunnel) interface which means that traffic that enters, needs to exit out on the other end, so a next-hop IP address is not necessary. You can’t configure an Ethernet interface (physical or SVI) as the next-hop interface, after that you always need to specify the next-hop interface. This is because an Ethernet interface is a multi-access network. Meaning that multiple hosts reside on this interface and the router will then never know where to forward the traffic to. Therefore it’s always required to specify this destination IP address along with the interface. Now the router will not do a look-up in its routing table to find the interface to send it out, as you just specified it.

After these items to configure the static route, a few properties can be configured which will be discussed later in this chapter.

SW1-1(config)# ip route ? A.B.C.D IP prefix in format i.i.i.i A.B.C.D/LEN IP prefix and network mask length in format x.x.x.x/m SW1-1(config)# ip route 1.1.1.1 ? A.B.C.D IP network mask in format m.m.m.m



SW1-1(config)# ip route 198.18.0.14/32 ? A.B.C.D IP next-hop address in format i.i.i.i A.B.C.D/LEN IP next-hop prefix in format i.i.i.i/m ethernet Ethernet IEEE 802.3z loopback Loopback interface mgmt Management interface null Null interface port-channel Port Channel interface vlan Vlan interface SW1-1(config)# ip route 1.1.1.1/32 vlan 111 ? <CR> <1-255> Route preference *Default value is 1 A.B.C.D IP next-hop address in format i.i.i.i name Specify name of the next hop tag Supply tag value with static route SW1-1(config)# ip route 1.1.1.1/32 2.2.2.2 ? <CR> <1-255> Route preference *Default value is 1 name Specify name of the next hop tag Supply tag value with static route

The static route pointing to the Loopback interface of SW1-4 on SW1-3:

SW1-3(config)# ip route 198.18.0.14/32 198.18.134.14

The destination prefix is the Loopback interface of SW1-4 and the next-hop address is the directly connected SVI interface where the next-hop is the host on the other-end (SW1-4).

SW1-3# ping 198.18.0.14 PING 198.18.0.14 (198.18.0.14): 56 data bytes 64 bytes from 198.18.0.14: icmp_seq=0 ttl=255 time=0.759 ms 64 bytes from 198.18.0.14: icmp_seq=1 ttl=255 time=0.458 ms 64 bytes from 198.18.0.14: icmp_seq=2 ttl=255 time=0.366 ms 64 bytes from 198.18.0.14: icmp_seq=3 ttl=255 time=0.363 ms 64 bytes from 198.18.0.14: icmp_seq=4 ttl=255 time=0.377 ms --- 198.18.0.14 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.363/0.464/0.759 ms SW1-3#

Now we verify and it works! Or not?

The second part of the question states that we should be able to ping it from SW1-3’s loopback address. Now if we try that. This doesn’t work!

SW1-3# ping 198.18.0.14 source 198.18.0.13 PING 198.18.0.14 (198.18.0.14) from 198.18.0.13: 56 data bytes Request 0 timed out Request 1 timed out



Request 2 timed out Request 3 timed out Request 4 timed out --- 198.18.0.14 ping statistics --- 5 packets transmitted, 0 packets received, 100.00% packet loss SW1-3#

Pay attention that in classical IOS you were able to use the interface name as a source in this command, but within NX-OS you always need to specify the source IP address instead of the interface.

Why doesn’t the ping success? Well that’s because SW1-4 has no routing entry for the Loopback address of SW1-3 in its routing table. Therefore a static route also needs to be configured on SW1-4 for the return traffic back to SW1-3.


After configuring this static route on SW1-4. Both switches are able to ping each other’s Loopbacks while using their own Loopback address as source for the traffic.

SW1-3# ping 198.18.0.14 source 198.18.0.1 PING 198.18.0.14 (198.18.0.14) from 198.18.0.13: 56 data bytes 64 bytes from 198.18.0.14: icmp_seq=0 ttl=255 time=0.759 ms 64 bytes from 198.18.0.14: icmp_seq=1 ttl=255 time=0.458 ms 64 bytes from 198.18.0.14: icmp_seq=2 ttl=255 time=0.366 ms 64 bytes from 198.18.0.14: icmp_seq=3 ttl=255 time=0.363 ms 64 bytes from 198.18.0.14: icmp_seq=4 ttl=255 time=0.377 ms --- 198.18.0.14 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.363/0.464/0.759 ms SW1-3#

Next up in the second assignment of this task is to configure a static route so that SW1-1 and SW1-2 can ping each other’s loopback addresses, but instead of using the directly connected links between those 2 switches they should travel across SW1-3 and SW1-4. This means that we need to point the static routes over the correct interfaces.

SW1-1


SW1-2


Now the correct static routes are created, but a ping won’t succeed.

SW1-1# ping 198.18.0.12 source 198.18.0.11 PING 198.18.0.12 (198.18.0.12) from 198.18.0.11: 56 data bytes Request 0 timed out Request 1 timed out



Request 2 timed out Request 3 timed out Request 4 timed out --- 198.18.0.12 ping statistics --- 5 packets transmitted, 0 packets received, 100.00% packet loss SW1-1#

This is because the switches that are in transit of the path, also need to know the routes back to the loopback addresses of the switches. This means another 2 static routes need to be created on both SW1-3 and SW1-4 to ensure traffic going back and forth knows the way to go.

SW1-3

SW1-3(config)# ip route 198.18.0.12/32 198.18.134.14 SW1-3(config)# ip route 198.18.0.11/32 198.18.113.11

SW1-4

SW1-4(config)# ip route 198.18.0.12/32 198.18.124.12 SW1-4(config)# ip route 198.18.0.11/32 198.18.134.13

After configuring those additional static routes the ping from both switches succeeds, even while using the local Loopback interface as the source address.

SW1-1# ping 198.18.0.12 source 198.18.0.11 PING 198.18.0.12 (198.18.0.12) from 198.18.0.11: 56 data bytes 64 bytes from 198.18.0.12: icmp_seq=0 ttl=255 time=0.759 ms 64 bytes from 198.18.0.12: icmp_seq=1 ttl=255 time=0.458 ms 64 bytes from 198.18.0.12: icmp_seq=2 ttl=255 time=0.366 ms 64 bytes from 198.18.0.12: icmp_seq=3 ttl=255 time=0.363 ms 64 bytes from 198.18.0.12: icmp_seq=4 ttl=255 time=0.377 ms --- 198.18.0.12 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.363/0.464/0.759 ms SW1-1# SW1-2# ping 198.18.0.11 source 198.18.0.12 PING 198.18.0.11 (198.18.0.11) from 198.18.0.12: 56 data bytes 64 bytes from 198.18.0.11: icmp_seq=0 ttl=255 time=0.759 ms 64 bytes from 198.18.0.11: icmp_seq=1 ttl=255 time=0.458 ms 64 bytes from 198.18.0.11: icmp_seq=2 ttl=255 time=0.366 ms 64 bytes from 198.18.0.11: icmp_seq=3 ttl=255 time=0.363 ms 64 bytes from 198.18.0.11: icmp_seq=4 ttl=255 time=0.377 ms --- 198.18.0.11 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.363/0.464/0.759 ms SW1-2#

The path between the switches can be checked via a Traceroute

SW1-1(config)# traceroute 198.18.0.12 sour 198.18.0.11 traceroute to 198.18.0.12 (198.18.0.12) from 198.18.0.11, 30 hops max, 40 byte packets



1 198.18.113.13 (198.18.113.13) 0.671 ms 0.505 ms 0.631 ms 2 198.18.134.14 (198.18.134.14) 1.003 ms 0.8 ms 0.905 ms 3 198.18.124.12 (198.18. 124.12) 0.903 ms 0.6 ms 0.542 ms SW1-1(config)#

The next assignment is to configure some properties along with the static route. Besides that we need to configure a black hole for a specific subnet. This means that we point the static route not towards an interface or to a next-hop IP address, but towards a special interface. This interface is called the null0 interface. In other words: bit bucket. Every packet that is sent to that interface will be dropped and not processed further.

Besides pointing the static towards the bit-bucket it also needs to have some properties attached, which are easy to spot with use of the question mark in the CLI.

The only tricky part is that you need to know the default route preference or administrative distance on NX-OS of static routes as you need to increase it with +1.

SW1-2(config)# ip route 192.0.1.0/24 null 0 ? <CR> <1-255> Route preference *Default value is 1 name Specify name of the next hop tag Supply tag value with static route

After typing the question mark menu, you can already see the default route preference value (in classical IOS the administrative distance). This should then be upped with 1 to comply with the question.

Now it’s possible to configure a name for the static route, but for some reason it’s then impossible to change the route preference again. There is no specific reason why this isn’t possible, this is a limitation in the current CLI. The name feature has just been released, so the CLI is not totally good with it. These are things you will see more common in NX-OS, where new features get introduced very frequently. The OS is still growing in features, where Classical IOS has grown for the last 20 years, therefore some features are not mature yet.

SW1-2(config)# ip route 192.0.1.0/24 null 0 2 name BLACKHOLE ? ^ % Invalid number at '^' marker. SW1-2(config)# ip route 192.0.1.0/24 null 0 2 tag 666

After configuring the right static route it can be verified using a show ip route where it’s shown that the route preference is now 2 instead of 1 and that the static route is tagged with a value of 666. Route tagging is used during redistribution to recognize certain routes in your topology as those tags are also preserved through dynamic routing protocols.

SW1-2(config)# sh ip route IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop



'[x/y]' denotes [preference/metric] 1.2.3.4/32, ubest/mbest: 1/0 *via Null0, [1/0], 1w0d, static 192.0.1.0/24, ubest/mbest: 1/0 *via Null0, [2/0], 00:00:02, static, tag 666 198.18.0.1/32, ubest/mbest: 2/0, attached *via 198.18.0.1, Lo0, [0/0], 16:23:24, local *via 198.18.0.1, Lo0, [0/0], 16:23:24, direct 198.18.0.12/32, ubest/mbest: 2/0, attached *via 198.18.0.14, Lo1, [0/0], 15:36:01, local *via 198.18.0.14, Lo1, [0/0], 15:36:01, direct

Verify the configuration using a show command. Pay attention that you need to use quotes in NX-OS while doing a search with multiple words.

SW1-2(config)# sh run | in "ip route" ip route 0.0.0.0/0 10.160.48.1 ip route 1.2.3.4/32 Null0 ip route 192.0.1.0/24 Null0 tag 666 2 ip route 198.18.0.14/32 198.18.134.14

Now by default traffic that is dropped by the switch will send ICMP unreachable messages back to the sender. These messages can be used by hackers to detect whether a certain destination is reachable or not. In Classical IOS it was possible to configure this on the Null0 interface to generally disable it for the dropped traffic through the Null0 interface, but in NX-OS this is only possible to configure on Layer 3 routed interfaces. Besides that a difference is made for ICMP unreachables and port-unreachables. The question clearly states that the interfaces should not send out any unreachable message, so don’t forget to add this one as well. SW1-2 has 5 VLAN interfaces (SVIs) so they should all be configured to not send out the unreachable messages.

SW1-2

SW1-2(config)# int vlan 124 SW1-2(config-if)# no ip unreachables SW1-2(config-if)# no ip port-unreachables SW1-2(config-if)# int vlan 112 SW1-2(config-if)# no ip unreachables SW1-2(config-if)# no ip port-unreachables SW1-2(config-if)# int vlan 211 SW1-2(config-if)# no ip unreachables SW1-2(config-if)# no ip port-unreachables SW1-2(config-if)# int vlan 123 SW1-2(config-if)# no ip unreachables SW1-2(config-if)# no ip port-unreachables SW1-2(config-if)# int vlan 132 SW1-2(config-if)# no ip unreachables SW1-2(config-if)# no ip port-unreachables



To finish the task remove all previously configured static routes from the configuration and continue with the next task.

SW1-1

SW1-1(config)# no ip route 198.18.0.12/32 198.18.113.13

SW1-2

SW1-2(config)# no ip route 198.18.0.11/32 198.18.124.14 SW1-2(config)# no ip route 192.0.1.0/24 null 0

SW1-3

SW1-3(config)# no ip route 198.18.0.14/32 198.18.134.14 SW1-3(config)# no ip route 198.18.0.12/32 198.18.134.14 SW1-3(config)# no ip route 198.18.0.11/32 198.18.113.11

SW1-4

SW1-4(config)# no ip route 198.18.0.12/32 198.18.124.12 SW1-4(config)# no ip route 198.18.0.11/32 198.18.134.13 SW1-4(config)# no ip route 198.18.0.13/32 198.18.134.13

Task 3: EIGRP The next task is about configuring the Enhanced Interior Gateway Routing Protocol or EIGRP. The EIGRP protocol is the first dynamic routing protocol that is tested in the CCIE Data Center. The next task regarding Open Shortest Path First (OSPF) is the second routing protocol. Other dynamic routing protocols like RIP, IS-IS and BGP are not tested in this Data Center exam. Usually you will not find BGP configuration in a typical Data Center environment on the Nexus switches, but that will be the task of some kind of PE (Provider Edge) router that is above the core routing layer. This exam doesn’t focus on that part and is only interested in the most widely used Interior Gateway Protocol like EIGRP and OSPF. RIP is an outdated protocol which is used only in very special cases, whilst IS-IS is more a Service Provider protocol, due to scaling advantages.

The EIGRP protocol is a Cisco proprietary protocol. Officially it’s an open document where it’s based on, so every routing vendor should be able to write its own implementation, except nobody has. Therefore EIGRP is considered Cisco proprietary.

The EIGRP protocol is a combination of Distance-Vector and Link-State protocols. It’s a very fast converging protocol due to incremental updates sent to neighbors.



The DUAL (Diffusing Update Algorithm) calculates the best routes according to the shortest distance (based on various link properties). When a successor route is chosen this distance combined with the local link cost is then used as the Feasible Distance. This is the mechanism that makes EIGRP such a fast converging protocol.

The DUAL does a second run through the routing table and calculates a second best path based on the Feasible Distance. A route can become second best when that distance (advertised distance) is lower or equal to the feasible distance. When this would not be the case and a second best path could be any cost, you could run into routing loops as the router doesn’t know the exact topology like a link-state protocol does, so the traffic could be crossing the same router again. A Feasible Successor is selected and installed as a backup route to the Successor route. When the primary path fails the Feasible Successor route is immediately installed as new best path and the router doesn’t have to calculate another path.

Currently with initiatives like IP FastReRoute (FRR) or IP Loop-Free-Alternate (LFA) this mechanism is also possible on OSPF and IS-IS, but EIGRP has been using this mechanism for a long time and is known to be very successful in fast convergence!

The EIGRP metric / distance calculation is quite difficult. The full formula is as follows:

metric = [k1 x bandwidth + (k2 x bandwidth)/(256 - load) + k3 x delay + k6 x extended attributes] x [k5/(reliability + k4)]

This calculation is not used in any network at this moment. This is because most of the K-values area at 0 and multiplication to 0 always result in a 0. Currently only the values Bandwidth and Delay (K1 and K3) are used to calculate the metric of an interface in EIGRP.

With the introduction of Wide Metrics (64-bit metrics) additional K-values came into play. The first is Jitter (in microseconds) and the second is Energy (in watts per kilobit). With this case you can enable some sort of Performance Routing by using EIGRP metrics. A lower powered device should attract more traffic than a high powered device for example.

There are a lot of scenarios in which this would be useful. By default the new K6 values are turned off in the metric calculation. Though the values are added in the EIGRP update packets.

Wide metrics are also backwards compatible with regular metrics. When a non-wide-metric router receives an update where the extended attributes are included it will transparently pass it through without updating it, as a wide-metric-enabled router would do. Although it might result in suboptimal routing in some cases.

One big advantage of EIGRP on NX-OS is that the auto-summary (classful routing) feature has been disabled by default and is not possible to turn on as well.

The initial configuration of this task is to configure EIGRP on SW1-2 and SW1-4 whilst enabling loopbacks in this network as well without forming adjacencies.



EIGRP uses an autonomous system number to identify if the routers will allow a neighbor relationship with each other.

Let’s configure EIGRP in our Data Center network:

SW1-4(config-if)# feature eigrp LAN_ENTERPRISE_SERVICES_PKG license not installed. eigrp feature will be shutdown after grace period of approximately 119 day(s)

We first enable the EIGRP feature, which automatically enables our LAN_ENTERPRISE_SERVICES license on the box. This particular test router has no license installed, so the CLI warns you about the grace period which after the feature will be turned off. For testing purposes we don’t have anything to worry about for now.

SW1-4(config)# router eigrp IPEXPERT

The only thing we need to configure for the process is to create the process with a name. We use IPEXPERT in this question. The reason is that when the AS number is specified as the process name it automatically gets this AS number assigned and the process is running.

Currently without specifying the AS number, the process is in a shutdown state as we can see in the verification.

SW1-4(config-router)# sh ip eigrp IP-EIGRP AS 0 ID 198.18.0.14 VRF default Process-tag: IPEXPERT Status: shutdown Authentication mode: none Authentication key-chain: none Metric weights: K1=1 K2=0 K3=1 K4=0 K5=0 IP proto: 88 Multicast group: 224.0.0.10 Int distance: 90 Ext distance: 170 Max paths: 8 Number of EIGRP interfaces: 0 (0 loopbacks) Number of EIGRP passive interfaces: 0 Number of EIGRP peers: 0 Graceful-Restart: Enabled Stub-Routing: Disabled NSF converge time limit/expiries: 120/0 NSF route-hold time limit/expiries: 240/0 NSF signal time limit/expiries: 20/0 Redistributed max-prefix: Disabled SW1-4(config-router)# autonomous-system 64999



To get the process enabled we need to configure the AS number manually using the dedicated command for this. Now when we verify, the process is running with the correct name and AS number attached.

SW1-4(config-router)# sh ip eigrp IP-EIGRP AS 64999 ID 198.18.0.14 VRF default Process-tag: IPEXPERT Status: running Authentication mode: none Authentication key-chain: none Metric weights: K1=1 K2=0 K3=1 K4=0 K5=0 IP proto: 88 Multicast group: 224.0.0.10 Int distance: 90 Ext distance: 170 Max paths: 8 Number of EIGRP interfaces: 0 (0 loopbacks) Number of EIGRP passive interfaces: 0 Number of EIGRP peers: 0 Graceful-Restart: Enabled Stub-Routing: Disabled NSF converge time limit/expiries: 120/0 NSF route-hold time limit/expiries: 240/0 NSF signal time limit/expiries: 20/0 Redistributed max-prefix: Disabled SW1-4(config-router)#

The rest of the configuration is done under the interfaces that will be placed in the EIGRP process. Within Classical IOS much more configuration was done under the process, but this has now moved to the interface.

SW1-4

SW1-4(config-router)# int vlan124 SW1-4(config-if)# ip router eigrp IPEXPERT

SW1-2

SW1-2(config)# feature eigrp SW1-2(config)# router eigrp IPEXPERT SW1-2(config-router)# autonomous-system 64999 SW1-2(config-router)# exit SW1-2(config)# int vlan124 SW1-2(config-if)# ip router eigrp IPEXPERT SW1-2(config-if)#

Once we enable the EIGRP process on the correct interface, the neighbor adjacency almost immediately comes up and is working correctly.

SW1-4

SW1-4(config-if)# sh ip eigrp nei IP-EIGRP neighbors for process 64999 VRF default



H Address Interface Hold Uptime SRTT RTO Q Seq

(sec) (ms) Cnt Num

0 198.18.124.12 Vlan124 12 00:00:06 3 200 0 2

SW1-4(config-if)#

SW1-2

SW1-2(config-if)# sh ip eigrp nei IP-EIGRP neighbors for process 64999 VRF default H Address Interface Hold Uptime SRTT RTO Q Seq

(sec) (ms) Cnt Num

0 198.18.124.14 Vlan124 12 00:00:16 10 200 0 2

SW1-2(config-if)#

Once we enable the EIGRP process on the correct interface, the neighbor adjacency almost immediately comes up and is working correctly. Now without any advertised interfaces as it’s only enabled on a point-to-point link.

Before we will advertise any subnets we first need to ensure that the adjacency between the switches/routers is secure as stated in the question.

This is done using the authentication configuration keywords and a key chain with a valid key. It’s possible to configure multiple keys with lifetimes so you can change your EIGRP authentication keys without downtime by making the acceptance lifetime a little longer than the sending lifetime. That way new keys are automatically used, but old keys are still accepted for a longer period so other routers get time to also send this new key.

SW1-4

SW1-4(config)# key chain EIGRP_CHAIN SW1-4(config-keychain)# key 1 SW1-4(config-keychain-key)# key-string IPEXPERT

As by the default parameters stated at the beginning of this workbook we will configure a password of IPEXPERT to secure the adjacency.

SW1-4(config-keychain-key)# int vlan124 SW1-4(config-if)# ip authentication ? key-chain Key-chain mode Mode SW1-4(config-if)# ip authentication key-chain eigrp ? WORD Process tag (Max Size 20)



SW1-4(config-if)# ip authentication key-chain eigrp IPEXPERT ? WORD Name of key-chain SW1-4(config-if)# ip authentication key-chain eigrp IPEXPERT EIGRP_CHAIN SW1-4(config-if)# ip authentication mode eigrp IPEXPERT ? md5 Keyed message digest SW1-4(config-if)# ip authentication mode eigrp IPEXPERT md5

The only possible authentication mode is md5. Simple-text / Clear-text authentication is not supported in EIGRP.

SW1-4(config-if)# sh run int vlan124 !Command: show running-config interface Vlan124 version 5.0(3) interface Vlan124 ip address 198.18.124.14/24 ip router eigrp IPEXPERT ip authentication mode eigrp IPEXPERT md5 ip authentication key-chain eigrp IPEXPERT IPEXPERT no shutdown SW1-4(config-if)#

After all settings are configured we clearly see that the neighbor has gone down with the following message:

SW1-4

2012 Sep 19 14:57:39 SW1-4 %EIGRP-5-NBRCHANGE_DUAL: eigrp-IPEXPERT [14861] (def

ault-base) IP-EIGRP(0) 64999: Neighbor 198.18.124.12 (Vlan124) is down: auth

entication mode changed

On the other switch the neighbor has also gone down with another message:

SW1-2

2012 Sep 19 14:57:39 SW1-2 %EIGRP-5-NBRCHANGE_DUAL: eigrp-IPEXPERT [14789] (def

ault-base) IP-EIGRP(0) 64999: Neighbor 198.18.124.14 (Ethernet2/2) is down: Auth

failure

Now we configure the other switch with the authentication settings to have both ends configured equal.

SW1-2

SW1-2(config)# key chain EIGRP_CHAIN SW1-2(config-keychain)# key 1 SW1-2(config-keychain-key)# key-string IPEXPERT



SW1-2(config-keychain-key)# int vlan124 SW1-2(config-if)# ip authentication key-chain eigrp IPEXPERT EIGRP_CHAIN SW1-2(config-if)# ip authentication mode eigrp IPEXPERT md5 SW1-2(config-if)#

After configuring the correct authentication settings the neighbor immediately comes back up and is working as before:

SW1-2


(sec) (ms) Cnt Num

0 198.18.124.14 Vlan124 14 00:00:05 28 200 0 6

SW1-2(config-if)#

SW1-4


(sec) (ms) Cnt Num

0 198.18.124.12 Vlan124 11 00:00:37 7 200 0 6

SW1-4(config-if)#

Besides the securing of the adjacency we need to enable EIGRP on the Loopback interfaces so they can be reached from both switches and no attempts to make adjacencies should be done on the Loopback interfaces. This is done using the passive-interface command.

SW1-2

SW1-2(config)# int lo0 SW1-2(config-if)# ip router eigrp IPEXPERT SW1-2(config-if)# ip passive-interface eigrp IPEXPERT

SW1-4

SW1-4(config)# int lo0 SW1-4(config-if)# ip router eigrp IPEXPERT SW1-4(config-if)# ip passive-interface eigrp IPEXPERT

Now we are able to ping the loopback interfaces that are dynamically learned on SW1-2 and SW1-4.

SW1-2(config)# show ip route eigrp IP Route Table for VRF "default" '*' denotes best ucast next-hop



'**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 198.18.0.14/32, ubest/mbest: 1/0 *via 198.18.124.14, Vlan124, [90/130816], 00:03:47, eigrp-IPEXPERT, internal

SW1-2(config)# SW1-2(config)# SW1-2(config)# ping 198.18.0.14 sour 198.18.0.12 PING 198.18.0.14 (198.18.0.14) from 198.18.0.12: 56 data bytes 64 bytes from 198.18.0.14: icmp_seq=0 ttl=254 time=0.591 ms 64 bytes from 198.18.0.14: icmp_seq=1 ttl=254 time=0.402 ms 64 bytes from 198.18.0.14: icmp_seq=2 ttl=254 time=0.409 ms 64 bytes from 198.18.0.14: icmp_seq=3 ttl=254 time=0.394 ms 64 bytes from 198.18.0.14: icmp_seq=4 ttl=254 time=0.405 ms --- 198.18.0.14 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.394/0.44/0.591 ms SW1-2(config)#

The next step is to configure static routes on SW1-4 and ensure that they are advertised as a summary route towards SW1-2. We cannot overlap any other networks. With the statics we need to configure this is done quite easily.

When we configure the routes:

• 198.18.4.0/24

• 198.18.5.0/24

• 198.18.6.0/24

• 198.18.7.0/24

We see that the third octet is changed.

In binary the third octet looks like this:

• 0 0 0 0 0 1 0 0

• 0 0 0 0 0 1 0 1

• 0 0 0 0 0 1 1 0

• 0 0 0 0 0 1 1 1

Anything between the last 2 bits is changed in these static routes. So when we create a static route with a mask that is 2 bits shorter than the /24 meaning a /22 we are good to go!



First we configure the static routes on SW1-4 pointing to null0 to ensure they are installed in the routing table. When you create a static with a next-hop address which is not valid you will not see them being installed in the routing table, except when they are marked with a permanent option, but this is not available in NX-OS.

SW1-4(config)# ip route 198.18.4.0/24 null0 SW1-4(config)# ip route 198.18.5.0/24 null0 SW1-4(config)# ip route 198.18.6.0/24 null0 SW1-4(config)# ip route 198.18.7.0/24 null0

Next we configure the summary on the interface towards SW1-2 to ensure the statics are summarized as one entry. Note that summary addresses get a default route preference of 5 in the routing table.

SW1-4(config-router)# int Vlan124 SW1-4(config-if)# ip summary-address eigrp IPEXPERT 198.18.0.0/22 ? <CR> <1-255> Administrative distance *Default value is 5 leak-map Allow dynamic prefixes based on the leak-map SW1-4(config-if)# ip summary-address eigrp IPEXPERT 198.18.0.0/22

Now when we check SW1-2 we still don’t see the summary address or any of the static routes appear in the routing table.

SW1-2(config)# show ip route eigrp IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 198.18.0.14/32, ubest/mbest: 1/0 *via 198.18.124.14, Vlan124, [90/130816], 00:05:26, eigrp-IPEXPERT, internal

SW1-2(config)#

We first need to ensure that the prefixes are injected in the EIGRP topology database by using a process called Redistribution. This process ensures that prefixes from another protocol (or connected/static routes) are injected the protocol where it’s configured. This can cause serious routing loops in the network so beware when configuring this. As a pre-caution Cisco implemented a failure check that redistribution always needs to be configured using a route-map to select which prefixes are entering the redistribution process or not.

In our case we only need to configure these 4 static routes so additional filtering on the route-map is not strictly necessary though highly recommended when this would be a real test. During the real test you might find that a later task asks you to configure an additional static route which will then automatically be redistributed in a protocol, which you probably don’t want. This might cause you loosing points in both that new task and the older EIGRP task which you finished in the morning.



Configuring the redistribution enables the static routes to appear in the EIGRP topology database.

SW1-4(config)# route-map STATIC->EIGRP permit 10 SW1-4(config-route-map)# exit SW1-4(config)# router eigrp IPEXPERT SW1-4(config-router)# redistribute static ? route-map Route-map to constrain redistribution

A route-map is required when configuring. You can not apply this command without specifying it anymore.

SW1-4(config-router)# redistribute static route-map STATIC->EIGRP

Now when checking the routing table on SW1-2 we see the new summary address configured.

SW1-2(config)# sh ip route eigrp IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 198.18.0.14/32, ubest/mbest: 1/0 *via 198.18.124.14, Vlan124, [90/130816], 00:19:35, eigrp-IPEXPERT, internal

198.18.4.0/22, ubest/mbest: 1/0 *via 198.18.124.14, Vlan124, [90/51456], 00:00:02, eigrp-IPEXPERT, internal

SW1-2(config)#

In the topology database of SW1-4 we see all 5 prefixes enabled, but only 1 is advertised to the other routers. This is the advantage of distance-vector style protocols where you send your own routing table to the neighbor and can therefore easily filter out certain prefixes that you want to hide.

SW1-4(config-if)# sh ip eigrp topology IP-EIGRP Topology Table for AS(64999)/ID(198.18.0.14) VRF default Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply, r - reply Status, s - sia Status P 198.18.124.0/24, 1 successors, FD is 2816 via Connected, Ethernet2/2 P 198.18.4.0/22, 1 successors, FD is 51200 via Summary (51200/0), Null0 P 198.18.4.0/24, 1 successors, FD is 51200 via Rstatic (51200/0) P 198.18.0.12/32, 1 successors, FD is 130816 via 198.18.124.12 (130816/128320), Ethernet2/2 P 198.18.0.14/32, 1 successors, FD is 128320 via Connected, loopback0



P 198.18.5.0/24, 1 successors, FD is 51200 via Rstatic (51200/0) P 198.18.6.0/24, 1 successors, FD is 51200 via Rstatic (51200/0) P 198.18.7.0/24, 1 successors, FD is 51200 via Rstatic (51200/0) SW1-4(config-if)#

Now we don’t comply with the question yet as we also want one specific prefix to be ‘leaked’ through the summary. This is where we are going to use a leak-map to specify this single prefix to be leaked to the other router (SW1-2).

SW1-4(config-if)# ip prefix-list EIGRP permit 198.18.5.0/24 SW1-4(config)# route-map LEAK permit 10 SW1-4(config-route-map)# match ip add prefix-list EIGRP SW1-4(config-route-map)# exit SW1-4(config)# int vlan124 SW1-4(config-if)# ip summary-address eigrp IPEXPERT 198.18.4.0/22 leak-map LEAK

SW1-4(config-if)#

Now after applying the leak-map we see the leaked prefix also in the routing table of SW1-2.

SW1-2(config)# sh ip route eigrp IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 198.18.0.14/32, ubest/mbest: 1/0 *via 198.18.124.14, Vlan124, [90/130816], 00:25:06, eigrp-IPEXPERT, internal

198.18.4.0/22, ubest/mbest: 1/0 *via 198.18.124.14, Vlan124, [90/51456], 00:05:33, eigrp-IPEXPERT, internal

198.18.5.0/24, ubest/mbest: 1/0 *via 198.18.124.14, Vlan124, [170/51456], 00:00:06, eigrp-IPEXPERT, external

SW1-2(config)#

The rest of the questions in this task relate to certain EIGRP features. Again opening up the Unicast Routing Configuration Guide really helps when solving these tasks as most of the question can relate to something in this guide and walk you thru certain solutions.

The first is to enable wide metrics. Wide metrics are explained in the beginning of this task solution and enable 64-bit wide metrics instead of 32-bit. These can be scaled down to 32-bit numbers again using a scaling-factor. This factor can be adjusted along your preference which is also what the question states.

Be sure to enable it on both EIGRP routers.



SW1-2

SW1-2(config-if)# router eigrp IPEXPERT SW1-2(config-router)# metrics version 64bit SW1-2(config-router)# metrics rib-scale 64

SW1-4

SW1-4(config-if)# router eigrp IPEXPERT SW1-4(config-router)# metrics version 64bit SW1-4(config-router)# metrics rib-scale 64

Next up is changing the EIGRP bandwidth utilization on an interface. This value can be changed. The default is 50% of control-plane traffic, so lowering this with 10% brings it down to 40%.

SW1-2

SW1-2(config-if)# interface Vlan124 SW1-2(config-if)# ip bandwidth-percent eigrp IPEXPERT 40

SW1-4

SW1-4(config-if)# interface Vlan124 SW1-4(config-if)# ip bandwidth-percent eigrp IPEXPERT 40

Tuning the timers on an interface affects the neighborship. So beware of a flapping neighbor when timers are changed. This is because the holddown timer is negotiated in the hello packets of EIGRP. The question states to change the holddown timer to 4 times a hello packet.

By default hello packets are sent every 5 seconds and by default the holddown timer is set to 15 seconds, meaning it can miss 3 hello’s before declaring a neighbor down. So when we change SW1-2 to a holddown timer of 20 seconds it should negotiate it down towards SW1-4 and from then on use the new timer

SW1-2(config)# int vlan124 SW1-2(config-if)# ip hold-time eigrp IPEXPERT ? <1-65535> Seconds before neighbor is considered down *Default value is 15 SW1-2(config-if)# ip hold-time eigrp IPEXPERT 20

Now we can verify this by issuing a show ip eigrp neighbor detail on the neighboring switch to verify it’s using the new holddown timer.

SW1-4(config-if)# sh ip eigrp nei det IP-EIGRP neighbors for process 64999 VRF default H Address Interface Hold Uptime SRTT RTO Q Seq

(sec) (ms) Cnt Num

0 198.18.124.12 Eth2/2 18 00:00:01 39 234 0 31

Version 12.4/1.2, Retrans: 0, Retries: 0, Prefixes: 1



SW1-4(config-if)#

Note that SW1-2 and SW1-4 are now using different values to declare each other down. This is exactly what the question indicates.

The next 2 questions indicate some tuning to the EIGRP election process. The configuration guide is your friend while answering these types of questions. The Stuck in Active timer and the Max-Hops property needs to be changed to finish these questions and are easily found in the Tuning EIGRP section of the configuration guide.

SW1-2

SW1-2(config-if)# router eigrp IPEXPERT SW1-2(config-router)# timers active-time 5 SW1-2(config-router)# metric max-hops 50

SW1-4

SW1-4(config-if)# router eigrp IPEXPERT SW1-4(config-router)# timers active-time 5 SW1-4(config-router)# metric max-hops 50

The final question of this task requires you to change value K3 on the interfaces. This value is used for calculating the interface delay in the EIGRP metric formula:

metric = [k1 x bandwidth + (k2 x bandwidth)/(256 - load) + k3 x delay + k6 x extended attributes] * [k5/(reliability + k4)]

The delay value is an administrative value on the interface, just like bandwidth and can be administratively changed to do traffic engineering in EIGRP which is more recommended than to changing the bandwidth value.

SW1-2

SW1-2(config-if)# interface Vlan124 SW1-2(config-if)# delay 500

SW1-4

SW1-4(config-if)# interface Vlan124 SW1-4(config-if)# delay 500

After changing this on both sides of the link task 3 is finished!



Task 4: OSPF The next task is about configuring the Open Shortest Path First or OSPF. The OSPF protocol is considered the most widely used dynamic routing protocol. It’s an IETF standard and every networking hardware vendor has its own implementation of this protocol.

The current mostly used version and the version tested on this CCIE lab exam is OSPF version 2. This version only support IP version 4 and has been out for many years. Currently there is a new version of OSPF maturing which is Version 3. This version was initially developed solely for IP version 6, but has now been extended so it also supports IP version 4. In the near future it’s expected that version 3 will take over from version 2. The 2 versions are in no way compatible with each other, so you will have to migrate your whole network over from version 2 to version 3. Some key features have changed in the new version, particularly how interfaces/links are treated in the protocol.

This workbook focuses on the version used in the CCIE Data Center lab exam, which is OSPF version 2 for IP version 4.

The OSPF protocol is a Link-State routing protocol. Which means that instead of pushing the routing table to neighboring routers. The protocol will inform all neighboring routers within a single ‘area’ of the state of each connecting link using link-state-advertisements (LSAs). This information is flooded throughout the OSPF area to all routers. Once all information is gathered the protocol will start a algorithm known as the Dijkstra algorithm or the Shortest Path First algorithm which is also used in GPS navigation systems to determine the shortest route/path to any of the links/prefixes that has been received. To do this, the router places itself as the top of the tree and then branches out to all other links through routers to determine the fastest route.

The awareness of the entire topology for each router does impact on resources, but with the current control-plane architectures of modern routers this is never a problem. OSPF networks can now grow up to tens or hundreds of routers in a single area.

The behavior of OSPF changes as soon as the border of an area is reached. Information between areas is exchanged in a way more like distance vector routing protocols do this. Information about links in the area is summarized and then sent to the so-called backbone area (area 0). The behavior to connect every area to the backbone area is done to prevent loops from occurring in the network. It’s now always known that every next-hop area is area 0 of which then the decision is made where to move on next.

This means that area 0 contains all information about the OSPF network. It’s impossible to filter out any information (except for summarization and non-OSPF external routes) from area 0. The other areas may have limited visibility on the network. This is mostly done for a more lightweight approach to certain areas. It means that you can connect a very small router to a highly advances OSPF backbone network and still be able to participate in the OSPF as this lightweight router in its own area will only get very limited information regarding the topology (usually a default route).



Information is sent through LSA messages. For the CCIE exam it’s highly recommended to know all the different types of messages and the numbers attached to that. You have a high chance of running into questions stating these LSA numbers, where you should know what information is carried within them and when they are propagated in which kind of area.

The LSA types in OSPF version 2 are:

LSA # Name Flooding Description

1 Router area Includes link state information, cost and a list of neighbors. Every router sends this LSA throughout the area.

2 Network area Sent by the designated router on a multi-access network. Lists all the routers within the multi-access network.

3 Summary network Contains information regarding each destination in the area and is sent by the Area Border Router (ABR) towards area 0. It includes link cost from the ABR towards the destination (like a distance-vector protocol).

4 ASBR Summary network Contains information regarding the Autonomous System Boundary Router with a cost to reach it. It’s sent to area 0 by the ABR of the local area.

5 AS External network Contains information regarding routes external to the OSPF network. Generated by the ASBR and sent throughout the entire OSPF network.

6 Multicast n/a Multicast OSPF was never implemented.

7 NSSA External area Same as a type 5, but generated by an ASBR within a Not-so-stubby area. These are only flooded within an area and the ABR converts them to a regular Type 5.

9 Opaque 1 link Same as type 10 and 11. Used for expansion to the OSPF protocol for carrying application information. This type is mainly used for OSPF Graceful Restart

10 Opaque 2 area Same as type 9 and 11. Used in MPLS Fast Reroute scenarios

11 Opaque 3 network Same as type 9 and 10



During the solutions of the task more will be explained about the details regarding the different types of areas and the different types of network links.

When looking at drawing 2. An immediate issue is spotted when looking at the areas that we need to configure to comply with this topology. As just discussed, every area should be connected to area 0 to prevent loops in the network.

Area 264 in this drawing is not connected to area 0 and is therefore not going to work.

There are multiple solutions to this problem, but only one of them is built into NX-OS. OSPF has a built-in functionality to overcome these situations. In a real-life scenario this will hardly ever occur, because it’s much easier to configure an additional interface between 2 routers and create a area 0 connection on there. Another solution would be to use IP-IP or GRE tunnels between the routers to also ensure an additional interface between the routers where you could run area 0 on is created.

The built-in solution we are looking for is an OSPF virtual-link. This is also some kind of tunnel connection between 2 Area Border Routers (ABRs) to create a virtual area 0 connection between a backbone ABR and a partitioned ABR.

Let’s start by creating the OSPF network by building up the adjacencies and advertising the Loopback interfaces into the OSPF network.

SW1-1(config-if)# feature ospf



LAN_ENTERPRISE_SERVICES_PKG license not installed. ospf feature will be shutdown

after grace period of approximately 119 day(s)

Again the feature needs to be enabled. The Enterprise license is required to run any routing protocol, including OSPF.

We also need to supply a tag for the OSPF process. In classical IOS this used to be a process number (ID), but in NX-OS this has changed to a name. In the lab there is no mention of a name, so we use IPEXPERT, but you can also use ‘1’ just like you would in IOS.

SW1-1(config)# router ospf ? WORD Process tag (Max Size 20) SW1-1(config)# router ospf IPEXPERT

That should be sufficient to enable OSPF on the box. Now we need to enable it on the interfaces.

SW1-1(config-router)# int Vlan113 SW1-1(config-if)# ip router ospf IPEXPERT area 2 SW1-1(config-if)# int Vlan122 SW1-1(config-if)# ip router ospf IPEXPERT area 1 SW1-1(config-if)# int Vlan221 SW1-1(config-if)# ip router ospf IPEXPERT area 1 SW1-1(config-if)# int Vlan112 SW1-1(config-if)# ip router ospf IPEXPERT area 0 SW1-1(config-if)# int Vlan211 SW1-1(config-if)# ip router ospf IPEXPERT area 0 SW1-1(config-if)# SW1-1(config-if)# int lo0 SW1-1(config-if)# ip router ospf IPEXPERT area 0

Verifying OSPF is enabled on all interfaces with the correct area mapping:

SW1-1(config-if)# sh ip ospf interface brief OSPF Process ID IPEXPERT VRF default Total number of interface: 6 Interface ID Area Cost State Neighbors Status

Lo0 6 0.0.0.0 1 LOOPBACK 0 up

Vlan113 1 0.0.0.2 40 DR 0 up

Vlan122 2 0.0.0.1 40 DR 0 up

Vlan221 3 0.0.0.1 40 DR 0 up

Vlan112 4 0.0.0.0 40 DR 0 up

Vlan211 5 0.0.0.0 40 DR 0 up



SW1-1(config-if)#

Now we enable the interfaces on all OSPF routers and see that OSPF adjacencies start to come up.

SW1-2

SW1-2(config-if)# feature ospf SW1-2(config)# router ospf IPEXPERT SW1-2(config-router)# SW1-2(config-if)# int Vlan123 SW1-2(config-if)# ip router ospf IPEXPERT area 1 SW1-2(config-if)# int Vlan132 SW1-2(config-if)# ip router ospf IPEXPERT area 1 SW1-2(config-if)# int Vlan112 SW1-2(config-if)# ip router ospf IPEXPERT area 0 SW1-2(config-if)# int Vlan211 SW1-2(config-if)# ip router ospf IPEXPERT area 0 SW1-2(config-if)# SW1-2(config-if)# int lo0 SW1-2(config-if)# ip router ospf IPEXPERT area 0

SW1-3

SW1-3(config-if)# feature ospf SW1-3(config)# router ospf IPEXPERT SW1-3(config-router)# SW1-3(config-if)# int Vlan113 SW1-3(config-if)# ip router ospf IPEXPERT area 2 SW1-3(config-if)# int Vlan134 SW1-3(config-if)# ip router ospf IPEXPERT area 0.0.1.8 SW1-3(config-if)# SW1-3(config-if)# int lo0 SW1-3(config-if)# ip router ospf IPEXPERT area 2

The question states that the dotted decimal notation should be used to configure area 264 in this lab. This means that we can’t just use 265 when configuring it.

When you want to be sure to have the correct notation. You can configure it the normal way and check a show ip ospf interface brief, as that output always gives you the dotted decimal notation even when you configured a decimal based area number. Then you can reconfigure it again, as with grading the proctor will see your decimal based area number instead of the dotted decimal.

SW1-3(config-if)# sh ip ospf int brief OSPF Process ID IPEXPERT VRF default Total number of interface: 3 Interface ID Area Cost State Neighbors Status

Vlan113 1 0.0.0.2 40 BDR 1 up



Vlan134 2 0.0.1.8 40 DR 0 up

Lo0 3 0.0.0.2 1 LOOPBACK 0 up

SW1-3(config-if)#

Now the neighbor has been established, but when we check the routing table. We won’t see the Vlan134 interface in the routing table on SW1-1. This is because the areas are not allowed to be disconnected from area 0. The solution in this case would, as previously explained, be to create an OSPF virtual-link to create a virtual connection to area 0 to fix this issue.

SW1-1(config-if)# sh ip ospf nei OSPF Process ID IPEXPERT VRF default Total number of neighbors: 2 Neighbor ID Pri State Up Time Address Interface 198.18.0.2 1 FULL/BDR 00:01:05 198.18.122.2 Vlan122 198.18.0.13 1 FULL/DR 00:01:31 198.18.113.13 Vlan113 SW1-1(config-if)# sh ip route IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 198.18.0.2/32, ubest/mbest: 1/0 *via 198.18.122.2, Eth2/1, [110/41], 00:01:05, ospf-IPEXPERT, intra 198.18.0.11/32, ubest/mbest: 2/0, attached *via 198.18.0.11, Lo0, [0/0], 00:03:55, local *via 198.18.0.11, Lo0, [0/0], 00:03:55, direct 198.18.0.13/32, ubest/mbest: 1/0 *via 198.18.113.13, Eth2/2, [110/41], 00:01:30, ospf-IPEXPERT, intra 198.18.113.0/24, ubest/mbest: 1/0, attached *via 198.18.113.11, Eth2/2, [0/0], 00:03:45, direct 198.18.113.11/32, ubest/mbest: 1/0, attached *via 198.18.113.11, Eth2/2, [0/0], 00:03:45, local 198.18.122.0/24, ubest/mbest: 1/0, attached *via 198.18.122.11, Eth2/1, [0/0], 00:03:39, direct 198.18.122.11/32, ubest/mbest: 1/0, attached *via 198.18.122.11, Eth2/1, [0/0], 00:03:39, local SW1-1(config-if)#

The virtual-link is created, which results in a new OSPF adjacency. This adjacency is an area 0 adjacency and across that adjacency the networks area advertised and the communication with area 0 is established. Within the configuration of the virtual-link it’s required to specify the area thru which the virtual-link is transmitting to reach the next area. What’s typically seen is that students configure the area that needs to be extended as the virtual-link area, but this should be the transit-area. The second property that needs to be configured is the Router-ID of the Area Border Router (ABR) where the virtual-link terminates.

SW1-3

SW1-3(config-if)# router ospf IPEXPERT SW1-3(config-router)# area 2 virtual-link 198.18.0.11 SW1-3(config-router-vlink)# exit



SW1-1

SW1-1(config-router)# router ospf IPEXPERT SW1-1(config-router)# area 2 virtual-link 198.18.0.13 SW1-1(config-router-vlink)# exit

After the virtual-link is created the new VL1 adjacency automatically comes up and the network is now seen in the OSPF routing table!

SW1-1(config-router)# sh ip ospf nei OSPF Process ID IPEXPERT VRF default Total number of neighbors: 3 Neighbor ID Pri State Up Time Address Interface 198.18.0.13 1 FULL/ - 00:00:06 198.18.113.13 VL1 198.18.0.2 1 FULL/BDR 00:08:14 198.18.122.2 Eth2/1 198.18.0.13 1 FULL/DR 00:08:40 198.18.113.13 Eth2/2 SW1-1(config-router)# sh ip route ospf IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 198.18.0.2/32, ubest/mbest: 1/0 *via 198.18.122.2, Eth2/1, [110/41], 00:08:11, ospf-IPEXPERT, intra 198.18.0.13/32, ubest/mbest: 1/0 *via 198.18.113.13, Eth2/2, [110/41], 00:08:36, ospf-IPEXPERT, intra 198.18.134.0/24, ubest/mbest: 1/0 *via 198.18.113.13, Eth2/2, [110/80], 00:00:06, ospf-IPEXPERT, inter SW1-1(config-router)#

Now the rest of the OSPF network needs to be finalized

SW1-4

SW1-4(config-if)# feature ospf SW1-4(config)# router ospf IPEXPERT SW1-4(config-router)# SW1-4(config-if)# int Vlan134 SW1-4(config-if)# ip router ospf IPEXPERT area 0.0.1.8

SW2

SW2(config-if)# feature ospf SW2(config)# router ospf IPEXPERT SW2(config-router)# SW2(config-if)# int Vlan122 SW2(config-if)# ip router ospf IPEXPERT area 1 SW2(config-if)# int Vlan221 SW2(config-if)# ip router ospf IPEXPERT area 1 SW2(config-if)# int Ethernet1/5 SW2(config-if)# ip router ospf IPEXPERT area 1



SW2(config-if)# SW2(config-if)# int lo0 SW2(config-if)# ip router ospf IPEXPERT area 1

SW3

SW3(config-if)# feature ospf SW3(config)# router ospf IPEXPERT SW3(config-router)# SW3(config-if)# int Vlan123 SW3(config-if)# ip router ospf IPEXPERT area 1 SW3(config-if)# int Vlan132 SW3(config-if)# ip router ospf IPEXPERT area 1 SW3(config-if)# int Ethernet1/5 SW3(config-if)# ip router ospf IPEXPERT area 1 SW3(config-if)# SW3(config-if)# int lo0 SW3(config-if)# ip router ospf IPEXPERT area 1

After all OSPF configuration has been completed the Loopbacks are reachable throughout the network without any problems.

SW2(config)# ping 198.18.0.13 source 198.18.0.2 PING 198.18.0.13 (198.18.0.13) from 198.18.0.2: 56 data bytes 64 bytes from 198.18.0.13: icmp_seq=0 ttl=253 time=1.294 ms 64 bytes from 198.18.0.13: icmp_seq=1 ttl=253 time=0.675 ms 64 bytes from 198.18.0.13: icmp_seq=2 ttl=253 time=0.727 ms 64 bytes from 198.18.0.13: icmp_seq=3 ttl=253 time=0.721 ms 64 bytes from 198.18.0.13: icmp_seq=4 ttl=253 time=0.723 ms --- 198.18.0.13 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.675/0.828/1.294 ms SW2(config)# traceroute 198.18.0.13 source 198.18.0.2 traceroute to 198.18.0.13 (198.18.0.13) from 198.18.0.2, 30 hops max, 40 byte packets

1 198.18.122.11 (198.18.122.11) 0.822 ms 0.507 ms 0.503 ms 2 198.18.113.13 (198.18.113.13) 1.244 ms 0.823 ms 0.893 ms SW2(config)#

The next question asks you to ignore the MTU size. This is a command to overcome MTU problems when building up an OSPF adjacency. The OSPF adjacencies are only established when certain link properties and configuration settings are matched on both sides.

One of them is MTU size. This option however is configurable.

SW1-1

SW1-1(config-if)# interface Vlan113 SW1-1(config-if)# ip ospf mtu-ignore



SW1-3

SW1-3(config-if)# interface Vlan113 SW1-3(config-if)# ip ospf mtu-ignore

The next 2 questions in the task can be solved in different ways. You will need both different ways to solve this to get the solution correct. This is because of the behavior OSPF has by default on Ethernet interfaces. OSPF knows various ways of determining a network connection.

Within Classical IOS the choice of different network types was huge. Mainly due to all different serial interfaces that were available that required a different behavior (sometimes broadcast traffic wasn’t supported for example).

Within NX-OS the choice is narrowed down to 2:

Type Description

Broadcast Allows for a shared network (multi-access) with multiple routers that can become neighbors. Uses a Designated Router for centralizing communication.

Point-to-Point Simple point-to-point link where broadcast traffic is supported. There is no option for more than 2 routers on one link, making it a lot faster to converge and has no overhead in Designated Router elections.

By default all Ethernet interfaces (meaning all interfaces on an NX-OS device) are running the Broadcast network type as it would be possible that multiple routers are sharing the same Layer 3 network segment. This adds overhead to the set-up of the adjacency as OSPF Broadcast networks are using a principle of a Designated Router per network segment. This Designated Router (DR) maintains an adjacency with all routers. Other non-DR routers only have this single connection the DR router as there OSPF adjacency. This ensures a good scaling in OSPF broadcast networks. To ensure the network is secured to a DR failure, there is also an election of the Backup Designated Router (BDR).



All Full OSPF adjacencies are shown in the drawing below:

Routers use a different Multicast group to communicate to the DR and BDR routers. For normal communication the 224.0.0.5 group is used, but for DR and BDR communications the 224.0.0.6 group is used. As shown in the drawing. Routers only communicate with the DR and BDR routers. The Hello messages from other routers on the segment are seen, but a full adjacency will not be formed. The routers will remain in a 2WAY type of relationship with each other.

The DR election is a process which is using a Priority value to determine which router is elected. As a tie-breaker the Router ID is used. The router with highest priority value or in case of equal priorities the highest Router ID is elected as the Designated Router. After the DR is chosen the election is run a second time to determine the Backup Designated Router.

The default priority value is 1.

To ensure that a router will never become a designated router you can change the priority value to 0. When the priority is 0 the router will never participate in the DR or BDR election.

So we change the priority to 0 on the interfaces. Keep in mind that this change will flap your OSPF adjacency.

SW2

SW2(config-if)# interface Vlan122 SW2(config-if)# ip ospf priority 0 SW2(config-if)# interface Vlan221 SW2(config-if)# ip ospf priority 0 SW2(config-if)# interface Ethernet1/5 SW2(config-if)# ip ospf priority 0



SW3

SW3(config-if)# interface Vlan123 SW3(config-if)# ip ospf priority 0 SW3(config-if)# interface Vlan132 SW3(config-if)# ip ospf priority 0 SW3(config-if)# interface Ethernet1/5 SW3(config-if)# ip ospf priority 0

Now the 2 uplink interfaces towards the SW1-1 and SW1-2 switches will be switching to the fact that when you issue a show ip ospf neighbor. The neighbors will be shown as DR, meaning the local switch is a DROTHER as the priority value ensures that SW2 and SW3 will never come a DR or BDR on any segment.

Now the problem resides in the connection between SW2 and SW3. Both sides are now configured as priority 0, meaning that neither of the switches will become a DR or a BDR. This now results in a non-functional network between those 2 switches. Where we still want that adjacency to be up!

SW2(config-if)# sh ip ospf nei OSPF Process ID IPEXPERT VRF default Total number of neighbors: 2 Neighbor ID Pri State Up Time Address Interface 198.18.0.11 1 FULL/DR 00:08:14 198.18.122.2 Vlan211 198.18.0.3 1 TWOWAY/DROTHER 00:08:40 198.18.100.2 Eth1/5

We need to solve this to get the OSPF adjacency going between SW2 and SW3.

As just mentioned, OSPF knows 2 network types in NX-OS. The Point-to-Point network type is for Ethernet links that are directly connected between each other where no other router will ever come on the subnet. This type of network also doesn’t know the process of electing a DR or BDR as the 2 nodes are the only routers on the subnet.

When we configure this for the link between SW2 and SW3 we ensure that the routers establish a Full OSPF adjacency and that none of them is a DR or BDR on that segment!

SW2

SW2(config-if)# interface Ethernet1/5 SW2(config-if)# ip ospf network point-to-point

SW3

SW3(config-if)# interface Ethernet1/5 SW3(config-if)# ip ospf network point-to-point SW2(config-if)# sh ip ospf nei OSPF Process ID IPEXPERT VRF default Total number of neighbors: 2



Neighbor ID Pri State Up Time Address Interface 198.18.0.11 1 FULL/DR 00:08:14 198.18.122.2 Vlan211 198.18.0.3 1 FULL/ - 00:08:40 198.18.100.2 Eth1/5

After configuring the point-to-point network type the adjacency is up again and everything is working fine besides that we comply with all rules stated in the question of the task.

Next up is authentication. OSPF does support both MD5 and simple-text passwords for authenticating the protocol. In real-life networks there is no good reason to use simple-text passwords, but for lab purposes you should know how to use both.

Besides that OSPF also knows a difference in authenticating on a link basis or on an area basis. On area basis the OSPF protocol requires the entire area to authenticate its adjacencies. This is a good way to enforce authentication within an area. Link adjacencies are only dependent on the directly connected routers so no other routers in the area know that there is authentication happening.

The first assignment indicates to secure all area 0 adjacencies. One important note here is that most people are configuring authentication on all links between area 0 routers perfectly, but do forget a important area 0 connection. This connection is the virtual-link crossing through area 2 to connect area 264 to the network. This is also an area 0 adjacency and should therefore also be secured with a hashed password! You will lose all points for this task when you forget to secure the virtual-link!

The authentication is configured using a number of commands under the interface.

SW1-1

SW1-1(config-if)# interface Vlan112 SW1-1(config-if)# ip ospf auth authentication authentication-key SW1-1(config-if)# ip ospf authentication ? <CR> key-chain Authentication password key-chain message-digest Use message-digest authentication null Use null(disable) authentication SW1-1(config-if)# ip ospf authentication message-digest SW1-1(config-if)# ip ospf message-digest-key 1 md5 IPexpertSecure SW1-1(config-if)#

When OSPF authentication is enabled, you will enable simple-password (clear-text) password authentication. This is not what the question stated. We should enable a hashed version of the key as the authentication method. This is done using the message-digest authentication method. You are also able to configure a key chain, just like with EIGRP, but it’s not mandatory and in our example we have no reason to do so. Therefore we configure the MD5 key just on the interface. Do not configure the OSPF authentication-key, as that would configure the simple password authentication. Probably the authentication still works, but this is because an empty md5 hash is still an md5 hash and is a valid key for authentication. Still you would lose all the points on this question, because the



authentication is not done using the key as stated in the question. You will need to use the message-digest-key configuration command.

SW1-1(config-if)# interface Vlan211 SW1-1(config-if)# ip ospf authentication message-digest SW1-1(config-if)# ip ospf message-digest-key 1 md5 IPexpertSecure

SW1-2

SW1-2(config-if)# interface Vlan112 SW1-2(config-if)# ip ospf authentication message-digest SW1-2(config-if)# ip ospf message-digest-key 1 md5 IPexpertSecure SW1-2(config-if)# interface Vlan211 SW1-2(config-if)# ip ospf authentication message-digest SW1-2(config-if)# ip ospf message-digest-key 1 md5 IPexpertSecure

The final step is to enable authentication on the virtual-link as this is just an area 0 adjacency as all other.

SW1-1

SW1-1(config-if)# router ospf IPEXPERT SW1-1(config-router)# area 2 virtual-link 198.18.0.13 SW1-1(config-router-vlink)# ? authentication Authentication on the vlink authentication-key Configure the authentication key for the virtual-link

dead-interval Dead interval hello-interval Hello interval message-digest-key Message digest authentication password (key) no Negate a command or set its defaults retransmit-interval Packet retransmission interval transmit-delay Packet transmission delay end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1-1(config-router-vlink)# authentication ? <CR> key-chain Authentication password key-chain message-digest Use message-digest authentication null Use null(disable) authentication SW1-1(config-router-vlink)# authentication message-digest SW1-1(config-router-vlink)# message-digest-key 1 md5 IPexpertSecure

We go into the sub-configuration of the virtual-link and configure the same parameters as you would on an interface.

SW1-3

SW1-3(config-if)# router ospf IPEXPERT SW1-3(config-router)# area 2 virtual-link 198.18.0.11



SW1-3(config-router-vlink)# authentication message-digest SW1-3(config-router-vlink)# message-digest-key 1 md5 IPexpertSecure

The next authentication is area based, which means that adjacencies are enforced on to run authentication. This time it’s using simple-text passwords, so regular authentication parameters. One difference in this configuration is that the authentication is not enabled on the interface, only the key is specified on the interface configuration and the authentication is enabled within the routing process.

SW1-1

SW1-1(config-if)# router ospf IPEXPERT SW1-1(config-router)# area 1 authentication SW1-1(config-router)# SW1-1(config-router)# interface Vlan122 SW1-1(config-if)# ip ospf authentication-key IPexpert SW1-1(config-if)# interface Vlan221 SW1-1(config-if)# ip ospf authentication-key IPexpert

SW1-2

SW1-2(config-if)# router ospf IPEXPERT SW1-2(config-router)# area 1 authentication SW1-2(config-router)# SW1-2(config-router)# interface Vlan123 SW1-2(config-if)# ip ospf authentication-key IPexpert SW1-2(config-if)# interface Vlan132 SW1-2(config-if)# ip ospf authentication-key IPexpert

SW2

SW2(config-if)# router ospf IPEXPERT SW2(config-router)# area 1 authentication SW2(config-router)# SW2(config-router)# interface Vlan122 SW2(config-if)# ip ospf authentication-key IPexpert SW2(config-if)# interface Vlan221 SW2(config-if)# ip ospf authentication-key IPexpert SW2(config-if)# interface Ethernet1/5 SW2(config-if)# ip ospf authentication-key IPexpert

SW3

SW2(config-if)# router ospf IPEXPERT SW2(config-router)# area 1 authentication SW2(config-router)# SW2(config-router)# interface Vlan123 SW2(config-if)# ip ospf authentication-key IPexpert SW2(config-if)# interface Vlan132 SW2(config-if)# ip ospf authentication-key IPexpert SW2(config-if)# interface Ethernet1/5 SW2(config-if)# ip ospf authentication-key IPexpert



The next question is asking about summarization. Now summarization is different in OSPF compared to EIGRP. Distance vector protocols send their routing table towards other routers in the network. A link-state protocol will be default only advertise its link information. Every router needs to have a full view on the network before it can start calculating the shortest path to all destinations in the network.

Now as earlier discussed the OSPF routing protocol uses a distance vector style way of advertising networks to other areas. This means that we can summarize at the borders of areas, before injecting routes into area 0.

This is done using the area-range command in the OSPF protocol configuration. We first need to generate additional Loopback interfaces on SW2.

SW2

SW2(config)# int lo1 SW2(config-if)# ip add 198.18.128.1/24 SW2(config-if)# no shut SW2(config-if)# int lo2 SW2(config-if)# ip add 198.18.129.1/24 SW2(config-if)# no shut SW2(config-if)# int lo3 SW2(config-if)# ip add 198.18.130.1/24 SW2(config-if)# no shut SW2(config-if)# int lo4 SW2(config-if)# ip add 198.18.131.1/24 SW2(config-if)# no shut

When we configure the routes:

• 198.18.128.0/24

• 198.18.129.0/24

• 198.18.130.0/24

• 198.18.131.0/24

We see that the third octet is changed. In binary the third octet looks like this:

• 1 0 0 0 0 0 0 0

• 1 0 0 0 0 0 0 1

• 1 0 0 0 0 0 1 0

• 1 0 0 0 0 0 1 1



Anything between the last 2 bits is changed in these static routes. So when we create a static route with a mask that is 2 bits shorter than the /24 meaning a /22 we are good to go!

We need to get the networks into OSPF. This can be done in 2 ways. One by configuring the Loopback interfaces to appear in the OSPF database and the other is by doing redistribution

Be aware that the summary configuration for both ways is very different! As for the first way the area-range command is used and for the second the summary-address command. In the question it’s stated that the single entry should be seen as a Type 3 LSA in the backbone. When an External route is summarized the summary is also advertised as a Type 5. It’s therefore necessary to use the area-range style configuration as that will inject a regular Type 3 LSA instead of a Type 5.

SW2

SW2(config)# int lo1 SW2(config-if)# ip ospf network IPEXPERT area 1 SW2(config-if)# int lo2 SW2(config-if)# ip ospf network IPEXPERT area 1 SW2(config-if)# int lo3 SW2(config-if)# ip ospf network IPEXPERT area 1 SW2(config-if)# int lo4 SW2(config-if)# ip ospf network IPEXPERT area 1

Now SW2 is not the border router for the network. Summarization can only be done on the area border routers. Therefore the area-range command is not effective when configured on SW2, but only on SW1-1 and on SW1-2. Don’t forget you have 2 ABR’s in area 1!

SW1-1

SW1-1(config-if)# router ospf IPEXPERT SW1-1(config-router)# area 1 range 198.18.128.0/22

SW1-2

SW1-2(config-if)# router ospf IPEXPERT SW1-2(config-router)# area 1 range 198.18.128.0/22

After enabling the area range. Other areas will now only see a single Type 3 LSA for these 4 networks instead of the specific ranges!

The next question states to ensure the whole subnet is seen in the routing table. Isn’t this always done? Well in classical IOS this was the case, but in NX-OS the behavior of Loopback interfaces has changed. There is not a real reason why you would want to put an other than /32 mask on a Loopback interface, but it is possible in all Cisco platforms.

SW1-3

SW1-3(config-if)# interface Loopback1 SW1-3(config-if)# ip address 198.18.13.1/24 SW1-3(config-if)# ip router ospf IPEXPERT area 2 SW1-3(config-if)# no shutdown



By default all Loopback interfaces will be advertised with a /32 mask, no matter what mask has been configured on the Loopback interface itself.

SW1-1(config-if)# sh ip route ospf IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 198.18.0.2/32, ubest/mbest: 1/0 *via 10.10.1.1, Eth2/1, [110/41], 00:01:00, ospf-IPEXPERT, intra 198.18.0.13/32, ubest/mbest: 1/0 *via 10.10.3.2, Eth2/2, [110/41], 00:00:01, ospf-IPEXPERT, intra 198.18.13.1/32, ubest/mbest: 1/0 *via 10.10.3.2, Eth2/2, [110/41], 00:00:01, ospf-IPEXPERT, intra

To overrule this behavior a special command is necessary.

SW1-3

SW1-3(config-if)# interface Loopback1 SW1-3(config-if)# ip ospf advertise-subnet

After configuring this command the route is now shown with its correct /24 mask in the routing table on other routers.

SW1-1(config-if)# sh ip route ospf IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 198.18.0.2/32, ubest/mbest: 1/0 *via 10.10.1.1, Eth2/1, [110/41], 00:05:40, ospf-IPEXPERT, intra 198.18.0.13/32, ubest/mbest: 1/0 *via 10.10.3.2, Eth2/2, [110/41], 00:04:41, ospf-IPEXPERT, intra 198.18.13.0/24, ubest/mbest: 1/0 *via 10.10.3.2, Eth2/2, [110/41], 00:00:11, ospf-IPEXPERT, intra

The next task is a question regarding OSPF areas. The OSPF protocol, as previously mentioned, enables the use of different types of areas. This is done to introduce scaling in the routing protocol. Within an area every router needs to know about all links on all routers. When the area grows to hundreds of routers. There comes a point where nobody can keep up anymore. Then additional areas can be created to lower the impact on routers. With good summarization at area borders the scaling can be enormous. Now when you even want to lower the control-plane overhead on OSPF routers it’s possible to create so-called ‘stub’ areas.

There are several types of stub areas. Each with a different behavior. Below is a table explaining the different types of areas and the behaviors that are related to it.



Type Description

Stub Does not allow external routes (Type-5 LSA’s) in the area. Default route ‘can’ be advertised by the ABR to ensure full reach ability.

Totally Stub Same as a stub area, so not allowing external routes (Type 5 LSA’s). Besides that Summary routes (Type-3 LSA’s) are not allowed as well. Meaning that there is only reach ability within the area and the ABR’s advertise a default route into the area to ensure the routers in that area can still reach the rest of the network. This default is a Type-3 LSA. Meaning 1 Type-3 LSA is allowed in a Total Stub area.

Not-so-stubby Same as a stub area, so not allowing external routes (Type 5 LSA’s). But do allowing ASBR’s in the area. The result is that an ASBR within a not-so-stubby area (NSSA) will inject Type 7 LSA’s in the NSSA. These will be converted to a regular Type 5 on the ABRs of that area.

Totally Not-so-stubby Same as a not-so-stubby area, so not allowing Type 5 LSA’s to exist in the area, but allowing ASBR’s. Then Type-3 LSA’s are also not allowed as in a Totally Stub area. This means a default route is injected in the area to ensure reach ability to the rest of the network.



The drawing below illustrates the above table. Where ASBR’s can be placed in the network. Remind that area 0 can never be altered to a stub or not-so-stub area! What is not explained in the drawing, but is possible is that ABR’s are ASBR’s at the same time.

Now back to the question. The question states the following characteristics of area 1:

• No Type 5 LSA

o This means that a Stub area needs to be used

• No Type 4 LSA

o This is also included in the Stub area. As there are no Type 5 LSA’s in the area, it’s also not required to have a Type 4 LSA (reach ability to the ASBR) in the area

• No Type 3 LSA

o This points to a Totally Stub area, where all Type 3 LSA’s are filtered from crossing the ABRs and a Default route is advertised in to the area. This would solve the question. Except that the Default route that is injected into a Totally Stub area is still a Type 3 LSA. The question clearly states that this is not allowed

We can solve this behavior. Within a Totally NSSA area we do not allow Type 3,4 and 5 LSA’s. So it’s equal to a Totally Stub area. The difference is that we can introduce a new type of LSA in the area which



is a Type 7 for the External routes that are injected when the NSSA area contains an ASBR. These LSA’s are interpreted by every router in the area and therefore they install those external routes from the ASBR in the NSSA area.

What we can do at the ABRs is to ensure that the Default route they inject into the Totally NSSA area is not a Type 3 LSA (as it would be by default on IOS). It can be a Type 7 LSA. Be aware that you need to configure this and this is not supported on every IOS release as well. On NX-OS however it’s the default!

The configuration for this is relatively easy and is easy to spot with use of the question mark.

SW1-1

SW1-1(config)# router ospf IPEXPERT SW1-1(config-router)# area 1 ? authentication Enable authentication for the area default-cost Specify default-cost for default summary LSA filter-list Filter prefixes between OSPF areas nssa Configure area as NSSA range Configure an address range for an area stub Configure area as a stub virtual-link Define a virtual link and its parameters SW1-1(config-router)# area 1 nssa ? <CR> default-information-originate Originate Type-7 default LSA into NSSA area

no-redistribution Do not send redistributed LSAs into NSSA area

no-summary Do not send summary LSAs into NSSA area translate Translate LSA SW1-1(config-router)# area 1 nssa no-summary ? <CR> default-information-originate Originate Type-7 default LSA into NSSA area

no-redistribution Do not send redistributed LSAs into NSSA area

SW1-1(config-router)# area 1 nssa no-summary default-information-originate

SW1-2

SW1-2(config)# router ospf IPEXPERT SW1-2(config-router)# area 1 nssa no-summary default-information-originate

On the other routers within the area we do not have to configure any of the default route information. What’s important for the adjacency to come up is that the area is configured as an NSSA area. This is advertised within the adjacency packets. Therefore we only require the following information on the other switches.



SW2

SW2(config)# router ospf IPEXPERT SW2(config-router)# area 1 nssa

SW3

SW3(config)# router ospf IPEXPERT SW3(config-router)# area 1 nssa

Before applying the NSSA configuration the routing table shows more specific routes within area 1.

SW2(config-if)# sh ip route ospf IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 198.18.0.11/32, ubest/mbest: 1/0 *via 198.18.122.11, Eth2/1, [110/41], 00:02:53, ospf-IPEXPERT, inter 198.18.0.13/32, ubest/mbest: 1/0 *via 198.18.122.11, Eth2/1, [110/81], 00:02:38, ospf-IPEXPERT, inter 198.18.113.0/24, ubest/mbest: 1/0 *via 198.18.122.11, Eth2/1, [110/80], 00:02:53, ospf-IPEXPERT, inter SW2(config-if)#

After applying the configuration every route is summarized into the default route.

SW2(config-router)# sh ip route ospf IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 0.0.0.0/0, ubest/mbest: 1/0 *via 198.18.122.11, Eth2/1, [110/41], 00:00:05, ospf-IPEXPERT, inter SW2(config-router)#

When checking the OSPF database, there are no Type 3, 4 or 5 LSA’s visible so we accomplished this question!

SW2# sh ip ospf database OSPF Router with ID (198.18.0.2) (Process ID IPEXPERT VRF default) Router Link States (Area 0.0.0.1) Link ID ADV Router Age Seq# Checksum Link Count 198.18.0.2 198.18.0.2 215 0x8000000a 0xcff7 2 198.18.0.11 198.18.0.11 215 0x80000009 0xf0a5 1 Network Link States (Area 0.0.0.1) Link ID ADV Router Age Seq# Checksum 198.18.122.11 198.18.0.11 216 0x80000002 0xa0b1



Type-7 AS External Link States (Area 0.0.0.1) Link ID ADV Router Age Seq# Checksum Tag 0.0.0.0 198.18.0.11 542 0x80000002 0x1dcf 0

The final question of this task states to ensure that all OSPF routers do not attract traffic just after start-up of the devices. This should last for 2 minutes. The command we are looking for is the max-metric command, where you can add several values of how this max metric is configured.

The result is that once the router boots up and OSPF adjacencies come online. The router will advertise everything with a maximum metric value until the timer expires. Default this timer is 600 seconds, but is now adapted to 120.

SW1-1

SW1-1(config-if)# router ospf IPEXPERT SW1-1(config-router)# max-metric router-lsa on-startup 120 New command will take effect at next reload

SW1-2

SW1-2(config-if)# router ospf IPEXPERT SW1-2(config-router)# max-metric router-lsa on-startup 120

SW1-3


SW1-4


SW2

SW2(config-if)# router ospf IPEXPERT SW2(config-router)# max-metric router-lsa on-startup 120

SW3

SW3(config-if)# router ospf IPEXPERT SW3(config-router)# max-metric router-lsa on-startup 120

And task 4 is finished!



Task 5: Redistribution, BFD and ECMP Redistribution is the process to leak information between different dynamic routing protocols. This is done to ensure a full reach ability in your whole network. Chances are that you will not run into situations where you need to redistribute between 2 IGP protocols. In real life scenarios redistribution is more commonly done between BGP and an IGP.

Pay close attention when using redistribution in your topology. You can easily run into issues and cause routing loops in the network. This is especially the case for protocols without distinction in route preference / administrative difference like OSPF. The active route is the one with the lowest Administrative Distance. The following table describes the default values which can be changed per protocol. The change can be done for certain routes, certain upstream routers or types of routes (internal/external/local).

Source Administrative Distance

Connected 0

Static 1

EIGRP Summary 5

External BGP (EBGP) 20

EIGRP 90

OSPF 110

IS-IS 115

RIP 120

External EIGRP 170

Internal BGP (iBGP) 200

Local BGP 220

Unreachable 255

Some protocols distinct routes that are internal to their own protocol and prefixes that were learnt via redistribution. OSPF does not do this by default, but can be configured to ensure Type 5 LSA prefixes will have another Administrative Distance attached. In most cases this is even recommended. It’s easy to get a routing loop when there is no distinction between the administrative distances for internal and external routes.



In our network we will also run into some issues and these will cause issues (no connectivity issues). With a normal redistribution the EIGRP learned routes will be learned back by the other router back into EIGRP. The mechanism that EIGRP uses in this case is that Externally learned routes will have a higher administrative distance attached to the routes and therefore select their primary internal routes as the ones where traffic will be send to.

Within the OSPF network there will be issues with routes learning 2 destinations.

We first configure the switches for redistribution on 2 positions in the network. We are required to configure a route-map for the redistribution to ensure only routes are advertised that you really want to.

The drawing below illustrates what is about to happen when we do not enter some filtering information. To ensure that routes are not redistributed back their to their own protocol.

Now we configure the switches with filtering and tagging in place so that routes that are advertised to either EIGRP or OSPF are tagged with a value equal to the Administrative Distance (for easier recognition, but this could be any value). On the next redistribution point this value gets rejected and others get tagged. This way we have a very easy way to filter routes and ensure that we do not cause loops in our redistribution scenario.



SW1-2

SW1-2(config)# route-map E->O deny 10 SW1-2(config-route-map)# match tag 110 SW1-2(config-route-map)# route-map E->O permit 20 SW1-2(config-route-map)# set tag 90 SW1-2(config-route-map)# exit SW1-2(config)# router ospf IPEXPERT SW1-2(config-router)# redistribute eigrp IPEXPERT route-map E->O SW1-2(config-router # SW1-2(config-if)# route-map O->E deny 10 SW1-2(config-route-map)# match tag 90 SW1-2(config-route-map)# route-map O->E permit 20 SW1-2(config-route-map)# set tag 110 SW1-2(config-route-map)# exit SW1-2(config)# router eigrp IPEXPERT SW1-2(config-router)# redistribute ospf IPEXPERT route-map O->E

SW1-4

SW1-4(config)# route-map E->O deny 10 SW1-4(config-route-map)# match tag 110 SW1-4(config-route-map)# route-map E->O permit 20 SW1-4(config-route-map)# set tag 90 SW1-4(config-route-map)# exit SW1-4(config)# router ospf IPEXPERT SW1-4(config-router)# redistribute eigrp IPEXPERT route-map E->O SW1-4(config-router # SW1-4(config-if)# route-map O->E deny 10 SW1-4(config-route-map)# match tag 90 SW1-4(config-route-map)# route-map O->E permit 20 SW1-4(config-route-map)# set tag 110 SW1-4(config-route-map)# exit SW1-4(config)# router eigrp IPEXPERT SW1-4(config-router)# redistribute ospf IPEXPERT route-map O->E



The drawing below illustrates what is happening in the network.

Routes being redistributed between protocols and re-redistributing routes back into their original protocol. Due to the intelligence in most protocols this will not cause any harm and traffic will be forwarded. Still it will cause a lot of ‘dirt’ in the routing protocol databases.



Now we have successful redistribution and have a full reach ability throughout the network without any loops!

The next couple tasks are regarding the Bidirectional Forwarding Detection (BFD) protocol. This protocol ensures a very fast detection that a neighbor router is down, regardless of link status.

Usually the link status is used for detecting a neighbor down event. This is not reliable in situations where for example a Ethernet circuit or DWDM circuit, without propagation of link-loss to client ports, the routers will not know if something happens within the circuit.

The BFD protocol is the Cisco best practice for a fast detection mechanism. The protocol itself is just a very basic Layer 3 protocol which sends very fast Hello messages. Once a certain threshold (by default 3) is reached a trigger is given. The BFD protocol is independent of the routing protocol and can actually be used for any protocol when supported (for example HSRP). Multiple protocols can be attached to the same BFD session, making it a very lightweight protocol for detecting these neighbor loss events as only a single session is running even when multiple protocols are used across the link between the 2 routers.

The above illustration shows the BFD session being totally independent of the OSPD adjacency running on the router. Once the BFD session is detected to be offline, the OSPF protocol is signaled that this specific router has gone offline and therefore OSPF cuts of the adjacency right away, before waiting on its own Holddown-timer thresholds. Multiple protocols can be signaled via this single BFD session.

One additional advantage of the BFD protocol is that it’s so simple, that all Hello packets are generated and handled within the Hardware ASICs. The Nexus 7000 and Nexus 5500 interfaces are all able to handle BFD within the hardware and therefore BFD has no impact on the CPU load of the control-plane. Sub-second hello timers in OSPF due have impact on the CPU load!



As the BFD protocol is just an IP protocol, it can also be secured, just like the mainstream dynamic routing protocols with an MD5 hash. Detection is by default using a hello timer of 300ms and a threshold of 3 missed hello packets, before declaring a neighbor down.

In this case we only need to ensure that failure detection of SW1-2 happens very fast. We will use the BFD protocol for this. We first need to configure the BFD session under the interface and after that we configure the routing protocol to converge along with the BFD protocol. As the BFD protocol is also a Layer 3 protocol, we need to enable it on the NX-OS device first.

SW1-2

SW1-2(config)# feature bfd Please disable the ICMP redirects on all interfaces running BFD sessions using the command below 'no ip redirects '

We need to configure no ip redirects as a best practice. This has to do that it might be possible for an interface to return BFD hellos with a redirect (when not properly configured). This might cause that a lot of packets are send and need to be answered with an ICMP message. All ICMP traffic is handled by the control-plane and not in the ASIC as BFD messages are. Therefore it’s highly recommended to disable this.

SW1-2(config-if)# interface Vlan123 SW1-2(config-if)# no ip redirects SW1-2(config-if)# bfd interval ? <50-999> TX interval in milliseconds SW1-2(config-if)# bfd interval 100 ? min_rx Minimum RX interval SW1-2(config-if)# bfd interval 100 min_rx ? <50-999> RX interval in milliseconds SW1-2(config-if)# bfd interval 100 min_rx 100 ? multiplier Configure detect multiplier for bfd sessions SW1-2(config-if)# bfd interval 100 min_rx 100 multiplier 3 SW1-2(config-if)# interface Vlan132 SW1-2(config-if)# no ip redirects SW1-2(config-if)# bfd interval 100 min_rx 100 multiplier 3 SW1-2(config-if)# interface Vlan112 SW1-2(config-if)# no ip redirects SW1-2(config-if)# bfd interval 100 min_rx 100 multiplier 3 SW1-2(config-if)# interface Vlan211 SW1-2(config-if)# no ip redirects SW1-2(config-if)# bfd interval 100 min_rx 100 multiplier 3 SW1-2(config-if)# interface Vlan124 SW1-2(config-if)# no ip redirects SW1-2(config-if)# bfd interval 100 min_rx 100 multiplier 3



There are a number of things that need to be configured when configuring BFD sessions. You immediately require configuring the timers of the session. When you look a little further to the questions in this task it’s stated that the sessions should detect failures within 300ms. Therefore we are configuring hello-intervals of 100ms and a multiplier of 3. Meaning the device can miss 3 hello’s before declaring the neighbor down. With hellos of 100ms, this occurs at a maximum of 300ms, which after the connected protocols are signaled.

What remains is that the OSPF and EIGRP protocols need to be configured for the BFD protocol and we need to enable the interfaces that we want to use the protocol on.

If we would be configuring BGP, we would configure these options on the Neighbor sessions. The BFD session configuration would still be done under the interface. Remember to configure the correct routing protocols and also the correct interface configuration!

SW1-2

SW1-2(config)# router ospf IPEXPERT SW1-2(config-router)# bfd SW1-2(config-router)# router eigrp IPEXPERT SW1-2(config-router)# bfd SW1-2(config-router)# interface Vlan132 SW1-2(config-if)# ip ospf bfd SW1-2(config-if)# interface Vlan123 SW1-2(config-if)# ip ospf bfd SW1-2(config-if)# interface Vlan112 SW1-2(config-if)# ip ospf bfd SW1-2(config-if)# interface Vlan211 SW1-2(config-if)# ip ospf bfd SW1-2(config-if)# interface Vlan124 SW1-2(config-if)# ip eigrp IPEXPERT bfd

Now we configured only 1 router. BFD consists of that both sides need to be configured before everything works. We now configure the other sides of the interfaces that connect to SW1-2.

SW1-1

SW1-1(config)# feature bfd SW1-1(config)# router ospf IPEXPERT SW1-1(config-router)# bfd SW1-1(config-router)# exit SW1-1(config-if)# interface Vlan112 SW1-1(config-if)# no ip redirects SW1-1(config-if)# bfd interval 100 min_rx 100 multiplier 3 SW1-1(config-if)# ip ospf bfd SW1-1(config-if)# interface Vlan211 SW1-1(config-if)# no ip redirects SW1-1(config-if)# bfd interval 100 min_rx 100 multiplier 3 SW1-1(config-if)# ip ospf bfd



SW3

SW3(config)# feature bfd SW3(config)# router ospf IPEXPERT SW3(config-router)# bfd SW3(config-router)# exit SW3(config-if)# interface Vlan123 SW3(config-if)# no ip redirects SW3(config-if)# bfd interval 100 min_rx 100 multiplier 3 SW3(config-if)# ip ospf bfd SW3(config-if)# interface Vlan132 SW3(config-if)# no ip redirects SW3(config-if)# bfd interval 100 min_rx 100 multiplier 3 SW3(config-if)# ip ospf bfd

SW1-4

SW1-4(config-router)# router eigrp IPEXPERT SW1-4(config-router)# bfd SW1-4(config-router)# exit SW1-4(config)# interface Vlan124 SW1-4(config-if)# ip eigrp IPEXPERT bfd

The questions also state that the BFD session between SW1-2 and SW3 should be secured using an SHA-1 hash. We can configure this under the interfaces. Don’t forget that there are 2 interfaces and therefore 2 sessions between the switches, which both need to be secured.

SW1-2

SW1-2(config)# interface Vlan123 SW1-2(config-if)# bfd authentication keyed-sha1 keyid 1 key IPexpertSecure SW1-2(config)# interface Vlan132 SW1-2(config-if)# bfd authentication keyed-sha1 keyid 1 key IPexpertSecure

SW3

SW3(config)# interface Vlan123 SW3(config-if)# bfd authentication keyed-sha1 keyid 1 key IPexpertSecure SW3(config)# interface Vlan132 SW3(config-if)# bfd authentication keyed-sha1 keyid 1 key IPexpertSecure

Which completes task 5!



Task 6: Layer 3 switching features The final task regarding layer 3 routing protocols is about miscellaneous features that are possible with the NX-OS. These features are again easily found in the Unicast Routing Guide and Interfaces Configuration Guide in the online documentation that is also available to you when you are doing the actual exam.

The first task is to configure a static ARP mapping on a specific interface. In Classical IOS this configuration was done in Global Configuration mode. Within NX-OS this is done under the interface where the ARP entry is reachable. This means that it’s not necessary to include an interface anymore in the ARP command, but you only need to configure which mac addresses are found on which interface.

SW1-1

SW1-1(config-if)# int vlan 112 SW1-1(config-if)# ip arp ? A.B.C.D IP address gratuitous Gratuitous timeout ARP timeout SW1-1(config-if)# ip arp 198.18.112.24 ? E.E.E MAC address (Option 1) EE-EE-EE-EE-EE-EE MAC address (Option 2) EE:EE:EE:EE:EE:EE MAC address (Option 3) EEEE.EEEE.EEEE MAC address (Option 4) SW1-1(config-if)# ip arp 198.18.112.24 abcd.1234.5678 SW1-1(config-if)# sh run int vlan 112 !Command: show running-config interface Vlan112 !Time: Mon Sep 24 01:40:00 2012 version 5.1(5) interface Vlan112 no shutdown ip address 198.18.112.11/24 ip arp 198.18.112.24 ABCD.1234.5678 SW1-1(config-if)#

We configured the static ARP entry directly under the interface and completed the question.

Next is a bit cryptic description regarding Gratuitous ARPs. With this mechanism the switch will broadcast an ARP entry for its own IP address and checks if anybody replies with a different MAC address (different than its own local interface). After which it can send a warning about a duplicate IP address on the segment.

This mechanism is enabled by default. Though one configuration option, to actively update its cache is disabled. This is what the question asks for.



SW2

SW2(config-if)# int Ethernet1/5 SW2(config-if)# ip arp gratuitous ? request Enable/Disable sending grat. arp request when duplicate address detected

update Enable/Disable arp cache updates for gratuitous arp SW2(config-if)# ip arp gratuitous

The next question is also a bit cryptic which means you have to read the Unicast Routing configuration guide pretty well to find this. The Nexus 7000 keeps a memory reservation for outstanding ARP entries in its Forwarding Information Base (FIB). This memory is cleared after a while. This is done to protect the memory from overflowing with ARP entries when an attack is started (flooding of MAC addresses by an attacker to overflow the memory of switches).

This is called ARP throttling or Glean throttling (Glean ARPs are incomplete ARP entries waiting for a reply) and is configured using special hardware commands.

SW1-1

SW1-1(config)# hardware ? access-list Access Control List ejector Card ejector functionality fabrics Fabrics supported in the switch forwarding Forwarding information ip IP ipv6 (no abbrev) IPv6 proxy Proxy Forwarding Information rate-limiter Configure Rate-Limiter for packets forwarded to supervisor

SW1-1(config)# hardware ip ? glean Glean verify Enable IPv4 and some IPv6 packet validation checks in hardware SW1-1(config)# hardware ip glean throttle ? <CR> maximum Maximum number of entries syslog Threshold for syslog for number of packets hitting the entry timeout Timeout SW1-1(config)# hardware ip glean throttle SW1-1(config)# hardware ip glean throttle maximum 2750

Remind to first enable the feature and then change the maximum.

The final question is regarding RFC 1191 to be enabled on all switches. A quick search through the documentation (Ctrl-F or Cmd-F) learns that this is related to TCP Path-MTU-Discovery! Easily enabled on all switches with a single command.



SW1-1

SW1-1(config)# ip tcp path-mtu-discovery

SW1-2


SW1-3


SW1-4


SW2

SW2(config)# ip tcp path-mtu-discovery

SW3

SW3(config)# ip tcp path-mtu-discovery

Task 6 is completed! Now before moving on with the next Task it’s recommended to take a little break as the next task requires loading a new initial configuration on the switches. This will take some time, approximately 10-15 minutes, as the switches need to be reloaded to load a new VDC configuration with other mappings and pre-configuration.



Task 7: FabricPath and OTV This final task of the Chapter is regarding very cool technologies called FabricPath (FP) and Overlay Transport Virtualization (OTV). These technologies are very specific to the Nexus family and even there dedicated to some platforms.

FabricPath is supported on the following platforms:

• Nexus 5500 series

• Nexus 7000 series with F1-linecards

• Requires Enhanced Layer 2 license on both platforms

OTV is supported on the following platforms:

• Nexus 7000 series with M1-linecards

• Requires Transport Services license

We purposely excluded newer generation linecards (F2 and M2) as these are not in the hardware blueprint of the CCIE Data Center and therefore not in scope of this workbook.

The questions are very basic formulated in this question. This is due to the fact that the technology underneath is quite complex and very groundbreaking to begin with. Still Cisco made a great effort in making this one of the easiest things to configure on the box.

FabricPath and OTV are literally just a few commands to enable and get it up and running, just the way you want. Modifications can be done to the protocols, but these are limited. Also in scope of the CCIE Data Center, the amount of topologies that can be created with these technologies is limited.

Before you know what exactly to configure to stretch the layer 2 VLAN across multiple datacenters over a Fabric Path and OTV network, you need to know some structure of the protocols.

FabricPath

The FabricPath protocol is a Cisco proprietary protocol for making Layer 2 routed instead of switched. This is done by eliminating ARP from the network and implementing IS-IS as routing protocol to advertise reach ability in the network. This way there is no broadcast traffic occurring when it’s not necessary. FabricPath certainly supports all various ways of Multicast and Broadcast traffic, but it’s sent in a very optimal way.

The next big step is that it’s not necessary for all switches to have knowledge about all MAC addresses in the network. Therefore a mechanism is introduced called Conversational MAC learning. This mechanism is only available on the F-linecards of the Nexus 7000 switches and on the Nexus 5500 series switches. It means that switches only remember the MAC address information, once there is a bidirectional traffic flow. Instead of just learning the MAC address once a packet is seen from it.



The IS-IS routing protocol is totally invisible for the FabricPath protocol and for the CCIE configuring it. Only when troubleshooting is necessary, certain commands related to IS-IS are required to check if adjacencies are up and that topology information is exchanged.

The FabricPath protocol works with an additional encapsulation on the packet (the protocol kind of looks like MPLS). The encapsulation is a ‘FabricPath’ encapsulation. Which means that it’s an adjusted Ethernet frame that is added in front of the packet. FabricPath is also called a “MAC-in-MAC” protocol. This header is modified for FabricPath and therefore it’s also NOT supported to run FabricPath over an Ethernet circuit or to have Switches in between the path. The connection between 2 FabricPath devices needs to be a direct (dark) fiber connection to support it.

There is an IEEE initiative called TRILL that is very much alike FabricPath in many ways. The main difference is that TRILL does use a standard Ethernet header as the encapsulation and therefore does support that switches or Ethernet circuits are in between the TRILL devices. This enables a lot more use of the protocol and it’s not necessary to have all TRILL enabled devices in your network. For example you could upgrade your access or distribution switches to TRILL and have your core still run as standard Ethernet switches. They switch the TRILL traffic and do not have any awareness of what’s inside the packet and what kind of high advanced technology is happening on the Edge of the network.

All this sounds very familiar when you are used to work with MPLS networks. Where also the core is a very ‘dumb’ device that just forwards packets to the right interfaces and that all intelligence of the network is inside the network edge. This also accounts for both FabricPath and TRILL.

Below is a drawing explaining the traffic flow in-case of a known-unicast, meaning that ARPs have successfully passed along. In case of an unknown unicast, the ARP request of the access device or end-host is flooded throughout the network. This is done in an optimal way using so-called Broadcast Trees. Every edge switch (called leaf switches in FabricPath) uses a dedicated tree of switches, calculated using the SPF algorithm of IS-IS where it will forward the traffic along. By using the Broadcast tree, every switch in the network receives the packet only once and never multiple times, as could be the case with ARPs and normal broadcasts.

The destination host will reply to the ARP and send it back along the same distribution / broadcast tree through the FabricPath network. Once traffic starts being sent, the network learns about this flow and will remember the traffic flow in its tables (conversational learning). In that case the first leaf switch knows about where the packet should go and sends it unicast to the next switch. Where it’s decapsulated and another lookup is done where the destination MAC address is located on that particular switch. The packet is then forwarded out in plain Ethernet to the end-host.

The drawing explains this traffic flow in a very easy way so the flow is known to you.



The drawing explains the following flow:

1. PC 1 sends a packet to PC 2 across a standard Ethernet segment (VLAN)

2. Leaf 1 receives the packet and sees that it’s destination (PC 2) is on Leaf 2

3. Leaf 1 encapsulates the packet with a FabricPath header, with a destination address of Leaf 2 and a source address of Leaf 1

4. The FabricPath network forwards the packet based on the FabricPath header to Leaf 2 without looking at the rest of the packet headers

5. Leaf 2 decapsulates the packet and does a lookup on the Destination of the Ethernet frame inside of the FabricPath frame

6. Leaf 2 forwards the packet out of its local interface to PC 2

By using this method of traffic forwarding the core network has no awareness of what’s happening inside of the FabricPath frame and can therefore have a very limited FIB table. The core network only needs to know the way to the destination leaf switches, which is advertised using a modified version of IS-IS.

Now on to the configuration. We also configure some SVI’s to test the configuration and ensure hosts on VLAN 666 can reach the rest of the network by using FabricPath. As you can see in the network drawing, there are no direct connections between the switches with a Classical Ethernet encapsulation. So the FabricPath network needs to work before replies are received from other hosts on the network.



SW2

SW2(config)# vlan 666 SW2(config-vlan)# exit SW2(config)# feature interface-vlan SW2(config)# interface vlan666 SW2(config-if)# ip address 198.18.66.1/24 SW2(config-if)# no shutdown SW2(config-if)# exit

SW3

SW3(config)# vlan 666 SW3(config-vlan)# exit SW3(config)# feature interface-vlan SW3(config)# interface vlan666 SW3(config-if)# ip address 198.18.66.2/24 SW3(config-if)# no shutdown SW3(config-if)# exit

SW1-3

SW1-3(config)# vlan 666 SW1-3(config-vlan)# exit SW1-3(config)# feature interface-vlan SW1-3(config)# interface vlan666 SW1-3(config-if)# ip address 198.18.66.3/24 SW1-3(config-if)# no shutdown SW1-3(config-if)# exit

SW1-4

SW1-4(config)# vlan 666 SW1-4(config-vlan)# exit SW1-4(config)# feature interface-vlan SW1-4(config)# interface vlan666 SW1-4(config-if)# ip address 198.18.66.3/24 SW1-4(config-if)# no shutdown SW1-4(config-if)# exit

FabricPath only works on the F1 linecards. The encapsulation and FabricPath VLANs can only live on the F1 linecard and can never be configured on an M1 linecard. However, the routing feature of the M1 linecard can be ‘borrowed’ by the F1 linecards. Traffic entering on the FabricPath interface will be directed over the Fabric towards the M1 linecard to ensure that layer 3 traffic is properly handled by the switch.

On the Nexus 5500 series, a similar configuration is done by using the Layer 3 daughter module that is also inserted like an interface in the router. Therefore the layer 2 only ASICs can forward traffic to the layer 3 module and comes back again to be re-encapsulated into the FabricPath network.



SW2

SW2(config)# SW2(config)# feature-set fabricpath SW2(config)# vlan 666 SW2(config-vlan)# mode fabricpath

SW3

SW3(config)# SW3(config)# feature-set fabricpath SW3(config)# vlan 666 SW3(config-vlan)# mode fabricpath

SW1-3

SW1-3(config)# SW1-3(config)# feature-set fabricpath SW1-3(config)# vlan 666 SW1-3(config-vlan)# mode fabricpath

SW1-4

SW1-4(config)# SW1-4(config)# feature-set fabricpath SW1-4(config)# vlan 666 SW1-4(config-vlan)# mode fabricpath

SW1-1

SW1-1(config)# SW1-1(config)# feature-set fabricpath

SW1-2

SW1-2(config)# SW1-2(config)# feature-set fabricpath

With enabling the fabricpath feature-set, you enable a whole suit of protocols in a single command. For example IS-IS is started and the FP protocol itself. What’s also started is an automatic switch ID selection. It’s possible to manually configure the switch ID’s (where traffic is forwarded to in the FP frames), but the automatic selection will pick values which are unique in the network. Therefore we will leave it as automatic in our network.

We also configure the VLAN which is to be extended through the FabricPath network (VLAN 666) as a FabricPath VLAN. This is to configure the VLAN that it’s allowed to travel the FabricPath network. By default all VLANs remain in so-called ‘CE’ (Classical Ethernet) mode, so will never cross the FabricPath network.



Now what’s left is the configuration of the interfaces that are pointing to the other FabricPath switches. Ensure that you do not configure any other interfaces with this encapsulation as a lot of processes will be configured under this interface and for example IS-IS will start attempting to make adjacencies on these interfaces.

SW2

SW2(config)# SW2(config)# interface Ethernet1/1 SW2(config-if)# switchport mode fabricpath SW2(config)# interface Ethernet1/5 SW2(config-if)# switchport mode fabricpath

SW3

SW3(config)# SW3(config)# interface Ethernet1/1 SW3(config-if)# switchport mode fabricpath SW3(config)# interface Ethernet1/5 SW3(config-if)# switchport mode fabricpath

SW1-3

SW1-3(config)# SW1-3(config)# interface Ethernet2/9 SW1-3(config-if)# switchport mode fabricpath SW1-3(config)# interface Ethernet2/11 SW1-3(config-if)# switchport mode fabricpath

SW1-4


SW1-1


SW1-2




After a while our interface should come up, IS-IS adjacencies have formed and our FabricPath network is up and running!

The next and final step of this chapter is to configure the inter-datacenter communication. This is accomplished using another Nexus 7000 specific Cisco Proprietary protocol called Overlay Transport Virtualization.

Overlay Transport Virtualization

The OTV protocol works, from a data plane perspective, very similar as FabricPath works. The next-hop of the MAC address-table is changed from a destination interface to a Destination IP address instead of a Destination MAC address as it would be in a FabricPath network.

The OTV protocol uses IP encapsulation instead of Ethernet. Different from FabricPath is that OTV is created to run over standard IP networks. This means that the only requirement you have between data centers to extend layer 2 between them is a Layer 3 connection with IPv4 routing. Any IP connection between the data centers is enough to ensure that OTV can function.

While FabricPath is called a MAC-in-MAC protocol. OTV can be called a MAC-in-IP protocol.

OTV is designed to support numerous data centers that share the same Overlay network. Meaning that one VLAN can be extended across many different data centers. This means that the OTV network is optimized to work in a Multicast enabled environment. The configuration becomes quite complex when unicast is used. In our case with only 2 devices and a directly connected link we will certainly use Multicast for our solution.

The unicast model works with a central node (or 2) that will be in charge of maintaining adjacencies with all sites for spreading all control-plane information throughout the Overlay network. The model looks a bit like the OSPF DR and BDR principle or the BGP route reflector principle, where devices only maintain 2 connections to these devices and with nobody else. These primary devices will exchange the information about the network to all nodes.

As previously mentioned the protocol works very similar to FabricPath in traffic flow in the data plane, so we will not describe this in further detail. Only the final traffic flow will be shown in a diagram.

OTV requires a little more configuration than FabricPath, as it has no knowledge of the rest of the network and needs some IP addressing to be configured. Besides that it requires an Overlay network to be configured.

OTV also uses the IS-IS routing protocol to exchange the MAC address information between all nodes. This information is exchanged through Multicast messages. This IS-IS protocol is a different instance than the IS-IS protocol used for FabricPath. This is done to ensure they will never interfere with each other.



The following parameters need to be configured for a successful OTV implementation:

1. IP addressing on the IP interfaces that are connected to the WAN network

2. Multicast needs to be enabled in the network and on the interface (IGMP)

3. A Multicast group needs to be selected for transferring control-plane traffic (control-group)

4. A block of SSM (Source Specific Multicast) groups (8) needs to be selected for transferring multicast and broadcast traffic of the network across the Overlay.

5. VLANs that need to be extended should be known

6. A local VLAN (site VLAN) needs to be selected for local control-plane traffic within the datacenter. Mainly used for split-brain detection and loop prevention. The local VLAN may never be extended to other sites!

7. A site-identifier to ensure the site is uniquely numbered. When using multiple OTV nodes at a single site, this number should match on all devices on that site.

OTV only works on the M1 linecards of the Nexus 7000 series switches. This line card is typically used in core switch scenarios where also the layer 2 to layer 3 border is placed. Meaning that SVI’s are configured on the core switches to terminate layer 3 routing of the datacenter network.

There is one important limitation with OTV! The switch that is configured for OTV may not contain any SVI’s of the VLANs that are extended to other sites! This is a hardware limitation in the M1 line cards. In future software and/or hardware releases of NX-OS and/or linecards it’s expected that this limitation is removed. The current workaround is to use a different VDC on the Nexus 7000 just for OTV extension. In our network we do not have an SVI of the extended VLAN configured on that switch (VDC) therefore we do not run into this limitation.

Time for some configuration!

First we configure the required configuration that will support the OTV protocol.

SW1-1

SW1-1(config)# SW1-1(config)# interface Ethernet1/10 SW1-1(config-if)# no switchport SW1-1(config-if)# ip address 198.18.10.1/24 SW1-1(config-if)# ip igmp version 3 SW1-1(config-if)# no shutdown SW1-1(config-if)# SW1-1(config-if)# vlan 666 SW1-1(config-vlan)# mode fabricpath



SW1-2

SW1-2(config)# SW1-2(config)# interface Ethernet1/17 SW1-2(config-if)# no switchport SW1-2(config-if)# ip address 198.18.10.2/24 SW1-2(config-if)# ip igmp version 3 SW1-2(config-if)# no shutdown SW1-1(config-if)# SW1-1(config-if)# vlan 666 SW1-1(config-vlan)# mode fabricpath

We are told to use the 198.18.10.0/24 subnet for a Layer 3 connection. This will be the only layer 3 connection in our topology, so we can use the entire /24 network if we want. We also enable IGMP to ensure Multicast processing is enabled on the interface. OTV requires version 3 of IGMP to work correctly. At this stage we also enable the VLAN that is about to be extended to the other site. Meanwhile it’s also the FabricPath VLAN as the traffic coming in for this VLAN is coming in via a FabricPath enabled network in our topology.

Next up is the OTV configuration.

SW1-1

SW1-1(config)# SW1-1(config)# feature otv

As with all features. It needs to be enabled first.

SW1-1(config)# otv site-identifier 1 SW1-1(config)# vlan 10 SW1-1(config-vlan)# exit SW1-1(config)# otv site-vlan 10

We are required to configure a site-identifier for OTV. In FabricPath the switch ID’s are automatically selected, but for OTV the site-ID needs to be equal on all nodes within the same data center (site), therefore this needs to be configured manually.

The site-vlan is used for sending local control-plane traffic between the OTV enabled switches. This VLAN may never be extended across the Overlay to other nodes in the OTV enabled network. The information sent in this VLAN is used for loop prevention and split-brain detection. By default VLAN 1 is used for this. It’s highly recommended to change this to a dedicated VLAN. There were no indications in the question which value is required to be used, so we can pick any number. We also don’t have any other OTV enabled switches in this network, so the VLAN is unused in this topology, however required in the configuration.

SW1-1(config)# interface Overlay1 SW1-1(config-if)# otv control-group 239.1.1.1 SW1-1(config-if)# otv data-group 232.11.22.0/28 SW1-1(config-if)# otv join-interface Ethernet1/10



The Overlay interface is the interface used for the encapsulation onto the OTV enabled network. All OTV related configuration is done under this interface.

The control-group is the multicast group that is used for control-plane traffic sent across the OTV network to exchange all information to other OTV nodes. This can be any multicast group that is forwarded correctly through the multicast enabled network.

The data-group is the range of multicast groups, with a maximum of 8, on which the multicast and broadcast traffic of the extended VLANs is sent. These groups should be from the Source Specific Multicast (SSM) range. Again we didn’t have any value given, so we can use any /28 subnet to ensure we have 8 groups available for multicast/broadcast transmission across the OTV network.

The last option in this snippet is the join-interface. This is the IP interface (interface configured with an IP address) that is used for sending the IP encapsulated traffic on. You can only configure 1 Ethernet interface with this command. When resiliency and/or load-balancing is required there are 2 options. The first is to use a Port-channel of M1 linecard interfaces as the join-interface. The second is to use 2 or more Overlay interfaces with different OTV parameters, but with the same VLAN extension configured.

SW1-1(config)# interface Overlay1 SW1-1(config-if)# otv extend-vlan 666 SW1-1(config-if)# no shutdown

The final step in the configuration is to configure the VLANs that are going to be extended to the other data center. In our topology this is VLAN 666 which is extended. All the above mentioned parameters on the Overlay1 interface need to be configured first, before the no shutdown command is issued. When some pieces of the configuration are missing, the system will report an error, or the OTV network will not work properly.

Once the OTV network is discovered and converged. We have extended VLAN 666 across a FabricPath network across 2 data centers interconnected via OTV!



In the final topology, the traffic flow will be as the following diagram illustrates.

The steps that are taken in the life of the packet are:

1. SW2 SVI1 (198.18.66.1) sends an ICMP ping to SW1-4 SVI4 (198.18.66.4) in a standard/plain Ethernet/IP frame

2. The packet is encapsulated by SW2 into a FabricPath / Ethernet frame and sent to the switch where the destination resides.

3. SW1-1 decapsulates the FabricPath header and does a lookup on the destination address (SW1-4 SVI4) and finds an OTV IP address next-hop in its MAC address-table

4. SW1-1 encapsulates the packet in an OTV / IP frame and sends it across the OTV network towards the destination

5. SW1-2 decapsulates the OTV header and performs a lookup on the Ethernet frame. The destination address is a FabricPath Switch ID

6. SW1-2 encapsulates the frame in a FabricPath / Ethernet frame and sends it towards the destination switch ID.

7. SW1-4 decapsulates the FabricPath header from the packet and performs a lookup on the Ethernet frame. The destination interface is it’s local SVI

8. The local SVI receives the Plain Ethernet frame and arrived at its destination.



When ICMP pings between all SVI interfaces in the network are successful the task has been completed and the chapter has been completed successfully!

The next chapter will be an in-depth overview of NX-OS High Availability features. FabricPath will come back in this chapter to demonstrate the possibility to connect hosts in a resilient way to 2 leaf switches.

Take a break and freshen up before continuing to the next chapter!



Chapter 4: Data Center Networking High Availability (NX-OS)

Chapter 4: Data Center Networking High Availability (NX-OS) is intended to let you be familiar with the NX-OS High Availability features on the Nexus platforms to create a high available network. Various types of deployments of Port-channels and Virtual Port-channels are discussed in this chapter. The second part of this chapter focuses on First Hop Redundancy Protocols (FHRPs) and High Available features of dynamic routing protocols. The third part focuses on a special implementation of virtual port-channels in FabricPath networks.

We highly recommend creating your own diagram at the beginning of each lab so you are able to draw on your own diagram, making it much easier when you step into the real lab.

Multiple topology drawings are available for this chapter.



Solutions

Task 1: Topology set-up The first task of this high availability chapter is to configure the topology for this particular chapter. The Nexus 7000 is pre-configured using VDC’s (Virtual Device Contexts) so 2 switches are simulated out of 1 physical switch.

The layer 3 drawing is in this case more important as the VLANs and SVI’s need to be created out of this drawing. The drawing shows a number of VLANs that will be running between the routers. Keep in mind that the layer 2 topology can be very different from the layer 3 topology.

Especially in this chapter we will be creating several layer 2 port-channels (link bundles) that are crossing all of the switches. Our layer 3 topology however, looks very different than this physical layer 2 topology.

Drawing 2 below illustrates which VLANs and which SVIs need to be created on all switches.



The following configuration is required to finish this initial task of this chapter:

SW1-1

SW1-1(config)# vlan 112, 111, 121, 222 SW1-1(config-vlan)# exit SW1-1(config)# feature interface-vlan SW1-1(config)# SW1-1(config)# interface vlan 112 SW1-1(config-if)# ip address 198.18.112.11/24 SW1-1(config-if)# no shutdown SW1-1(config)# interface vlan 111 SW1-1(config-if)# ip address 198.18.111.11/24 SW1-1(config-if)# no shutdown SW1-1(config)# interface vlan 121 SW1-1(config-if)# ip address 198.18.121.11/24 SW1-1(config-if)# no shutdown SW1-1(config)# interface vlan 222 SW1-1(config-if)# ip address 198.18.222.11/24 SW1-1(config-if)# no shutdown SW1-1(config)# interface Loopback0 SW1-1(config-if)# ip address 198.18.0.11/32

SW1-2

SW1-2(config)# vlan 123, 111, 121, 222 SW1-2(config-vlan)# exit SW1-2(config)# feature interface-vlan SW1-2(config)# SW1-2(config)# interface vlan 123 SW1-2(config-if)# ip address 198.18.123.12/24 SW1-2(config-if)# no shutdown SW1-2(config)# interface vlan 111 SW1-2(config-if)# ip address 198.18.111.12/24 SW1-2(config-if)# no shutdown SW1-2(config)# interface vlan 121 SW1-2(config-if)# ip address 198.18.121.12/24 SW1-2(config-if)# no shutdown SW1-2(config)# interface vlan 222 SW1-2(config-if)# ip address 198.18.222.12/24 SW1-2(config-if)# no shutdown SW1-2(config)# interface Loopback0 SW1-2(config-if)# ip address 198.18.0.12/32

SW2

switch(config)# switchname SW2 SW2(config)# vlan 112 SW2(config-vlan)# exit SW2(config)# feature interface-vlan SW2(config)# SW2(config)# interface vlan 112 SW2(config-if)# ip address 198.18.112.2/24 SW2(config-if)# no shutdown SW2(config)# interface Loopback0 SW2(config-if)# ip address 198.18.0.2/32



SW3

switch(config)# switchname SW3 SW3(config)# vlan 123 SW3(config-vlan)# exit SW3(config)# feature interface-vlan SW3(config)# SW3(config)# interface vlan 123 SW3(config-if)# ip address 198.18.123.3/24 SW3(config-if)# no shutdown SW3(config)# interface Loopback0 SW3(config-if)# ip address 198.18.0.3/32

After this initial configuration the task is finished and we continue with configuring the layer 2 infrastructure of this topology.

Task 2: Port-Channels The second task will be about configuring port-channels in a traditional way. The technology for bundling links has been around for al long time. The technology ensures that multiple physical connections are bundled into a single logical connection. This bundling is done for multiple reasons:

• Redundancy

o By separating the link bundle across multiple line-cards within a switch, redundancy can be achieved. When links are removed or added to a bundle, this does not result in loss of traffic and is therefore a very good failover mechanism.

• Increased bandwidth

o The increased bandwidth is achieved by load balancing traffic over the 2 or more available connections in the bundle.

• Scalability

o Another reason for this is that when adding links to an existing bundle, it does not result in downtime. Therefore, a bandwidth capacity increase of connections is very easily done and does not require a lot of configuration. This makes the connection between devices very scalable.

• Cost

o When more bandwidth is necessary between 2 devices, you could also upgrade the link from 1Gbps to 10Gbps for example. However the cost associated with an upgrade to such a high bandwidth connection is relatively very high, compared to adding a second 1Gbps connection.



o There is always a turning point in which adding more connections to a bundle will cost more (in OPEX for example) then moving over to a single higher bandwidth connection.

• Loop prevention

o Link bundles are a single connection and therefore there is no reason to run spanning-tree between the individual members.

The bundling of physical links into a single logical one has many names in different types of documentation. Vendors also use different naming Some of the names are:

• Port-channels

• EtherChannels

• Link Aggregation Groups (LAGs)

• Trunks (very difficult when people mean different things with the same word)

• Aggregated Ethernets (AE)

• VIFs

• (Link) Bundles

• etcetera

Cisco also uses different naming for their link bundles. Within NX-OS everything has been standardized and is now referred to as Port-Channels. In our workbook we will either be talking about Port-Channels or Link Bundles.

Link bundling is always done between 2 devices. The logical interface is still a connection between device A and device B. The drawing below illustrates the bundling of links between 2 devices:



There are a number of scenarios described in this drawing.

• Port-channel numbers do not need to match

o It’s not necessary that port-channel numbers match on both sides of the connections. In no protocol these numbers are transported. When you are bundling links between different vendors switches the numbering is usually different and sometimes not even possible to match.

o Some hardware platforms might have limitations to the numbering of the link bundles (older Catalyst switches only supported up to 16 port-channels).

• Interfaces in a bundle can be on different line cards in the switch

o Interfaces can be mixed and matched across different line cards. Switches have built-in mechanisms to ensure that packets arrive in the same order (in case of distributed forwarding) as they arrive on the switch.



Dependent on the port-channel configuration, port-channel 3 on Switch A will be a working connection or not

There are a number of compatibility requirements for interfaces to come online within a link bundle. The following list describes the capabilities an interface that need to be matched on all participating interfaces before they can be added to any bundle:

• Link speed

• Duplex setting

• No combined layer 2 / layer 3

• Layer 3 mode

o No layer 3 sub-interfaces may exist on link bundle members

• Layer 2 mode

o Access or trunk mode

o Allowed VLAN list

• MTU size

• Flow-Control settings

• StormControl settings

• Equal type/generation line card

• Equal oversubscription configuration

• No Private VLAN configuration

The switch will notify you when certain parameters on an interface are incorrect for the switch to add it to a bundle. The best practice to do is to use a (previously ‘hidden’) command in NX-OS to erase all interface configurations and reset it to the defaults:

SW1-1# sh run int e1/25 !Command: show running-config interface Ethernet1/25 !Time: Sun Sep 30 08:12:00 2012 version 5.2(4) interface Ethernet1/25 switchport switchport mode trunk switchport monitor



storm-control broadcast level 50.00 storm-control multicast level 50.00 storm-control unicast level 50.00 service-policy type qos input COS-LIMIT service-policy type queuing output SHAPE_MY_QUEUE no shutdown SW1-1# conf t Enter configuration commands, one per line. End with CNTL/Z. SW1-1(config)# default interface Ethernet1/25 SW1-1(config)# sh run int e1/25 !Command: show running-config interface Ethernet1/25 !Time: Sun Sep 30 08:12:35 2012 version 5.2(4) interface Ethernet1/25 SW1-1(config)#

When using this command you are sure that the interface will be added to the bundle without any problems.

Now on to some configuration; the first 2 questions state that a link bundle should be configured between some switches. It’s best to read ahead on some of the next questions as question 3 and 4 state some important information to bring up the link bundle.

Port-Channels are given a number to work on. The logical interface which is created uses this number to be identified throughout the configuration. Numbering is very different per platform. Both platforms used in our topology (Nexus 7000 and Nexus 5000) support up to 4094 port-channel numbers on the switch. It’s not said that the switches also supports that amount of port-channels to become operational, but it’s easier to have your own numbering scheme than to be limited in numbering by the switch capabilities.

SW1-1(config)# interface port-channel ?

<1-4096> Port Channel number

When no number is given, just use your own best practice for it. Ensure that you do not overlap on the same switch and I would use the same number on both sides of the link so you never make mistakes when configuring certain port-channels on different devices.

We will first configure the link bundle between SW1-2 and SW3 as we require additional configuration for the first question.

SW1-2

SW1-2(config)# default interface Ethernet3/5 SW1-2(config)# default interface Ethernet3/6 SW1-2(config)# interface e3/5-6



SW1-2(config-if-range)# channel-group 2 mode on SW1-2(config-if-range)# no shutdown SW1-2(config-if-range)# interface port-channel2 SW1-2(config-if)# switchport SW1-2(config-if)# sw mode trunk SW1-2(config-if)# sw trunk allowed vlan 123 SW1-2(config-if)# no shut SW1-2(config-if)# SW1-2(config-if)# sh run int e3/5 !Command: show running-config interface Ethernet3/5 !Time: Sun Sep 30 08:31:54 2012 version 5.2(4) interface Ethernet3/5 switchport switchport mode trunk switchport trunk allowed vlan 123 channel-group 2 no shutdown SW1-2(config-if)# sh run int e3/6 !Command: show running-config interface Ethernet3/6 !Time: Sun Sep 30 08:31:56 2012 version 5.2(4) interface Ethernet3/6 switchport switchport mode trunk switchport trunk allowed vlan 123 channel-group 2 no shutdown SW1-2(config-if)# do sh run int port-channel2 !Command: show running-config interface port-channel2 !Time: Sun Sep 30 08:32:00 2012 version 5.2(4) interface port-channel2 switchport switchport mode trunk switchport trunk allowed vlan 123 SW1-2(config-if)#

As there is no information about any port-channeling protocol, we will use no protocol and just use the static port-channeling configuration. Now one side of the connection is configured.



The interfaces do not use any port-channeling protocol as mentioned, but are statically configured inside of a port-channel. This means that when both links are up. The interfaces are forced into a port-channel.

Keep in mind that this might cause strange behaviors on the other side of the connection as it will not expect traffic to be load balancing between 2 individual links. This means that you might see errors for MAC address flapping or even get ports that are becoming error disabled (err-disabled) because of a potential loop in the network.

SW3

SW3(config)# default interface Ethernet1/1 SW3(config)# default interface Ethernet1/2 SW3(config)# interface e1/1-2 SW3(config-if-range)# channel-group 2 mode on SW3(config-if-range)# no shutdown SW3(config-if-range)# interface port-channel2 SW3(config-if)# switchport SW3(config-if)# sw mode trunk SW3(config-if)# sw trunk allowed vlan 123 SW3(config-if)# no shut

Now both sides of the connection are configured and therefore the port-channel has successfully come online and the SVI interfaces can ping each other on the switches.

SW1-2(config-if-range)# sh port-channel summ Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 1 Po2(SU) Eth NONE Eth3/5(P) Eth3/6(P) SW1-2(config-if-range)# SW1-2(config-if-range)# ping 198.18.123.3 PING 198.18.123.3 (198.18.123.3): 56 data bytes 36 bytes from 198.18.123.12: Destination Host Unreachable Request 0 timed out 64 bytes from 198.18.123.3: icmp_seq=1 ttl=254 time=1.422 ms 64 bytes from 198.18.123.3: icmp_seq=2 ttl=254 time=0.738 ms 64 bytes from 198.18.123.3: icmp_seq=3 ttl=254 time=0.71 ms 64 bytes from 198.18.123.3: icmp_seq=4 ttl=254 time=0.981 ms --- 198.18.123.3 ping statistics --- 5 packets transmitted, 4 packets received, 20.00% packet loss round-trip min/avg/max = 0.71/0.962/1.422 ms



SW1-2(config-if-range)#

After convergence the logical interface (port-channel) 2 is up and running!

Next up is the first connection. We want to configure port-channel 1 with some special parameters. These include that the bundle should be negotiated between the switches before it comes online. This is related to port-channel protocols like PAgP and LACP. The first protocol is an older Cisco proprietary protocol which is no longer supported on NX-OS platforms. The standards based solution is the Link Aggregation Control Protocol (LACP). This protocol uses a hello mechanism to identify certain interfaces and switches between each other. When every capability is correctly advertised between the switches the port-channel interfaces will come online. The most important thing a switch negotiates with LACP is the system ID. This ID needs to be advertised on both of the interfaces. This ensures that the switch is connected to one device only and that there is no loop in the network.

The hello mechanism also helps in redundancy switchovers. With a static port-channel it is required for the interface to go down. Otherwise the link will remain in the bundle. With LACP an intelligent mechanism ensures that when a certain amount of hello packets is missed, according to a threshold, the link is removed from the bundle. This makes it possible that when there is equipment between the 2 switches which will not propagate link down events, the switch will still detect this by the missing LACP packets.

In the following drawing a few scenarios are described that would cause errors in a configuration with static port-channels, but is automatically solved when using the LACP protocol.



• Links configured in a port-channel on only 1 side will become ‘Individual’ or ‘Down’

o Dependent on the configuration of the OS, links that are configured in a LACP port-channel on only 1 side of the connection. They can still become online in an Individual mode (no bundling) without any port-channeling capabilities. This is different on NX-OS where by default links are disabled.

• Hot-standby interfaces

o When interfaces are inserted into the link bundle that go over the maximum amount. A static port-channel would reject the configuration, but in an LACP configuration it can still be accepted (Still dependent on other maximums). When one of the active interfaces than fails the now ‘Hot-Standby’ interface will become inserted into the bundle, so that in case of a loss of link, there is no capacity decrease in the link bundle.

• Protection against wrong port-channel configuration

o When one interface is placed in one bundle, where the other device is configured for another configuration of bundles (like the bottom links in the drawing). The LACP port-



channel will go in a Suspended state, where as a static port-channel will have serious problems!

Now on to the configuration of our LACP port-channel. The fourth question states that SW2 should never start the negotiation of the port-channel. This refers to the 2 modes of LACP port-channeling that can be configured. With all above information we can summarize in the 3 different modes that port-channels can be configured in.

Mode Description

On The default configuration. This mode does not initiate LACP messages on the port-channel. This mode is also referred to as ‘static’ port-channels. Interfaces that come online are immediately placed

Active This mode enables the LACP protocol on this port-channel. After the interface comes online, it immediately starts sending LACP packets across the link.

Passive This mode also enables the LACP protocol on a port-channel. The difference with active mode is that in this passive mode the switch waits for another LACP packet to be received on the interface, before it replies back with an LACP message

This results in the following table of modes. Only the yellow marked fields will result in an operational port-channel. The other mode combinations will result in a Suspended or Non functional port-channel.

On Active Passive

On Yes No No

Active No Yes Yes

Passive No Yes No

Now on to the configuration of our second port-channel. SW1-1 and SW2 will be configuring a port-channel with ID 1 between each other.

SW1-1

SW1-1(config)# feature lacp SW1-1(config)# default interface e3/1 SW1-1(config)# default interface e3/2 SW1-1(config)# int e3/1-2 SW1-1(config-if-range)# channel-group 1 mode active SW1-1(config-if-range)# no shut SW1-1(config-if-range)# int po1 SW1-1(config-if)# sw



SW1-1(config-if)# sw mode trunk SW1-1(config-if)# sw trunk allowed vlan 112 SW1-1(config-if)# no shut

SW2

SW2(config)# feature lacp SW2(config)# default interface e1/1 SW2(config)# default interface e1/2 SW2(config)# int e1/1-2 SW2(config-if-range)# channel-group 1 mode passive SW2(config-if-range)# no shut SW2(config-if-range)# int po1 SW2(config-if)# sw SW2(config-if)# sw mode trunk SW2(config-if)# sw trunk allowed vlan 112 SW2(config-if)# no shut

SW2 is stated to never negotiate the interface bundle. Therefore we will use the Passive mode of the LACP port-channeling. After the configuration the port-channel comes online and the SVI is working correctly.

SW1-1(config-if)# sh port-channel summary Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 1 Po1(SU) Eth LACP Eth3/1(P) Eth3/2(P) N7K11-SW1-1(config-if)# do ping 198.18.112.2 PING 198.18.112.2 (198.18.112.2): 56 data bytes 36 bytes from 198.18.112.11: Destination Host Unreachable Request 0 timed out 64 bytes from 198.18.112.2: icmp_seq=1 ttl=254 time=1.504 ms 64 bytes from 198.18.112.2: icmp_seq=2 ttl=254 time=0.678 ms 64 bytes from 198.18.112.2: icmp_seq=3 ttl=254 time=0.708 ms 64 bytes from 198.18.112.2: icmp_seq=4 ttl=254 time=0.723 ms --- 198.18.112.2 ping statistics --- 5 packets transmitted, 4 packets received, 20.00% packet loss round-trip min/avg/max = 0.678/0.903/1.504 ms SW1-1(config-if)#

The next two questions in this task are regarding minimum and maximum interfaces in a port-channel. The second question is regarding the hot-standby feature.



For various reasons it’s possible that you want to steer the amount of interfaces that are up in the port-channel. One of the most important reasons is that you have a network topology where the port-channels are not used for redundancy.

In the case which is illustrated in the drawing above, you would not want the port-channel to remain up in this case. Imagine that the left-side path is chosen in this topology and one of the interfaces fails while there is 16Gbps of traffic crossing the links. Now there would only be 10Gbps available on that connection. The link cost/metric is not changed when a link fails and therefore dynamic routing protocols are not aware of any changes in the network.

The 16Gbps of traffic will still be sent across the interface that now only has 10Gbps of bandwidth. In this case you will want the network to failover to the right-side path of the drawing so the traffic can still be sent to where it needs to go, across a 20Gbps connection.

SW1-1

SW1-1(config)# interface Port-channel1 SW1-1(config-if)# lacp min-links 2

SW2

SW2(config)# interface Port-channel1 SW2(config-if)# lacp min-links 2

Now when this feature is configured and we disable one of the member links in the port-channel. We see that immediately the interface is disabled and no longer working.

SW1-1

SW1-1(config)# interface Ethernet3/1 SW1-1(config-if)# shutdown



SW2

SW2(config)# 2012 Sep 30 10:37:41 SW2 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel1: Ethernet1/1 is down 2012 Sep 30 10:37:41 SW2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel1: first operational port changed from Ethernet1/1 to Ethernet1/2 2012 Sep 30 10:37:41 SW2 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/1 is down (Link failure) 2012 Sep 30 10:37:43 SW2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No operational members) 2012 Sep 30 10:37:43 SW2 %STP-6-PORT_RANGE_DELETED: Interface port-channel1 removed from vlan=112 2012 Sep 30 10:37:43 SW2 %STP-6-PORT_RANGE_STATE: new_state=disabled interface=port-channel1 vlan=112 2012 Sep 30 10:37:43 SW2 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel1: Ethernet1/2 is down 2012 Sep 30 10:37:43 SW2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel1: first operational port changed from Ethernet1/2 to none 2012 Sep 30 10:37:43 SW2 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/2 is down (Initializing) 2012 Sep 30 10:37:43 SW2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No operational members) SW2(config)# 2012 Sep 30 10:37:43 SW2 %ETHPORT-5-SPEED: Interface port-channel1, operational speed changed to 10 Gbps 2012 Sep 30 10:37:43 SW2 %ETHPORT-5-IF_DUPLEX: Interface port-channel1, operational duplex mode changed to Full 2012 Sep 30 10:37:43 SW2 %INTERFACE_VLAN-5-UPDOWN: Line Protocol on Interface vlan 112, changed state to down 2012 Sep 30 10:37:46 SW2 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/2: Ethernet2/2 is suspended SW2(config)# sh port-channel summary Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 1 Po1(SM) Eth LACP Eth2/1(D) Eth2/2(s) SW2(config)# sh int po1 port-channel1 is down (suspended(min-links)) Hardware: Port-Channel, address: 001b.54c1.3429 (bia 001b.54c1.3429) MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA Port mode is trunk auto-duplex, 10 Gb/s Input flow-control is off, output flow-control is off



Switchport monitor is off EtherType is 0x8100 Members in this channel: Eth2/1, Eth2/2 Last clearing of "show interface" counters never 3 interface resets 30 seconds input rate 88 bits/sec, 0 packets/sec 30 seconds output rate 128 bits/sec, 0 packets/sec Load-Interval #2: 5 minute (300 seconds) input rate 144 bps, 0 pps; output rate 408 bps, 0 pps RX 5 unicast packets 273 multicast packets 3 broadcast packets 281 input packets 43984 bytes 0 jumbo packets 0 storm suppression packets 0 runts 0 giants 0 CRC 0 no buffer 0 input error 0 short frame 0 overrun 0 underrun 0 ignored 0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop 0 input with dribble 0 input discard 0 Rx pause TX 6 unicast packets 817 multicast packets 2 broadcast packets 825 output packets 82131 bytes 0 jumbo packets 0 output error 0 collision 0 deferred 0 late collision 0 lost carrier 0 no carrier 0 babble 0 output discard 0 Tx pause SW2(config)#

The interface went down with a clear indication that this is because the minimum amount of links that is configured (Default is 1) is not met in this case.

The next question is regarding the maximum amount of interfaces that may be placed into a bundle. This command is not available in a static port-channel. The maximum amount of interfaces that can be enabled on a Nexus 7000 or Nexus 5000 is 8 active interfaces.

When you go over this amount and try to activate a 9th link in a static bundle. It will reject the command and will not place the link into the bundle. When this same task is done on an LACP bundle, the command is accepted and the link is placed into a ‘hot-standby‘ state. This means that when one of the active links fail in the bundle, the hot-standby link will take its place and will become active in the link bundle.

The maximum amount of active interfaces in a link bundle remains 8, also with LACP port-channels. Then the maximum amount of hot-standby interfaces is also 8. This means that the maximum amount of interfaces that can be configured within a LACP port-channel is 16.

Next up we configure the maximum amount of interfaces in Port-Channel1. As we need to place interfaces into hot-standby mode, when another interface is added. We set the maximum to the current amount of active interfaces.



SW1-1

SW1-1(config)# interface Port-channel1 SW1-1(config-if)# lacp max-bundle 2

SW2

SW2(config)# interface Port-channel1 SW2(config-if)# lacp max-bundle 2

Now we add another interface into the link bundle.

SW2

SW2(config)# interface Ethernet1/3 SW2(config-if)# channel-group 1 mode passive 2012 Sep 30 11:02:09 N7K12-SW2 %ETH_PORT_CHANNEL-5-PORT_HOT_STANDBY: port-channel1: Ethernet1/3 goes to hot-standby SW2(config-if)# sh port-chan summ Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 1 Po1(SU) Eth LACP Eth1/1(P) Eth1/2(P) Eth1/3(H)

We see that this is accepted, but the interface is placed in hot-standby mode, because the maximum amount of interfaces is set to 2.

The next question is regarding a new port-channel between SW2 and SW3. This interface is not used as a layer 2 connection. This one should be configured as a Layer 3 routed interface. Last thing is that the interface should be negotiated by both sides. In an earlier question it was stated that SW2 should never be actively negotiating it’s connection. So SW2 should be set on passive mode.

In production networks it’s a best practice to always configure both sides to be active.

SW2

SW2(config)# default interface e1/5 SW2(config)# default interface e1/6 SW2(config)# int e1/5-6 SW2(config-if-range)# channel-group 3 mode passive SW2(config-if-range)# no shut SW2(config-if-range)# int po3 SW2(config-if)# no switchport SW2(config-if)# ip address 198.18.23.2/24



SW2(config-if)# no shut

SW3

SW3(config)# default interface e1/5 SW3(config)# default interface e1/6 SW3(config)# int e1/5-6 SW3(config-if-range)# channel-group 3 mode active SW3(config-if-range)# no shut SW3(config-if-range)# int po3 SW3(config-if)# no switchport SW3(config-if)# ip address 198.18.23.3/24 SW3(config-if)# no shut

To verify the layer 3 connection we ping the other end.

SW2(config-if)# ping 198.18.23.3 PING 198.18.23.3 (198.18.23.3): 56 data bytes 36 bytes from 198.18.23.2: Destination Host Unreachable Request 0 timed out 64 bytes from 198.18.23.3: icmp_seq=1 ttl=254 time=1.249 ms 64 bytes from 198.18.23.3: icmp_seq=2 ttl=254 time=0.794 ms 64 bytes from 198.18.23.3: icmp_seq=3 ttl=254 time=0.71 ms 64 bytes from 198.18.23.3: icmp_seq=4 ttl=254 time=1.104 ms --- 198.18.23.3 ping statistics --- 5 packets transmitted, 4 packets received, 20.00% packet loss round-trip min/avg/max = 0.71/0.964/1.249 ms N7K12-SW2(config-if)#

Port-channel 3 is configured successfully!

The next question is referring to a behavior that has changed in NX-OS compared to Classical IOS. In Classical IOS when an interface that was brought up in a LACP port-channel without any configuration on the other end of the link. Meaning that it would not receive any LACP packets from the other device. The link would still become available, but in a so-called ‘Individual’ state. This means the link would become available in a normal link state, unbundled. The Port-channel interface would not see this interface as a member until it receives LACP packets.

Within NX-OS this behavior changed. When the interface comes online on an NX-OS interface by default the interface will be in Suspended mode and not come online. This is the better behavior as the Individual interfaces might cause issues and can even cause loops.

SW2

SW2(config-if)# int po3 SW2(config-if)# no lacp suspend-individual ERROR: Cannot set/reset lacp suspend-individual for port-channel3 that is admin up SW2(config-if)# shut SW2(config-if)# no lacp suspend-individual Warning: !! Disable lacp suspend-individual only on port-channel with edge ports. Disabling this on network port port-channel could lead to loops.!



SW2(config-if)# no shut SW2(config-if)#

The switch already warns you that the Individual state might cause loops in the network, so the best practice is to leave this enabled!

SW3

SW3(config-if)# int po3 SW3(config-if)# shut SW3(config-if)# no lacp suspend-individual Warning: !! Disable lacp suspend-individual only on port-channel with edge ports. Disabling this on network port port-channel could lead to loops.! SW3(config-if)# no shut SW3(config-if)#

To test the behavior we will disable the LACP on one of the links and see what happens.

SW2(config-if)# int e1/5 SW2(config-if)# no channel-group 3 SW2(config-if)# int e1/6 SW2(config-if)# no channel-gr 3

On the other side of the link, the link went into suspended mode.

SW3(config-if)# do sh port-chan sum Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 3 Po3(RD) Eth LACP Eth1/5(s) Eth1/6(s)

When the link is configured like stated in question 9. The interfaces are up in an individual state and the port-channel remains down.

N7K12-SW2(config-if)# sh port-chan summ Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel --------------------------------------------------------------------------------



3 Po3(RD) Eth LACP Eth2/1(I) Eth2/2(I) SW2(config-if)# sh int e1/5 | in 1/5 Ethernet1/5 is up SW2(config-if)# sh int e1/6 | in 1/6 Ethernet1/6 is up SW2(config-if)# sh int po3 | in port-chan port-channel3 is down (No operational members) SW2(config-if)#

As there are no members of the port-channel that received LACP packets. The channel remains down and the individual interfaces are up!

Question 10 relates to the selection of ports within a Port-Channel. This selection is crucial for which interfaces become Participating in a bundle and which ones become hot-standby. It has also to do which interface is selected as the ‘Primary’ interface in a bundle, where the major amounts of LACP packets are sent through.

As the maximum amount of links in a bundle is 8, the maximum capacity of the bundle is 80Gbps.

This question states that one of the links should always be chosen to participate when it’s active and one should be selected as hot-standby immediately after new interfaces are added to the bundle.

The mechanism that is used for this task is ‘LACP port-priority’. This is a similar priority value to the Spanning-Tree port-priority value where a port on a local switch is selected. This value is never transported outside of the switch.

The lower the value that is configured. The higher priority is assigned to an interface. Therefore we give Ethernet1/5 a very low value (lowest possible) to ensure that it is always chosen.

SW2

SW2(config-if)# int Ethernet1/5 SW2(config-if)# lacp port-priority 1

SW3


The default priority value is 32768. This means that if we want the interface to become hot-standby as soon as an additional interface is added to the link-bundle this value should be increased.

SW2


SW3




The next question states that Port-channel3 should be using a very fast detection mechanism. Within NX-OS it’s not possible to change the LACP timers by configuring specific values. It is possible however to use sub-second hello messages and have a timeout timer of 1 second to ensure that a neighbor is signaled as down after 1 second.

This is done under individual interfaces and the interface needs to be disabled before this is turned on!

SW2

SW2(config-if)# int e1/5 SW2(config-if)# lacp ? port-priority Set LACP port priority rate Configure rate at which PDUs are sent by LACP SW2(config-if)# lacp rate ? fast Specifies that LACP control packets are transmitted at the fast rate, once every 1 second normal Specifies that LACP control packets are transmitted at the normal rate, every 30 seconds after the link is bundled SW2(config-if)# lacp rate fast ERROR: Cannot set lacp rate. Port Ethernet1/5 is not admin down in port-channel SW2(config-if)# shut SW2(config-if)# lacp rate fast SW2(config-if)# no shutdown SW2(config-if)# int e1/6 SW2(config-if)# shut SW2(config-if)# lacp rate fast SW2(config-if)# no shut

SW3

SW3(config-if)# int e1/5 SW3(config-if)# shut SW3(config-if)# lacp rate fast SW3(config-if)# no shutdown SW3(config-if)# int e1/6 SW3(config-if)# shut SW3(config-if)# lacp rate fast SW3(config-if)# no shut

The next question requires the configuration of load balancing parameters. This is a global setting for the entire switch. It can also be configured per line card. However there are very few cases in which this would be recommended. The global setting is relevant for outgoing traffic only. There is not option to control the way traffic is incoming on the switch itself.

The load-balancing can be a crucial setting on the switch, but is very dependent on what is connected to the port-channels that are configured on the switch. The configuration is dependent on the hardware platforms. Some platforms might not support all types of load balancing. Within the NX-OS platform,



they are all based on high-end switching platforms. So both the Nexus 7000 and the Nexus 5000 platforms in our topology support all of the options.

The load balancing is accomplished using a ‘hash’ of various kinds of information that is found in the header of a packet. This hash is generated through a combination of multiple values found in the Layer 2, Layer 3 and/or Layer 4 header.

The load-balancing options are separated in multiple categories. For layer 2 ethernet traffic a different load balancing is applied than for layer 3 IP traffic.

The following options are available for Layer 3 IP load-balancing:

• Source VLAN number

• Destination VLAN number

• Source and destination VLAN number

• Destination IP address

• Source IP address

• Source and destination IP address

• Source TCP/UDP port number

• Destination TCP/UDP port number

• Source and destination TCP/UDP port number

The following options are available for Layer 2 Ethernet load-balancing:

• Destination IP address and TCP/UDP port

• Destination IP address, TCP/UDP port and VLAN

• Destination IP address and VLAN

• Destination MAC address

• Destination TCP/UDP port

• Source & Destination IP address and TCP/UDP port

• Source & Destination IP address, TCP/UDP port and VLAN

• Source & Destination IP address and VLAN



• Source & Destination MAC address

• Source & Destination TCP/UDP port

• Source IP address and TCP/UDP port

• Source IP address, TCP/UDP port and VLAN

• Source IP address and VLAN

• Source MAC address

• Source TCP/UDP port

The NX-OS platform does not care when traffic that is sent on one interface and received on another interface. Therefore it’s not required to have an equal setting on both sides of the interfaces, but it is recommended to have it equal.

A simple mathematic calculation can be done that there when more fields are taken into the hash, the more different hashes are generated. This means in a more granular load balancing. The load balancing based on fields in the IP header also takes care of that all traffic that is related to a single traffic flow is always sent on a single interface. This overcomes the issue when the approach would be a per-packet load balancing scheme that traffic can arrive in a different order.

Having the traffic flows sent across a single interface helps in the performance of the application.

In case of layer 3 connections there is no reason to do load balancing across MAC addresses, because the routed interfaces are the only 2 MAC addresses that are used in the IP subnet. Therefore these options are not available in the Layer 3 load balancing options.

To complete this task it’s stated that the most information should be used for load balancing traffic across the member links in a port-channel.

On the Nexus 7000 this command cannot be issued from a Virtual Device Context (VDC). Only the default VDC can change the load balancing algorithm for the switch and modules. In our question we only need to change the Nexus 5000 switches.

Remember that layer 2 and layer 3 load balancing needs to be changed as we have both layer 2 and layer 3 Port-Channels on these switches.

SW2

SW2(config)# show port-channel load-balance Port Channel Load-Balancing Configuration: System: src-dst ip Port Channel Load-Balancing Addresses Used Per-Protocol: Non-IP: src-dst mac



IP: src-dst ip SW2(config)# port-channel load-balance ? dst Destination based parameters ethernet Ethernet port-channel src Source based parameters src-dst Source-destination based parameters SW2(config)# port-channel load-balance ethernet ? dest-ip-port Destination IP address and L4 port dest-ip-port-vlan Destination IP address, L4 port and VLAN destination-ip-vlan Destination IP address and VLAN destination-mac Destination MAC address destination-port Destination L4 port source-dest-ip-port Source & Destination IP address and L4 port source-dest-ip-port-vlan Source & Destination IP address, L4 port and VLAN source-dest-ip-vlan Source & Destination IP address and VLAN source-dest-mac Source & Destination MAC address source-dest-port Source & Destination L4 port source-ip-port Source IP address and L4 port source-ip-port-vlan Source IP address, L4 port and VLAN source-ip-vlan Source IP address and VLAN source-mac Source MAC address source-port Source L4 port SW2(config)# port-channel load-balance ethernet source-dest-ip-port-vlan

First we configure the Layer 2 Ethernet load balancing using the ‘ethernet’ statement.

SW2(config)# SW2(config)# SW2(config)# port-channel load-balance ? dst Destination based parameters ethernet Ethernet port-channel src Source based parameters src-dst Source-destination based parameters SW2(config)# port-channel load-balance src-dst ? ip IP ip-l4port IP and L4 port ip-l4port-vlan IP, L4 port and VLAN ip-vlan IP and VLAN l4port L4 port mac MAC SW2(config)# port-channel load-balance src-dst ip-l4port-vlan SW2(config)#



The other configuration options (src, dst, src-dst) are used for the Layer 3 IP load-balancing.

SW3

SW3(config)# port-channel load-balance ethernet source-dest-ip-port-vlan SW3(config)# port-channel load-balance src-dst ip-l4port-vlan

We verify the new configuration and now we are using the best load balancing possible.

SW2(config)# show port-channel load-balance Port Channel Load-Balancing Configuration: System: src-dst ip-l4port-vlan Port Channel Load-Balancing Addresses Used Per-Protocol: Non-IP: src-dst mac IP: src-dst ip-l4port-vlan SW2(config)#

Before we can continue with the next task we need to remove the port-channel configuration between the switches as we will be re-using those interconnecting links for exciting new virtual Port-Channels!

SW1-1

SW1-1(config)# no interface port-channel 1 SW1-1(config)# default interface Ethernet3/1 SW1-1(config)# default interface Ethernet3/2

SW1-2

SW1-2(config)# no interface port-channel 2 SW1-2(config)# default interface Ethernet3/5 SW1-2(config)# default interface Ethernet3/6

SW2

SW2(config)# no interface port-channel 1 SW2(config)# default interface Ethernet1/1 SW2(config)# default interface Ethernet1/2

SW3

SW3(config)# no interface port-channel 2 SW3(config)# default interface Ethernet1/1 SW3(config)# default interface Ethernet1/2



Task 3: Virtual Port-Channels (vPCs) The third task is about a unique feature only available on the Nexus series platforms. The building of so-called Virtual Port-Channels is only available on the Nexus 7000 and Nexus 5000 platforms. Similar technology is however possible on Catalyst 3750 and 6500 series switches, although it has absolutely no connection to the vPC technology.

This technology has been engineered to solve a very common problem. One of the reasons to implement Port-Channels is to deliver redundancy to the interfaces. As stated in the previous task the failover to a remaining interface or the insertion of a new link in the Port-Channel does not result in any traffic loss. This ensures a very smooth operational scenario where links can be easily expanded and failover occurs without any impact.

The redundancy however is not well engineered as the link is still made between 2 single equipped devices. If one of the 2 (core) devices fails, the entire link goes down as illustrated by the drawing below. The connecting device needs a manual switchover to the other link or another failure mechanism like Spanning-Tree to use the other link.

To overcome this redundancy issue we want to use a technology to be able to create a Port-Channel between 2 different switches that simulate to be one logical switch with one logical link between the host/switch and itself.

There are multiple names for this technology, all resulting in the same basic principle. A few examples are:

• Cross-stack-member (StackWise) Port-Channels on the Catalyst 3750 series

• Virtual Switching System (VSS) on the Catalyst 6500 series

• Other vendor implementations of stacking/virtual chassis/IRF/etc.



One generic vendor-independent term used in this case is Multi-Chassis Link Aggregation Groups (MC-LAG).

The Virtual Port-Channeling technology is the only technology that leverages a different approach than other implementations of MC-LAG. The vPC technology is the only one that uses separated control-planes for both switches. In other implementations the control-planes are merged so you only have a single management entity to mange.

In case of the vPC technology you keep 2 separate switches (control-planes) that have shared some features with each other. This has a very special reason why this is done on the Nexus series. The Nexus series are also capable of transferring FCoE traffic, which is explained in more detail in a later chapter. The Fibre-Channel protocol requires the separation of 2 ‘fabrics’. The traffic that uses Fabric A is never allowed to cross any devices that are related to Fabric B. For this reason the vPC technology does not combine the control-planes of the switches so you can keep your Fibre Channel (oE) fabrics separated!

The technology uses a number of components to be able to function correctly and to ensure a loop free topology for the vPC connected devices.

A vPC identifies itself towards and end-host as a normal LACP or Static port-channel. The end device has no indication that it is connected to 2 different switches and therefore assumes that it’s a normal Port-Channel. The technology only operates on the Nexus switches. Before a vPC will come online a number of topics need to be in place:

• vPC Domain configuration

o The vPC domain configuration configures the role of the device (primary / secondary) and the Domain ID.

o The Domain ID needs to match on both devices that are able to create vPCs between each other

• vPC Peer-Link

o The vPC peer-link is by best-practice a Port-Channel of 2 times 10Gbps between the 2 vPC peer devices. This link is used for control information, synchronization and data traffic when traffic needs to exit out of another vPC peer device

• vPC Peer Keepalive link

o The vPC peer keepalive link is also a very crucial part of the set-up. Without a dedicated keep-alive link the vPCs will not come online. This link is used for simple hello packets (can be send across a layer 2 or a layer 3 network).

o The Hello packets are required for split-brain detection, in case the Peer-Lin k fails, one of the vPC peer devices (the secondary) will shut down its vPC ports



o You require a dedicated GigabitEthernet interface or you can use the out-of-band management Ethernet port (mgmt0)

The drawing below illustrates the terminology of the mentioned required topics:

All topics are discussed in the questions in this task. The initial question is to ensure that SW1-1 and SW1-2 will form a pair to create vPC’s. Again it’s good to read ahead as we require some additional information regarding the set-up of our peer-link and peer keepalive link.

We need all this information before the vPC’s will be able to form and before we will see information regarding the switches forming an adjacency.

The information that is exchanged between the vPC switches uses the Cisco Fabric Services over IP protocol (CFS). This protocol has been used by the Cisco MDS (fibre channel) switches for years to exchange information over the Fibre Channel protocol about the other switches. Using this protocol Cisco has been able to extend the Fibre Channel features with this synchronization protocol. The IP version of this protocol has been extended to support vPC’s on the Nexus series.

There are some limitations on the vPC technology. These are a few of the most important limitations:

• No mix of Nexus models

o You can’t mix Nexus 5000 and Nexus 7000 switches to be in the same vPC domain

• No mix of line card types / generations

o On the Nexus 7000 you cannot mix line card types. When the first switches uses M1 line cards, the second switch should also use an M1 line card for both the vPC Peer Link and the vPC client interfaces



• There are only 2 switches in a domain

o You can’t have more than 2 switches in a single vPC domain.

o A switch can only belong to 1 domain at any given time.

• Only Layer 2 port-channels

o You can’t create a Layer 3 port-channel which is a vPC

Now on to some configuration for the 2 domains that we are going to configure.

We enable the vPC functionality first and need to ensure that SW1-1 is the Primary switch in the vPC configuration and SW1-2 the secondary.

SW1-1

SW1-1(config)# feature vpc SW1-1(config)# vpc domain 100 SW1-1(config-vpc-domain)# role ? priority Configure priority to be used during vPC role (primary/secondary) election SW1-1(config-vpc-domain)# role priority 100 Warning: !!:: vPCs will be flapped on current primary vPC switch while attempting role change ::!! Note: --------:: Change will take effect after user has re-initd the vPC peer-link ::--------

SW1-2

SW1-2(config)# feature vpc SW1-2(config)# vpc domain 100 SW1-1(config-vpc-domain)# role priority 1000 Warning: !!:: vPCs will be flapped on current primary vPC switch while attempting role change ::!! Note: --------:: Change will take effect after user has re-initd the vPC peer-link ::--------

We use the Role Priority command to determine which switch will be the primary and which switch will be the secondary switch in our vPC configuration. Like the switch mentions. When you are doing this in an active configuration. The peer-link should be flapped to ensure the new roles are assigned. This has to do with the nature of the role. It’s not pre-empted, meaning that it will not flap when it sees a peer come online with a better priority value. Only after re-initialization the roles are (re-)assigned. In this case we make sure that SW1-1 has a much lower priority value (and therefore a higher priority) than SW1-2.



On SW2 and SW3 we do not care about the role of the devices in the vPC configuration. Remember that when you are configuring vPC’s between 2 different sets of Nexus switches like in our topology. You need to use different Domain ID’s. Otherwise the vPC’s will not come online when equal Domain ID’s are used between the 2 sets of switches.

SW2

SW2(config)# feature vpc SW2(config)# vpc domain 200

SW3

SW3(config)# feature vpc SW3(config)# vpc domain 200

Next up are the configurations for the peer keep alive links. These should always be configured before you go to the vPC Peer-link configuration. The vPC Peer-link will not come online when no hello messages are received over the peer keep alive link. This link is crucial in the set-up!

On the Nexus 5000 switches we will be using the mgmt0 interface. On the Nexus 7000 it’s the best practice to use a dedicated GigabitEthernet interface from the line cards instead of the management interface on the supervisor.

SW2

SW2(config-vpc-domain)# peer-keepalive destination 172.16.100.3 vrf management SW3(config-vpc-domain)#

SW3

SW3(config-vpc-domain)# peer-keepalive destination 172.16.100.2 vrf management SW3(config-vpc-domain)#

Remember to always specify the VRF that is used for these keep alive messages. As a best practice they should never be sent from the Global Routing Table. On the Nexus 5000 we use the mgmt0 interface for our keepalives.

After configuration, the vPC protocol has not been successfully established, but we do see another peer and mention that it is alive!

SW2(config-vpc-domain)# sh vpc brief Legend: (*) - local vPC is down, forwarding via vPC peer-link vPC domain id : 200 Peer status : peer link not configured



vPC keep-alive status : peer is alive Configuration consistency status : failed Per-vlan consistency status : failed Configuration inconsistency reason: vPC peer-link does not exist Type-2 consistency status : failed Type-2 inconsistency reason : vPC peer-link does not exist vPC role : none established Number of vPCs configured : 0 Peer Gateway : Disabled Dual-active excluded VLANs : - Graceful Consistency Check : Disabled (due to peer configuration) Auto-recovery status : Disabled

The next question is about configuring the vPC peer keepalives across a dedicated SVI and the traffic should be separated, like the best practice, from the global routing table. We do this by using a dedicated VRF for the keepalives. The question did not specify any VLAN number, so we can use anything we want. As all other IP subnets keep their VLAN number equal to the third octet, we will keep this standardization to make it easier for ourselves.

We configure the SVI and VRF first and then configure the peer-keepalive statement:

SW1-2

SW1-2(config)# int e3/12 SW1-2(config-if)# sw mode trunk SW1-2(config-if)# sw trunk allowed vlan 5 SW1-2(config-if)# no shut SW1-2(config-if)# vlan 5 SW1-2(config-vlan)# exit SW1-2(config)# feature interface-vlan SW1-2(config)# int vlan 5 SW1-2(config-if)# ip add 198.18.5.12/24 SW1-2(config-if)# no shut SW1-2(config-if)# ping 198.18.5.11 PING 198.18.5.11 (198.18.5.11): 56 data bytes Request 0 timed out 64 bytes from 198.18.5.11: icmp_seq=1 ttl=254 time=1.518 ms 64 bytes from 198.18.5.11: icmp_seq=2 ttl=254 time=0.825 ms 64 bytes from 198.18.5.11: icmp_seq=3 ttl=254 time=0.838 ms 64 bytes from 198.18.5.11: icmp_seq=4 ttl=254 time=0.731 ms --- 198.18.5.11 ping statistics --- 5 packets transmitted, 4 packets received, 20.00% packet loss round-trip min/avg/max = 0.731/0.977/1.518 ms SW1-2(config-if)#

We can ping the other end successfully, but we also need to configure the VRF.

SW1-2(config-if)# vrf context keepalives SW1-2(config-vrf)# int vlan 5 SW1-2(config-if)# vrf member keepalives % Deleted all L3 config on interface Vlan5



Be aware that all IP addressing and other Layer 3 configuration is removed from the interface once it’s placed into a VRF!

SW1-2(config-if)# ip add 198.18.5.12/24 SW1-2(config-if)# ping 198.18.5.11 vrf keepalives PING 198.18.5.11 (198.18.5.11): 56 data bytes Request 0 timed out 64 bytes from 198.18.5.11: icmp_seq=1 ttl=254 time=1.281 ms 64 bytes from 198.18.5.11: icmp_seq=2 ttl=254 time=0.813 ms 64 bytes from 198.18.5.11: icmp_seq=3 ttl=254 time=0.781 ms 64 bytes from 198.18.5.11: icmp_seq=4 ttl=254 time=0.729 ms --- 198.18.5.11 ping statistics --- 5 packets transmitted, 4 packets received, 20.00% packet loss round-trip min/avg/max = 0.729/0.901/1.281 ms SW1-2(config-if)# SW1-2(config-if)# vpc domain 100 SW1-2(config-vpc-domain)# peer-keepalive destination 198.18.5.11 vrf keepalives ERROR: Management IP cannot be used as source IP for non-default VRF, please enter source IP SW1-2(config-vpc-domain)# peer-keepalive destination 198.18.5.11 vrf keepalives source 198.18.5.12

By default the management IP address is used as the source address for the vPC keep alives. Therefore it’s required to configure the source IP address when you are using another interface than the mgmt0.

SW1-1

SW1-1(config)# int e3/10 SW1-1(config-if)# sw mode trunk SW1-1(config-if)# sw trunk allowed vlan 5 SW1-1(config-if)# no shut SW1-1(config-if)# vlan 5 SW1-1(config-vlan)# exit SW1-1(config)# feature interface-vlan SW1-1(config-if)# vrf context keepalives SW1-1(config-vrf)# int vlan 5 SW1-1(config-if)# vrf member keepalives SW1-1(config-if)# ip add 198.18.5.11/24 SW1-1(config-if)# no shut SW1-1(config-if)# vpc domain 100 SW1-2(config-vpc-domain)# peer-keepalive destination 198.18.5.12 vrf keepalives source 198.18.5.11 SW1-1(config-vpc-domain)#

The last required configuration before we can bring up the vPC’s itself are the vPC peer-links. In one example we will be using a port-channel (standard) to ensure that a link can fail within the peer-link before it’s resulting in erroneous operational issues. This is however not required and we can also configure just a single link as the vPC peer-link.

Remember that the peer-link might also carry data traffic, so scaling the link in terms of uplink capacity is very important.



First we create a single link peer-link between SW1-1 and SW1-2. The peer-link should either be a FabricPath enabled interface or a Layer 2 trunk interface with all VLANs enabled.

SW1-1

SW1-1(config)# int e3/9 SW1-1(config-if)# sw mode trunk SW1-1(config-if)# vpc peer-link SW1-1(config-if)# no shut

SW1-2

SW1-2(config)# int e3/11 SW1-2(config-if)# sw mode trunk SW1-2(config-if)# vpc peer-link SW1-2(config-if)# no shut

The second example is configuring the peer-link as a port-channel. We first create a standard LACP based port-channel between the 2 switches and enable the vPC peer-link functionality on these interfaces.

SW2

SW2(config)# int e1/7-8 SW2(config-if-range)# channel-group 23 mode active SW2(config-if-range)# exit SW2(config)# interface port-channel23 SW2(config-if)# sw mode trunk SW2(config-if)# vpc peer-link SW2(config-if)# no shut

SW3

SW3(config)# int e1/7-8 SW3(config-if-range)# channel-group 23 mode active SW3(config-if-range)# exit SW3(config)# interface port-channel23 SW3(config-if)# sw mode trunk SW3(config-if)# vpc peer-link SW3(config-if)# no shut

After the configuration the vPC peer-link is signaled up and working and the switches are ready to be configuring vPC port-channels!

SW2(config-if)# sh vpc brief Legend: (*) - local vPC is down, forwarding via vPC peer-link vPC domain id : 200 Peer status : peer adjacency formed ok vPC keep-alive status : peer is alive Configuration consistency status : success Per-vlan consistency status : success Type-2 inconsistency reason : Consistency Check Not Performed



vPC role : primary Number of vPCs configured : 0 Peer Gateway : Disabled Dual-active excluded VLANs : - Graceful Consistency Check : Enabled Auto-recovery status : Disabled vPC Peer-link status --------------------------------------------------------------------- id Port Status Active vlans -- ---- ------ -------------------------------------------------- 1 Po23 up 1

The next question asks to ensure that the vPC’s remain up and running when a peer fails or reboots. The split-brain detection works in a way that the secondary switch turns it’s vPCs off when it suspects a split-brain. To recover from this you can force the switch to bring up its VDCs after a time-out.

SW1-1

SW1-1(config-if)# vpc domain 100 SW1-1(config-vpc-domain)# auto-recovery ? <CR> reload-delay Duration to wait before assuming peer dead and restoring vpcs SW1-1(config-vpc-domain)# auto-recovery reload-delay ? <240-3600> Time-out for restoring vPC links (in seconds) SW1-1(config-vpc-domain)# auto-recovery reload-delay 300 Warning: Enables restoring of vPCs in a peer-detached state after reload, will wait for 300 seconds to determine if peer is un-reachable

SW1-2

SW1-2(config-if)# vpc domain 100 SW1-2(config-vpc-domain)# auto-recovery reload-delay 300

Remember that there are 2 features that do almost the same. Before NX-OS 5.2(1) the command used to solve this was ‘reload restore’. After this release it has been deprecated and replaced with auto-recovery.

The reload restore command did not solve the issue when a peer failed, only in case of a reboot. Therefore in this question the only right answer is the new version of the command.

The spanning-tree question is related to vPC configuration where the vPC switches will identify themselves as a single Spanning-Tree switch to the connecting switches that think they are connecting with a single switch. Therefore the spanning-tree configuration that is made locally only applies to that



specific switch and not to connecting switches that are using vPCs. Except when the ‘peer-switch’ configuration is applied. Then the CFS protocol is used to push the spanning-tree control plane messages to the other switch. You still need to configure the spanning-tree settings independently, but when they match. The switch will be identified as a single root bridge towards the rest of the network.

SW2

SW2(config-if)# vpc domain 200 SW2(config-vpc-domain)# peer-switch

SW3

SW3(config-if)# vpc domain 200 SW3(config-vpc-domain)# peer-switch

Now to verify we check the spanning-tree output of VLAN 1 for both switches. We see a default priority value is set.

SW2(config)# sh spanning-tree VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 32769 Address 0023.04ee.be64 This bridge is the root Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 32769 (priority 32768 sys-id-ext 1) Address 0023.04ee.be64 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Po23 Desg FWD 1 128.4118 (vPC peer-link) Network P2p SW3# sh spanning-tree VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 32769 Address 0023.04ee.be64 This bridge is the root Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 32769 (priority 32768 sys-id-ext 1) Address 0023.04ee.be64 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Po23 Root FWD 1 128.4118 (vPC peer-link) Network P2p



N7K-8-1#

Now we change the priority to the asked 8192 and check again.

SW2

SW2(config-if)# spanning-tree vlan 1 prio 8192

SW3

SW3(config-if)# spanning-tree vlan 1 prio 8192

Now we need to verify again.

SW2(config)# sh span VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 8193 Address 0023.04ee.be64 This bridge is the root Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 8193 (priority 8192 sys-id-ext 1) Address 0023.04ee.be64 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Po23 Desg FWD 1 128.4118 (vPC peer-link) Network P2p SW3(config)# sh span VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 8193 Address 0023.04ee.be64 This bridge is the root Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 8193 (priority 8192 sys-id-ext 1) Address 0023.04ee.be64 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Po23 Root FWD 1 128.4118 (vPC peer-link) Network P2p

Although the switches both identify as root. SW3 also has a root port as SW2 is the true root bridge in this network. Still SW3 will recognize itself as the root bridge and will act like it for the rest of the network!



The next task is one where we finally begin configuring the vPCs!

The initial connection is made between vPC domain 100 and a ‘client’ switch, which is SW2 in this case. From SW2’s perspective this is a standard port-channel towards another standard switch.

We first configure the port-channel on SW2. We use LACP in this case, but we could’ve used a static Port-Channel (mode on) as well. The best practice however is to use LACP port-channels.

SW2

SW2(config-if)# int e1/1-2 SW2(config-if-range)# channel-group 101 mode active SW2(config-if-range)# no shut SW2(config-if-range)# SW2(config-if-range)# int port-channel101 SW2(config-if)# sw mode trunk SW2(config-if)# no shut SW2(config-if)# sh port-chan summ Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 101 Po101(SD) Eth LACP Eth1/1(I) Eth1/5(I) SW2(config-if)#

The port-channel is currently not operational because we haven’t configured the other side yet.

SW1-1

SW1-1(config-if)# int e3/1 SW1-1(config-if)# channel-group 101 mode active SW1-1(config-if)# no shut SW1-1(config-if)# int port-channel101 SW1-1(config-if)# sw mode trunk SW1-1(config-if)# vpc 101 SW1-1(config-if)# no shut SW1-1(config-if)#

SW1-2

SW1-2(config-if)# int e3/3 SW1-2(config-if)# channel-group 101 mode active SW1-2(config-if)# no shut SW1-2(config-if)# int port-channel101 SW1-2(config-if)# sw mode trunk SW1-2(config-if)# vpc 101 SW1-2(config-if)# no shut



SW1-2(config-if)#

The only thing specific about this configuration is the ‘vpc’ command. This command needs to be the same on both vPC peer switches. Port-channel numbers do not need to match, but vpc numbers do! It’s highly recommended to keep the port-channel number and vpc number equal, because it eases your troubleshooting a lot! There is also no reason why these should be different.

After configuring the vPC peer switches we see the port-channel (standard) is now up and running on

SW2.

SW2(config-if)# sh port-chan summ Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 101 Po101(SU) Eth LACP Eth1/1(P) Eth1/5(P)

On the vPC switches we also see a perfectly functioning port-channel, except with just one member.

SW1-1(config-if)# sh port-chan sum Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 101 Po101(SU) Eth LACP Eth3/1(P) SW1-2(config-if)# sh port-chan sum Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports



Channel -------------------------------------------------------------------------------- 101 Po101(SU) Eth LACP Eth3/3(P)

The most important verification command is to see if the vPC has been formed OK and if the state information is correctly synced between the switches.

SW1-1(config-if)# sh vpc brief Legend: (*) - local vPC is down, forwarding via vPC peer-link vPC domain id : 100 Peer status : peer adjacency formed ok vPC keep-alive status : peer is alive Configuration consistency status : success Per-vlan consistency status : success Type-2 inconsistency reason : Consistency Check Not Performed vPC role : secondary Number of vPCs configured : 1 Peer Gateway : Disabled Dual-active excluded VLANs : - Graceful Consistency Check : Enabled Auto-recovery status : Disabled vPC Peer-link status --------------------------------------------------------------------- id Port Status Active vlans -- ---- ------ -------------------------------------------------- 1 Po23 up 1 vPC status ---------------------------------------------------------------------- id Port Status Consistency Reason Active vlans -- ---- ------ ----------- ------ ------------ 101 Po101 up success success 1

The vPC is up and running! A lot of things can go wrong in your configuration before the vPC is up and running. For example, the allowed VLAN list should be 100% equal on both sides, or the vPC will stay down with an error message. The ‘show vpc’ command will show you a lot of information regarding the reason of the vPC being down. Usually it’s very easily spotted when you issue a show running for both interfaces and see that there are differences in the configuration.

The next vPC that needs to be configured is very much the same as the previous one. Except that this one is using vPC domain 200 instead of 100. This means that both Nexus 5000s are used as vPC peers and the Nexus 7000 is the client switch where the vPC is connected to.

The configuration is straight forwards and a good practice for another vPC connection:



SW1-2

SW1-2(config-if)# int e3/5,e3/7 SW1-2(config-if)# channel-group 102 mode on SW1-2(config-if)# no shut SW1-2(config-if)# int port-channel102 SW1-2(config-if)# sw mode trunk SW1-2(config-if)# no shut SW1-2(config-if)#

SW2

SW2(config-if)# int e1/3 SW2(config-if)# channel-group 102 mode on SW2(config-if)# no shut SW2(config-if)# int port-channel102 SW2(config-if)# sw mode trunk SW2(config-if)# vpc 102 SW2(config-if)# no shut SW2(config-if)#

SW3

SW3(config-if)# int e1/3 SW3(config-if)# channel-group 102 mode on SW3(config-if)# no shut SW3(config-if)# int port-channel102 SW3(config-if)# sw mode trunk SW3(config-if)# vpc 102 SW3(config-if)# no shut

One important command to check is the following command, giving an overview of all settings that are checked on the ports before the vPC comes online.

SW2(config-if)# sh vpc consistency-parameters vpc 102 Legend: Type 1 : vPC will be suspended in case of mismatch Name Type Local Value Peer Value ------------- ---- ---------------------- ----------------------- Shut Lan 1 No No STP Port Type 1 Default Default STP Port Guard 1 None None STP MST Simulate PVST 1 Default Default mode 1 on on Speed 1 10 Gb/s 10 Gb/s Duplex 1 full full Port Mode 1 trunk trunk Native Vlan 1 1 1 MTU 1 1500 1500 Admin port mode 1 Allowed VLANs - 1-3967,4048-4093 1-3967,4048-4093



Local suspended VLANs - - - SW2(config-if)#

When we, for example, change the Native VLAN on the port-channel on one of the vPC peers. The vPC is brought down, because traffic is no longer guaranteed to be treated as it should on each of the vPC peers. This check is pro-active and happens right after the command has been issued.

SW2

SW2(config-if)# interface port-channel102 SW2(config-if)# switchport trunk native vlan 5 SW2(config-if)# sh vpc consistency-parameters vpc 102 Legend: Type 1 : vPC will be suspended in case of mismatch Name Type Local Value Peer Value ------------- ---- ---------------------- ----------------------- Shut Lan 1 No No STP Port Type 1 Default Default STP Port Guard 1 None None STP MST Simulate PVST 1 Default Default mode 1 on on Speed 1 10 Gb/s 10 Gb/s Duplex 1 full full Port Mode 1 trunk trunk Native Vlan 1 5 1 MTU 1 1500 1500 Admin port mode 1 Allowed VLANs - 1 1-3967,4048-4093 Local suspended VLANs - - - SW2(config-if)# sh vpc Legend: (*) - local vPC is down, forwarding via vPC peer-link vPC domain id : 200 Peer status : peer adjacency formed ok vPC keep-alive status : peer is alive Configuration consistency status: success Per-vlan consistency status : success Type-2 consistency status : success vPC role : secondary Number of vPCs configured : 1 Peer Gateway : Disabled Dual-active excluded VLANs : - Graceful Consistency Check : Enabled vPC Peer-link status --------------------------------------------------------------------- id Port Status Active vlans -- ---- ------ -------------------------------------------------- 1 Po23 up 1



vPC status ---------------------------------------------------------------------------- id Port Status Consistency Reason Active vlans ------ ----------- ------ ----------- -------------------------- ----------- 102 Po102 down* failed Compatibility check failed - for native vlan

Like previously mentioned. The show vpc shows already most important information.

In a more recent NX-OS release, it became possible to have the vPC come online even when there is a consistency error. The graceful consistency-check enables the interface of the vPC terminating on the Primary vPC peer device to come online and have the interface terminating on the Secondary peer go in suspended mode. This way the traffic will not have impact on a configuration error.

In this example we used a Static Port-Channel instead of an LACP one.

You could’ve used an LACP channel as well, as the question did not indicate which way to use. To demonstrate that static port-channels also work in this configuration this was used in the example.

The next task is to use the remaining connections to finish the vPC configurations. These connections that remain will be a vPC that is connected to another vPC. For the switches itself they will look like standard vPC port-channels, but they are what is called ‘back-to-back-vPCs’.

This single connection is definitely a best practice. The important part of this is that the vPC Domain ID’s not never overlap in this configuration. This can lead to very strange situations.

The port-channel is configured just like any other vPC interface:

SW1-1

SW1-1(config-if)# int e3/2,e3/4,e3/6 SW1-1(config-if)# channel-group 103 mode active SW1-1(config-if)# no shut SW1-1(config-if)# int port-channel103 SW1-1(config-if)# sw mode trunk SW1-1(config-if)# vpc 103 SW1-1(config-if)# no shut SW1-1(config-if)#

SW1-2

SW1-2(config-if)# int e3/5 SW1-2(config-if)# channel-group 103 mode on SW1-2(config-if)# no shut SW1-2(config-if)# int port-channel103



SW1-2(config-if)# sw mode trunk SW1-2(config-if)# vpc 103 SW1-2(config-if)# no shut SW1-2(config-if)#

SW2

SW2(config-if)# int e1/4 SW2(config-if)# channel-group 103 mode on SW2(config-if)# no shut SW2(config-if)# int port-channel103 SW2(config-if)# sw mode trunk SW2(config-if)# vpc 103 SW2(config-if)# no shut SW2(config-if)#

SW3

SW3(config-if)# int e1/1-2,e1/4 SW3(config-if)# channel-group 103 mode on SW3(config-if)# no shut SW3(config-if)# int port-channel103 SW3(config-if)# sw mode trunk SW3(config-if)# vpc 103 SW3(config-if)# no shut

The next question in which we are asked to allow all VLANs required for our Layer 3 topology to be traveling the vPC links is already completed as we did not place an VLAN allowed list on the vPCs. So we can just use them for the next tasks.

The final task is to let the vPC domain 100 identify itself through LACP using a specific MAC address. One reason for LACP port-channels to work is, that they ensure loop prevention. When different switches are seen on interfaces the port-channel will not come online. This would cause issues on vPC enabled switches as they still have 2 separate control planes and therefore different MAC addresses. To be able for LACP port-channels to work. The vPC switch ‘spoofs’ a single MAC address of one of the vPC peer members for this.

The field we are looking for in the LACP packet is the System-MAC field. This is changed in the vPC domain configuration.

SW1-1

SW1-1(config-if)# vpc domain 100 SW1-1(config-vpc-domain)# system-mac 1234.5678.90ab Changing system id might flap peer-link and vPCs. Continue (yes/no)? [no] y

SW1-2

SW1-2(config-if)# vpc domain 100 SW1-2(config-vpc-domain)# system-mac 1234.5678.90ab Changing system id might flap peer-link and vPCs. Continue (yes/no)? [no] y



As the command already mentions. The vPC peer-link and vPC’s will flap when this command is issued and you have LACP based port-channels running on the vPC enabled switch.

After this configuration task 3 is completed!

Task 4: Graceful Restart / Non-Stop Forwarding The next task is regarding a feature which is by default configured in NX-OS based systems. The technology has been around for a long time, but wasn’t widely used, except in Service Provider environments.

The Graceful Restart functionality is a process that works in dynamic routing protocols. It only works on platforms that have a clear separation between control-plane and data-plane. The technology requires the support of the dynamic routing protocols on which it is enabled. These protocols are able to signal a special packet once the router switches over to another Supervisor (Route Processor, Route Switch Processor, Routing Engine, etc.) or when they will completely reboot.

By signaling this special packet. The neighboring routers will know that a router is undergoing a change (software upgrade, reboot, switchover, etc.). They will keep the routing information about this neighbor in their respective routing tables and signal the neighbor down according to standard time-outs. The traffic can still be directed towards the neighbor who is rebooting, because the data/forwarding plane is not dependent on the control-plane to operate. Therefore traffic can still be sent to it.

When the router is rebooted / switched over, the dynamic routing protocol adjacencies are established again. Routes are being learned using the normal process and the router is back up and running. Traffic had no impact on this change.

The routing protocols that are supported on the NX-OS devices like OSPF and EIGRP are by default enabled for graceful-restart. The feature will only work when both neighbors advertise the capability to each other. So it’s safe to enable the feature, if the other neighbor doesn’t support it, the feature is not used.

In this task we first need to configure some dynamic routing protocols. We will use the routing information configured now in later tasks!

First the OSPF configuration:

SW1-1

SW1-1(config-if)# feature ospf SW1-1(config)# router ospf IPX SW1-1(config-router)# int vlan 112 SW1-1(config-if)# ip router ospf IPX area 0 SW1-1(config-if)# int lo0 SW1-1(config-if)# ip router ospf IPX area 0



SW2

SW2(config-if)# feature ospf SW2(config)# router ospf IPX SW2(config-router)# int vlan 112 SW2(config-if)# ip router ospf IPX area 0 SW2(config-if)# int lo0 SW2(config-if)# ip router ospf IPX area 0

After configuration we can verify that switches can ping each other’s Loopback addresses.

SW1-1(config-if)# sh ip route ospf IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 198.18.0.2/32, ubest/mbest: 1/0 *via 198.18.112.2, Vlan112, [110/41], 00:00:06, ospf-IPX, intra SW1-1(config-if)# ping 198.18.0.2 sour 198.18.0.11 PING 198.18.0.2 (198.18.0.2) from 198.18.0.11: 56 data bytes 64 bytes from 198.18.0.2: icmp_seq=0 ttl=254 time=1.213 ms 64 bytes from 198.18.0.2: icmp_seq=1 ttl=254 time=0.842 ms 64 bytes from 198.18.0.2: icmp_seq=2 ttl=254 time=0.841 ms 64 bytes from 198.18.0.2: icmp_seq=3 ttl=254 time=7.059 ms 64 bytes from 198.18.0.2: icmp_seq=4 ttl=254 time=12.102 ms --- 198.18.0.2 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.841/4.411/12.102 ms SW1-1(config-if)#

Next up is the EIGRP configuration. In the question there is no mentioning about a EIGRP autonomous-system number. So we can use any number we like in this case.

SW1-2

SW1-2(config-if)# feature eigrp SW1-2(config)# router eigrp IPX SW1-2(config-router)# autonomous-system 65000 SW1-2(config-router)# int vlan 123 SW1-2(config-if)# ip router eigrp IPX SW1-2(config-if)# int lo0 SW1-2(config-if)# ip router eigrp IPX

SW3

SW3(config-if)# feature eigrp SW3(config)# router eigrp IPX SW3(config-router)# autonomous-system 65000 SW3(config-router)# int vlan 123 SW3(config-if)# ip router eigrp IPX SW3(config-if)# int lo0 SW3(config-if)# ip router eigrp IPX



After configuration we can verify that switches can ping each other’s Loopback addresses.

SW3(config-if)# sh ip eigrp nei IP-EIGRP neighbors for process 65000 VRF default H Address Interface Hold Uptime SRTT RTO Q Seq (sec) (ms) Cnt Num 0 198.18.123.12 Vlan123 13 00:00:01 1 2000 1 0 SW3(config-if)# SW3(config-if)# sh ip rout eigrp IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 198.18.0.12/32, ubest/mbest: 1/0 *via 198.18.123.12, Vlan123, [90/130816], 00:00:03, eigrp-IPX, internal SW3(config-if)# ping 198.18.0.12 sour 198.18.0.3 PING 198.18.0.12 (198.18.0.12) from 198.18.0.3: 56 data bytes 64 bytes from 198.18.0.12: icmp_seq=0 ttl=254 time=1.021 ms 64 bytes from 198.18.0.12: icmp_seq=1 ttl=254 time=0.719 ms 64 bytes from 198.18.0.12: icmp_seq=2 ttl=254 time=0.705 ms 64 bytes from 198.18.0.12: icmp_seq=3 ttl=254 time=0.769 ms 64 bytes from 198.18.0.12: icmp_seq=4 ttl=254 time=6.475 ms --- 198.18.0.12 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.705/1.937/6.475 ms SW3(config-if)#

The next question in this task is already a default configuration, so this doesn’t require any changes. It is stated to make you aware that Graceful Restart or Non-Stop Forwarding is by default configured on both EIGRP and OSPF configurations.

To verify you can issue a couple commands:

SW2(config-router)# sh ip ospf IPX Routing Process IPX with ID 198.18.0.2 VRF default Stateful High Availability enabled Graceful-restart is configured Grace period: 60 state: Inactive Last graceful restart exit status: None Supports only single TOS(TOS0) routes Supports opaque LSA Administrative distance 110 Reference Bandwidth is 40000 Mbps Initial SPF schedule delay 200.000 msecs, minimum inter SPF delay of 1000.000 msecs, maximum inter SPF delay of 5000.000 msecs



Initial LSA generation delay 0.000 msecs, minimum inter LSA delay of 5000.000 msecs, maximum inter LSA delay of 5000.000 msecs Minimum LSA arrival 1000.000 msec LSA group pacing timer 10 secs Maximum paths to destination 8 Number of external LSAs 0, checksum sum 0 Number of opaque AS LSAs 0, checksum sum 0 Number of areas is 1, 1 normal, 0 stub, 0 nssa Number of active areas is 1, 1 normal, 0 stub, 0 nssa Area BACKBONE(0.0.0.0) Area has existed for 00:02:15 Interfaces in this area: 2 Active interfaces: 2 Passive interfaces: 1 Loopback interfaces: 1 No authentication available SPF calculation has run 6 times Last SPF ran for 0.000210s Area ranges are Number of LSAs: 3, checksum sum 0xe524 SW2(config-router)#

Next we configure timers in which we want to change the graceful-restart behavior. By default the time-out of switches to reboot is set to 60 seconds. The question states that we will have a switch connected to SW2 which takes longer to reboot. Therefore we need to change SW2 as that will be the neighbor of the new switch.

SW2

SW2(config-if)# router ospf IPX SW2(config-router)# graceful-restart grace-period 120 SW2(config-router)# sh ip ospf IPX | sec Graceful- Graceful-restart is configured Grace period: 120 state: Inactive Last graceful restart exit status: None

The next configuration is a bit more tricky. It comes down to the fact that you need to have Graceful-Restart enabled on the EIGRP process before an In-Service-Software-Upgrade will work. You will be warned when you disable the graceful-restart option and start an ISSU.

We can verify that it’s by default enabled and that we are capable of supporting ISSU’s!

SW1-2(config-if)# sh ip eigrp IPX IP-EIGRP AS 65000 ID 198.18.0.12 VRF default Process-tag: IPX Status: running Authentication mode: none Authentication key-chain: none Metric weights: K1=1 K2=0 K3=1 K4=0 K5=0 IP proto: 88 Multicast group: 224.0.0.10 Int distance: 90 Ext distance: 170 Max paths: 8 Number of EIGRP interfaces: 2 (1 loopbacks) Number of EIGRP passive interfaces: 0



Number of EIGRP peers: 1 Graceful-Restart: Enabled Stub-Routing: Disabled NSF converge time limit/expiries: 120/0 NSF route-hold time limit/expiries: 240/0 NSF signal time limit/expiries: 20/0 Redistributed max-prefix: Disabled SW1-2(config-if)#

In the EIGRP process we have a few different timers than are available in OSPF. In the EIGRP database routes are maintained, but it’s already signaled to further EIGRP neighbors that one particular router in the network is rebooting.

SW1-2(config-router)# timers nsf ? converge EIGRP time limit for convergence after switchover route-hold EIGRP hold time for routes learned from nsf peer signal EIGRP time limit for signaling NSF restart

For this question we will change the hold-timer and the restart timer.

SW3

SW3(config-if)# router eigrp IPX SW3(config-router)# timers nsf route-hold 300 SW3(config-router)# timers nsf signal ? <10-360> Seconds *Default value is 20 SW3(config-router)# timers nsf signal 10

We should configure the fastest restart signaling timer. By using the question mark we find that the timer can be set to a value between 10 and 360. This means that 10 seconds is the shortest time period in which we can signal a restarting neighbor to other EIGRP neighbors.

Next up are the First Hop Redundancy Protocols!

Task 5: HSRP This is the first of three tasks regarding a subject which is a lot alike. HSRP or Hot Standby Routing Protocol is one of the 3 major First Hop Redundancy Protocols (FHRP). These protocols ensure that for any given Layer 2 segment the Default Gateway is protected against failures.

Clients can only set one default gateway IP address in their Network Interface Card (NIC) settings. This gateway should therefore always be up, otherwise clients are disconnected from the network as soon as 1 router fails.

The solution to this problem is a protocol which enables resiliency between routers while still being able to have only 1 IP address as the default gateway for a Layer 2 segment (IP subnet).



This is achieved by using a Virtual IP address (VIP) which is configured on 2 switches. One of the 2 switches will be the active or primary router and the other switch is the backup or hot-standby. As soon as something happens with the primary switch, the next one will take over.

There are 3 major protocols that have similar functionality. The first that is going to be discussed is the first protocol that was developed: HSRP.

The HSRP protocol is a Cisco proprietary FHRP. It uses a Virtual IP address connected to a Virtual MAC address from the MAC address range of 0000.0C07.AC00 up to 0000.0C07.ACFF for HSRP version 1 and 0000.0C9F.F000 up to 0000.0C9F.FFFF in HSRP version 2. Where the last byte of the MAC address is related to the HSRP group number that is configured.

The working of HSRP is illustrated in the following drawing:

The primary HSRP router will be the only router that responds to ARP requests towards the Default Gateway of the IP subnet. During the time both routers are active. The HSRP routers send Hello packets to each other, informing them to be alive. Once a router fails and the hot-standby router reaches a threshold of missing Hello packets it will become the new Primary router. The selection of the primary router is done based on priority values that can be configured (default 100). When the previously primary router comes back online, the HSRP protocol can, if configured, fall back to the scenario where the highest priority router is the active router.



HSRP is available in 2 versions. Version 1 is by default enabled, version 2 can be configured dependent on the requirements. These are the differences between the 2 versions:

Uses a dedicated Multicast group

HSRP version 2 uses 224.0.0.102 as its Multicast group to signal to the other router. HSRP version 1 used the All-Routers multicast group of 224.0.0.2

Group numbers can be configured from 0 to 4095

In HSRP version 1 this was limited from 0 to 255

Uses a different Virtual MAC address range

HSRP version 2 uses a different virtual MAC address range, to ensure that there are no compatibility errors between the 2 protocols. They are completely different in terms of communication

Support for MD5 authentication

HSRP version 1 only supported clear-text authentication

Under normal conditions in an HSRP environment there is always only 1 router active on any given time. In vPC configurations the Nexus switches can do a trick where they can both be active for an HSRP group. This is required, otherwise the vPC peer-link would be required to carry a lot of traffic between the 2 switches as all incoming traffic on the non-primary vPC peer would need to send it’s traffic to the primary vPC peer to be layer 3 switched.

In vPC environments both routers will respond to the HSRP Virtual MAC address. This is called ‘vPC peer gateway’ which needs to be enabled. This requires that both vPC peers have an equal amount of SVI or other routed interfaces to the Layer 3 network. Otherwise the vPC process can’t ensure a successful failover. Without the peer-gateway command, the vPC switches will not respond to their configured Virtual MAC Address, which may lead to a lot of traffic crossing the peer-link.

The initial configuration of this task is to configure the basic parameters of Cisco HSRP. We will be using the 198.18.111.1 IP address as the default gateway for this segment. After configuring the basic parameter (the Virtual IP address). The protocol already works!

SW1-1

SW1-1(config)# feature hsrp SW1-1(config)# int vlan 111 SW1-1(config-if)# hsrp 111 SW1-1(config-if-hsrp)# ip 198.18.111.1

SW1-2

SW1-2(config)# feature hsrp SW1-2(config)# int vlan 111



SW1-2(config-if)# hsrp 111 SW1-2(config-if-hsrp)# ip 198.18.111.1

According to the MAC address tie-breaker we see that SW1-1 has become the Primary router for this subnet and SW1-2 the hot-standby. For the group number we used the VLAN number. As there was no indication in the question to use any specific group number.

SW1-1(config-if-hsrp)# sh hsrp brief P indicates configured to preempt. | Interface Grp Prio P State Active addr Standby addr Group addr Vlan111 111 100 Active local 198.18.111.12 198.18.111.1 (conf) SW1-1(config-if-hsrp)# SW1-2(config-if-hsrp)# sh hsrp brief P indicates configured to preempt. | Interface Grp Prio P State Active addr Standby addr Group addr Vlan111 111 100 Standby 198.18.111.11 local 198.18.111.1 (conf) SW1-2(config-if-hsrp)#

The next question will decide which router will be the primary router in VLAN 111.

The question states that this should follow the best practice. If we take a look at the configuration guide about this point. We see that Cisco is recommending to make the router HSRP primary if it also has the vPC Primary role assigned. In our case it’s SW1-2 that is the primary vPC device in our network. So we will make this the primary router in our HSRP network too.

There are no priority values given for this task so we can use anything we like. However, when you read ahead of the questions. You will find a question related to Interface Tracking. These questions ask you to subtract some value from the priority. If we use numbers now, it can help us later with these questions.

We will use simple numbers for the priority values. I always keep as a best practice to make the standby switch lower than the default value and the primary switch higher than the default value of the HSRP priority, which is 100.

SW1-1

SW1-1(config)# int vlan 111 SW1-1(config-if)# hsrp 111 SW1-1(config-if-hsrp)# priority 90

SW1-2

SW1-2(config)# feature hsrp



SW1-2(config)# int vlan 111 SW1-2(config-if)# priority 110

After configuring the priority values and flapping the interface or the HSRP group. We see that SW1-2 is now the active router.

SW1-1(config-if)# sh hsrp brief P indicates configured to preempt. | Interface Grp Prio P State Active addr Standby addr Group addr Vlan111 111 90 Standby 198.18.111.12 local 198.18.111.1 (conf) SW1-1(config-if)# SW1-2(config-if-hsrp)# sh hsrp brief P indicates configured to preempt. | Interface Grp Prio P State Active addr Standby addr Group addr Vlan111 111 110 Active local 198.18.111.11 198.18.111.1 (conf) SW1-2(config-if-hsrp)#

Next is the security of the HSRP protocol. In the question it is asked to use an hashed key. Meaning that we will be using MD5 authentication. This also immediately implies that we need to use HSRP version 2, because we wouldn’t have the option to configure MD5 otherwise.

It’s a very dangerous configuration, because MD5 is perfectly configurable without HSRP version 2, but the protocol keeps functioning without any authentication applied then!

Even show commands do not show you what is happening at that moment, but the traffic is not authenticated without specifying the HSRP version 2.

In the question the authentication is also given a specific schedule. This means we will need to be using a ‘key-chain’ to support this. Key chains are a mechanism to switch from authentication keys in a given time schedule. This ensures security of the network and the exchange of keys. The timelines are configurable and it’s also possible to accept an older key longer to ensure a smooth migration to the new key.

SW1-1

SW1-1(config)# key chain HSRP SW1-1(config-keychain)# key 1 SW1-1(config-keychain-key)# key-string IPexpertYEAR1 SW1-1(config-keychain-key)# send-lifetime 12:00:00 Oct 4 2012 23:59:59 Dec 31 2012 SW1-1(config-keychain-key)# accept-life 12:00:00 Oct 4 2012 02:00:00 Jan 01 2013



SW1-1(config-keychain-key)# key 2 SW1-1(config-keychain-key)# key-string IPexpertYEAR2 SW1-1(config-keychain-key)# send-life 00:00:00 Jan 01 2013 infinite SW1-1(config-keychain-key)# accept-life 00:00:00 Jan 01 2013 infinite SW1-1(config-keychain-key)# interface vlan 111 SW1-1(config-if)# hsrp 111 SW1-1(config-if-hsrp)# authentication ? WORD Plain text authentication string (Max Size 8) md5 Use MD5 authentication text Plain text authentication SW1-1(config-if-hsrp)# authentication md5 ? key-chain Set key chain key-string Set key string SW1-1(config-if-hsrp)# authentication md5 key-chain HSRP SW1-1(config-if-hsrp)# exit SW1-1(config-if)# hsrp ? <0-4095> Group number bfd BFD protocol delay HSRP initialisation delay use-bia HSRP uses interface's burned in address version HSRP version SW1-1(config-if)# hsrp version 2 SW1-1(config-if)#

The key chain configuration is done by using a number of keys. By using a send-lifetime and accept-lifetime it’s configurable how long the key is sent and how long it is accepted as a valid key. The initial key that is configured is sent from the current date and time up until December 31st. As the question stated the key should be accepted for another 2 hours, to prevent time synchronization issues between routers, which could cause an outage when authentication is rejected. The next (new) key that is configured has an accept and send lifetime of infinite, because the question did not state when the key should be changed again. Therefore we just use it forever.

The key chain is then configured on the HSRP configuration. Like previously mentioned, this is not enough. HSRP will not use the MD5 configured authentication parameters when version 1 is used. The HSRP version is not configured under the regular HSRP process configuration, but under the interface itself. Version 2 is required for MD5 authentication. After this has been changed the HSRP group uses the authentication parameters and authenticates the hello messages that are send and received.

SW1-2

SW1-2(config)# key chain HSRP SW1-2(config-keychain)# key 1 SW1-2(config-keychain-key)# key-string IPexpertYEAR1 SW1-2(config-keychain-key)# send-lifetime 12:00:00 Oct 4 2012 23:59:59 Dec 31 2012 SW1-2(config-keychain-key)# accept-life 12:00:00 Oct 4 2012 02:00:00 Jan 01 2013 SW1-2(config-keychain-key)# key 2



SW1-2(config-keychain-key)# key-string IPexpertYEAR2 SW1-2(config-keychain-key)# send-life 00:00:00 Jan 01 2013 infinite SW1-2(config-keychain-key)# accept-life 00:00:00 Jan 01 2013 infinite SW1-2(config-keychain-key)# interface vlan 111 SW1-2(config-if)# hsrp 111 SW1-2(config-if-hsrp)# authentication md5 key-chain HSRP SW1-2(config-if-hsrp)# exit SW1-2(config-if)# hsrp version 2 SW1-2(config-if)#

After both switches have been configured, the authentication is now used for the HSRP process.

SW1-2# sh hsrp brief P indicates configured to preempt. | Interface Grp Prio P State Active addr Standby addr Group addr Vlan111 111 110 Active local 198.18.111.11 198.18.111.1 (conf) SW1-2# SW1-2# sh hsrp Vlan111 - Group 111 (HSRP-V2) (IPv4) Local state is Active, priority 110 (Cfged 110) Forwarding threshold(for vPC), lower: 1 upper: 110 Hellotime 3 sec, holdtime 10 sec Next hello sent in 0.163000 sec(s) Virtual IP address is 198.18.111.1 (Cfged) Active router is local Standby router is 198.18.111.11 , priority 90 expires in 5.503000 sec(s) Authentication MD5, key-chain HSRP Virtual mac address is 0000.0c9f.f06f (Default MAC) 5 state changes, last state change 01:38:49 IP redundancy name is hsrp-Vlan111-111 (default) SW1-2#

The following question relates to a feature called ‘pre-emption’. This feature ensures that when a router comes online in an HSRP network and it has a higher priority configured it will assume the active role. By default the current active router will keep its role when a higher priority router is seen. With preempt enabled the router with the higher priority value will always take over (take back) it’s active role.

The take over process can be delayed to ensure that the router that comes online (which is usually after a reboot), is delayed before taking back it’s active role to have all other protocol running on this router to synchronize correctly before it will attract traffic.

The question states that the active router has been rebooted and then comes online. This means that the delay is configured for a reload to ensure all protocols have synchronized.

SW1-1

SW1-1(config-if-hsrp)# int vlan 111



SW1-1(config-if)# hsrp 111 SW1-1(config-if-hsrp)# preempt delay reload 180 SW1-1(config-if-hsrp)#

SW1-2

SW1-2(config-if-hsrp)# int vlan 111 SW1-2(config-if)# hsrp 111 SW1-2(config-if-hsrp)# preempt delay reload 180 SW1-2(config-if-hsrp)#

The next question is related to giving the HSRP process a name, which is an easy knob in the HSRP configuration.

SW1-1

SW1-1(config-if-hsrp)# int vlan 111 SW1-1(config-if)# hsrp 111 SW1-1(config-if-hsrp)# name IPexpertVLAN111 SW1-1(config-if-hsrp)#

SW1-2

SW1-2(config-if-hsrp)# int vlan 111 SW1-2(config-if)# hsrp 111 SW1-2(config-if-hsrp)# name IPexpertVLAN111 SW1-2(config-if-hsrp)#

Next are changing timers. With the default timers it’s only possible to configure up to a hello-interval of 1 second. With the special knob ‘msec’ it’s possible to configure sub-second hello-timers and therefore a convergence within 1 second

SW1-1

SW1-1(config-if-hsrp)# int vlan 111 SW1-1(config-if)# hsrp 111 SW1-1(config-if-hsrp)# timers msec 300 msec 1000 SW1-1(config-if-hsrp)#

SW1-2

SW1-2(config-if-hsrp)# int vlan 111 SW1-2(config-if)# hsrp 111 SW1-2(config-if-hsrp)# timers msec 300 msec 1000 SW1-2(config-if-hsrp)#

The next 2 questions are related to HSRP object tracking. This tracking ensures that when the tracking group fails, the HSRP priority is lowered. In legacy scenarios when the priority falls under the priority of the hot-standby router and pre-emption is configured. The hot-standby takes over the active role.



In this scenario on NX-OS with vPC’s on the switch. The failure of a switch will result in a failover to the hot-standby router of course. But there is no single distinction anymore between the active and hot-standby router. Both routers are forwarding traffic. What can be configured by using the priority command is to configure thresholds when the traffic forwarding should be stopped and traffic is no longer layer 3 routed on the NX-OS device, but rather send across the vPC Peer-link towards the vPC peer for layer 3 routing.

In our case of the questions, it should take 2 uplinks to fail on the active switch before it stops forwarding traffic on Layer 3 and sending traffic on a Layer 2 basis towards the vPC peer.

Currently we have a priority of 110 configured on the primary switch (or any other number that you configured in a previous question). 10% of this value should be subtracted once an uplink fails. When a second interface fails it should stop layer 3 forwarding completely.

10% of the 110 value is 11. So we subtract 11 from each of the failing Ethernet uplinks. Then we configure the thresholds for forwarding appropriately. Remember to use a different value for the hot-standby switch as 10% from 90 is only 9 and not 11.

Depending on the values you chose at the earlier question. You need to ensure that when you subtract 2 times 10% from that value, it should be lower than the hot-standby switch, so it will assume the active role after 2 uplinks have failed.

In our case the priority value will drop with 22 (2 times 11) and therefore be 88. Therefore the hot-standby switch will take over as it’s configured for priority 90.

Remember that pre-emption needs to be configured for a fail-over when priority drops

SW1-1

SW1-1(config-if-hsrp)# int vlan 111 SW1-1(config-if)# hsrp 111 SW1-1(config-if-hsrp)# priority 90 forwarding-threshold lower 72 upper 81 SW1-1(config-if-hsrp)# exit SW1-1(config-if)# exit SW1-1(config)# track 1 ? <CR> interface Interface to track ip IPv4 protocol ipv6 IPv6 protocol list Object tracking list SW1-1(config)# track 1 interface ethernet3/3 ? . Sub interface separator ip IPv4 parameters ipv6 IPv6 parameters line-protocol Track interface line-protocol SW1-1(config)# track 1 interface ethernet3/3 line-protocol SW1-1(config-track)# exit SW1-1(config)# track 2 interface e3/5 line-protocol



SW1-1(config-track)# exit SW1-1(config)# int vlan 111 SW1-1(config-if)# hsrp 111 SW1-1(config-if-hsrp)# track 1 decrement 9 SW1-1(config-if-hsrp)# track 2 decrement 9 SW1-1(config-if-hsrp)#

SW1-2

SW1-2(config)# track 1 interface ethernet3/7 line-protocol SW1-2(config-track)# exit SW1-2(config)# track 2 interface e3/9 line-protocol SW1-2(config-track)# exit SW1-2(config)# int vlan 111 SW1-2(config-if)# hsrp 111 SW1-2(config-if-hsrp)# priority 110 forwarding-threshold lower 88 upper 99 SW1-2(config-if-hsrp)# track 1 decrement 11 SW1-2(config-if-hsrp)# track 2 decrement 11

Interface tracking can be done using multiple configuration options. The question now states that we need to track an interface for going down. This means we do not need to do anything with the IP related tracking features for now. We can just track on line-protocol going down and the tracking group will go down.

The final question of this task is to use a physical interface MAC address instead of a virtual MAC. Some systems might have trouble seeing special MAC addresses like the one used by HSRP. Therefore there is a fallback to a physical MAC address to use as Virtual. This feature will ‘borrow’ a MAC address from the physical interface or SVI from the Active router. When the active router fails it’s MAC address is then moved just like a regular HSRP action to the hot-standby router. The MAC address will stay the same even though it does not belong to the new active router.

The configuration is fairly easy, although not specified under the HSRP configuration, but under the interface directly.

SW1-1

SW1-1(config-if-hsrp)# int vlan 111 SW1-1(config-if)# hsrp use-bia

SW1-2

SW1-2(config-if-hsrp)# int vlan 111 SW1-2(config-if)# hsrp use-bia

If we now check the HSRP group, we see that it’s using a real MAC address as it’s virtual MAC address.

SW1-1(config-if)# sh hsrp Vlan111 - Group 111 (HSRP-V2) (IPv4) Local state is Active, priority 72 (Cfged 90) Forwarding threshold(for vPC), lower: 72 upper: 81



Hellotime 300 msec, holdtime 1 sec Next hello sent in 0.255000 sec(s) Virtual IP address is 198.18.111.1 (Cfged) Active router is local Standby router is 198.18.111.12 , priority 110 expires in 0.915000 sec(s) Authentication MD5, key-chain HSRP Virtual mac address is 0024.98e8.01c4 (Interface MAC - use-bia enabled) 2 state changes, last state change 01:21:04 Track object 1 state DOWN decrement 9 Track object 2 state DOWN decrement 9 IP redundancy name is IPexpertVLAN111 (cfgd) SW1-1(config-if)#

Next task is about the standards based solution based on the Cisco proprietary HSRP protocol!

Task 6: VRRP The second FHRP protocol that is discussed is the Virtual Router Redundancy Protocol (VRRP). This protocol is very similar in operation to Cisco’s HSRP. It borrows a lot of technical specifications. This protocol is an open standard based on RFC 3768 (version 2) or RFC 5798 (version 3).

All routing vendors have implementations for the VRRP. This makes it the best practice protocol to use when you are using a mixed-vendor environment. Cisco is not active in implementing new features in VRRP compared to HSRP. A lot of advanced features like MD5 authentication or IPv6 support are not available in latest releases of NX-OS. Version 3 of VRRP (which brings IPv6 support) is not available in NX-OS yet. Other features like BFD tracking, which have long been available in HSRP, are only recently implemented for VRRP.

The questions in this task are very similar to the questions in the HSRP task as the protocol works almost the same as HSRP.

The first few questions are related to configuring the basic configuration parameters of the VRRP. The third question however is asking for a special configuration. A special virtual MAC address should be used for the VRRP group.

VRRP uses MAC address from the range of 0000.5E00.0100 to 0000.5E00.01FF. We are asked to configure a virtual MAC address of 0000.5E00.0174. This means we should use a special group number. As the group number is the last byte of the MAC address, just like with HSRP.

When you convert the number 74 from hexadecimal to decimal we do the following steps:

We first convert the hex number 7 to binary:

0111

Next is converting the number 4 to binary:



0100

A hex character consists of 2 nibbles making up 1 byte. We put the 2 nibbles together and convert the binary byte to a decimal number:

01110100 = 116

This means we will be using VRRP group number 116.

SW1-1

SW1-1(config-if)# int vlan 121 SW1-1(config-if)# vrrp 116 SW1-1(config-if-vrrp)# address 198.18.121.254

SW1-2

SW1-2(config-if)# int vlan 121 SW1-2(config-if)# vrrp 116 SW1-2(config-if-vrrp)# address 198.18.121.254 SW1-2(config-if-vrrp)# priority 200

After configuration, it seems that the VRRP group is not working correctly.

SW1-2(config-if)# sh vrrp Interface VR IpVersion Pri Time Pre State VR IP addr --------------------------------------------------------------- Vlan121 116 IPV4 200 1 s Y Init 198.18.121.254

This is a very strange message as the reason why it’s staying on “Init” is very clear when you take a closer look.

SW1-2(config-if)# sh vrrp detail Vlan121 - Group 116 (IPV4) State is Init(Administratively down) Virtual IP address is 198.18.121.254 Priority 200, Configured 200 Forwarding threshold(for VPC), lower: 1 upper: 200 Advertisement interval 1 Preemption enabled Virtual MAC address is 0000.5e00.0174 Master router is Unknown

It’s clearly seen that the VRRP group is by default on a shutdown state and it needs to be enabled before it will start sending VRRP packets.

We can immediately see that the virtual MAC address is as it should be per the question. This means we did our calculation correctly.



SW1-1

SW1-1(config-if)# int vlan 121 SW1-1(config-if)# vrrp 116 SW1-1(config-if-vrrp)# no shutdown

SW1-2

SW1-2(config-if)# int vlan 121 SW1-2(config-if)# vrrp 116 SW1-2(config-if-vrrp)# no shutdown

After a no shutdown the VRRP group comes online!

SW1-2(config-if-vrrp)# sh vrrp Interface VR IpVersion Pri Time Pre State VR IP addr --------------------------------------------------------------- Vlan121 116 IPV4 200 1 s Y Master 198.18.121.254

The next task asks to configure authentication on the VRRP group. MD5 authentication is not supported in NX-OS at the time of writing. Authentication is therefore easily configured.

SW1-1

SW1-1(config-if)# int vlan 121 SW1-1(config-if)# vrrp 116 SW1-1(config-if-vrrp)# authentication text IPexpert

SW1-2

SW1-2(config-if)# int vlan 121 SW1-2(config-if)# vrrp 116 SW1-2(config-if-vrrp)# authentication text IPexpert

Now we verify we indeed see that the process is authenticated on the routers

SW1-2(config-if-vrrp)# sh vrrp detail Vlan121 - Group 116 (IPV4) State is Master Virtual IP address is 198.18.121.254 Priority 200, Configured 200 Forwarding threshold(for VPC), lower: 1 upper: 200 Advertisement interval 1 Preemption enabled Authentication text "IPexpert" Virtual MAC address is 0000.5e00.0174 Master router is Local

One thing different from the Cisco HSRP is that pre-emption is enabled by default. Again this ensures that when a higher priority router comes online on the subnet, the routers will take over the active role pro-actively. In the question is stated that we don’t want this to happen, so we should disable it.



We are asked to only configure the Active switch. This means that an initial failover will happen right away due to pre-emption, but a fall-back will only occur when the other switch fails.

SW1-2

SW1-2(config-if)# int vlan 121 SW1-2(config-if)# vrrp 116 SW1-2(config-if-vrrp)# no preempt

The next question is a little more tricky. VRRP does not support the tracking of Layer 2 interfaces. It does support direct tracking of Layer 3 interfaces, where no tracking groups are necessary. Instead of tracking an interface in this question we are tracking the reachability of a route in the routing table.

This means that as soon as SW3 fails, the Loopback address will disappear from the routing table on SW1-2. This as per our question means we need to do a failover to the other VRRP router. This process should be delayed with 30 seconds. Remember to not override your previous tracking group configuration, otherwise you will lose points in the HSRP task!

SW1-2

SW1-2(config)# track 3 ip route 198.18.0.3/32 reachability SW1-2(config-track)# ? delay Tracking delay no Negate a command or set its defaults vrf Configure VPN Routing/Forwarding table end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1-2(config-track)# delay ? down Delay down change notification up Delay up change notification SW1-2(config-track)# delay down 30

There is no specification how much the group should be decremented. Therefore we just ensure that the priority value is decreased under the default of 100. This would cause the switch to take over the active role as we did not disable pre-emption on the other switch.

SW1-2

SW1-2(config-if)# int vlan 121 SW1-2(config-if)# vrrp 116 SW1-2(config-if-vrrp)# track 1 decrement 101

The final question of this task is to configure timers of VRRP. The only trick you have to change timers is to change the interval in which VRRP packets are advertised to all VRRP running routers on the subnet. The holddown timer is not adjustable. This is by default 3 times the hello-interval including



some priority value timing. For any given configured priority value setting the timers are adjusted a little bit, which is the way how VRRP elects it’s active and standby router.

If we want to reach a down signaling within 10 seconds, we will use a hello-interval of 3 seconds. When 3 hello packets are missed after 9 seconds including a little priority timing we will signal a VRRP neighbor down within 10 seconds.

SW1-1

SW1-1(config-if)# int vlan 121 SW1-1(config-if)# vrrp 116 SW1-1(config-if-vrrp)# advertisement-interval 3

SW1-2

SW1-2(config-if)# int vlan 121 SW1-2(config-if)# vrrp 116 SW1-2(config-if-vrrp)# advertisement-interval 3

Now the VRRP task is finished. The final task for FHRP protocols is another Cisco proprietary protocol supporting Load Balancing.

Task 7: GLBP The last but not least FHRP that is tested in our workbook is another Cisco proprietary protocol called the Gateway LoadBalancing Protocol (GLBP). This protocol is the most recent one of all FHRP’s. It uses a little different approach than HSRP and VRRP. The other FHRP’s are using a single active router in normal environments. In vPC environments this is a little different as both HSRP and VRRP gateways are active and responding to client requests. The GLBP protocol supports this by default, but it’s also possible to configure the load balancing , which is not possible in case of the vPC load balancing trick.

Routers are given specific roles in the GLBP configuration:

Active Virtual Forwarder (AVF)

There can be multiple AVF routers in the GLBP network configuration

These are routers that are forwarding Layer 3 traffic according to the weight that is configured on them to allow the traffic coming in.

AVFs have their own dedicated Virtual MAC Address for each separate AVF

Active Virtual Gateway (AVG)



This is only a single router in the GLBP configuration which is selected based on priority values.

This router will be responding to all ARP requests of the subnet.

The Active Virtual Gateway is aware of all Virtual MAC Addresses in the GLBP configuration and uses a Round Robin, Weighted or Host Dependent mechanism to answer the ARP requests. This means that each ARP request can have a different MAC address in the ARP reply. This is how the load balancing is achieved.

The failover of an AVF is handled using 2 timers. The Redirect Timer is used to gracefully take out the Virtual MAC address of the AVF of the Virtual MAC Pool maintained by the AVG. The Secondary Hold Timer is used when a AVF has completely failed and the Virtual MAC Address of this AVF will be freed in the pool and is available for possible new AVF routers.

There are 3 ways of answering the ARP requests on the AVG and achieving load balancing. These are explained in the following table.

Mode Description

Round Robin This is the most granular way of load balancing as the AVG answers a other Virtual MAC Address on each ARP request. Meaning traffic is equally load balanced across all AVF’s

Weighted This mode enables the use of weight values on AVFs. When a higher weight is advertised by an AVF. The AVG will answer more ARP requests with this Virtual MAC Address so this AVF will handle more traffic than others

Host dependent The host dependent mode guarantees that the same host receives the same AVF each time it does an ARP request.

You can change the weight or priority values for becoming an AVF or AVG according to interface tracking, just like in the other FHRPs.

In the current software releases used for this version of the CCIE Datacenter test the GLBP is only supported on the Nexus 7000 series switches. Additionally there is currently no support for IPv6 in the protocol.



The GLBP protocol is illustrated using the following drawing.

Now to start our configuration we begin with fairly basic tasks of configuring the GLBP on VLAN 222 on our Nexus 7000 series switches.

SW1-1

SW1-1(config-if)# feature glbp SW1-1(config)# int vlan 222 SW1-1(config-if)# glbp 222 SW1-1(config-if-glbp)# ip 198.18.222.55 SW1-1(config-if-glbp)# no shutdown

SW1-2

SW1-2(config-if)# feature glbp SW1-2(config)# int vlan 222 SW1-2(config-if)# glbp 222 SW1-2(config-if-glbp)# ip 198.18.222.55 SW1-2(config-if-glbp)# no shutdown

The next 2 tasks first require to assign the AVG role to SW1-1 and that both routers should be AVF. By default all GLBP capable routers are used for load balancing traffic to. A router can decide for itself if it does not want to participate in the load balancing anymore, by configuring upper and lower limits. We will get there in a later question in this task.

SW1-1

SW1-1(config)# int vlan 222 SW1-1(config-if)# glbp 222 SW1-1(config-if-glbp)# priority 110



The next question is related to the previous one. We need to ensure that both routers are having an AVF role in the network, but do want to remove the router as an AVF when the Loopback address of the upstream switches disappears from the routing table.

Just like in the VRRP task we can use IP Route Reachability tracking for this. We will configure lower thresholds when the router will give up its AVF role. Again do not forget to use unique numbers for the tracking groups. Otherwise you will overwrite configuration from a previous task.

SW1-1

SW1-1(config)# track 4 ip route 198.18.0.2/32 reachability SW1-1(config)# int vlan 222 SW1-1(config-if)# glbp 222 SW1-1(config-if-glbp)# track 1 decrement 20 SW1-1(config-if-glbp)# weighting 110 lower 90 upper 110

SW1-2

SW1-2(config)# track 4 ip route 198.18.0.3/32 reachability SW1-2(config)# int vlan 222 SW1-2(config-if)# glbp 222 SW1-2(config-if-glbp)# track 1 decrement 20 SW1-2(config-if-glbp)# weighting 110 lower 95 upper 105

With this configuration we will take out the router from the AVF role (and the AVG from its role when applicable) when the reachability towards a IP prefix is gone.

This process should be delayed. We can do this in 2 ways. The first is what we did in the VRRP task, by delaying the tracking group for signaling the loss of reachability. If you read the question carefully it mentions that you need to support yourself for a failure of a current AVF. The AVG will only take out a failed AVF from the pool after the redirect timer has expired, after which it will keep its Virtual MAC reserved for another timer (secondary timeout timer).

When we change the redirect timer the AVG will take out the AVF and that is what this question is asking for. Keep in mind to change both switches as they both might become the AVG at some point.

SW1-1

SW1-1(config)# int vlan 222 SW1-1(config-if)# glbp 222 SW1-1(config-if-glbp)# timers redirect 180

SW1-2

SW1-2(config)# int vlan 222 SW1-2(config-if)# glbp 222 SW1-2(config-if-glbp)# timers redirect 180



The next task is regarding the pre-emption of the AVG role. The preempt command is disabled by default in the GLBP protocol. By default the delay for taking over from a lower priority router is set to 3600 seconds. This is significantly longer than any other timer in a FHRP. The reason for this is that a router immediately becomes an AVF when it comes online, so it’s already forwarding traffic. The switchover of the AVG role is therefore not that important anymore.

SW1-1

SW1-1(config)# int vlan 222 SW1-1(config-if)# glbp 222 SW1-1(config-if-glbp)# preempt delay minimum 30

SW1-2

SW1-2(config)# int vlan 222 SW1-2(config-if)# glbp 222 SW1-2(config-if-glbp)# preempt delay minimum 30

The final task is regarding ISSU’s on GLBP enabled routers. The GLBP protocol supports a nice feature for supporting Non Stop Forwarding. You can set timers as short as you like, but when a restart is done of any neighbor or the router itself, it can temporarily extend the timers for the duration of the Graceful Restart.

This is enabled with a single command under the global configuration:

SW1-1

SW1-1(config)# glbp timers extended-hold

SW1-2

SW1-2(config)# glbp timers extended-hold

And all FHRP tasks are successfully completed!

Task 8: Virtual Port-Channels (vPCs) and FabricPath The last task of this High Availability chapter will be about a special feature for using vPCs in a FabricPath enabled network. This is due to the architecture of the FabricPath protocol that it is not possible to configure by default.

The FabricPath network uses Equal Cost MultiPathing to reach any given destination (MAC address) in the network. Now when any FabricPath enabled router will perform a lookup on the next-hop it will result in a Destination Switch ID. For vPC connected hosts or switches this is different as they are connected to 2 different FabricPath Switch IDs. It will result in strange behavior and non-deterministic traffic flows throughout the FabricPath network.



The following drawing illustrates the problem in more detail:

The drawing illustrates that each packet, depending on the Port-channel hashing can be sent from or to either S1 or S2. Everytime S3 learns the MAC address of the PC connected it will update its routing table accordingly. This introduces a MAC address flapping issue on S3.

The solution that Cisco came up with has been called vPC+. This means that the vPC switches are compatible with FabricPath enabled networks.

The solution is that the whole vPC domain will be virtualized as a single FabricPath Switch ID. In that way the entry for the MAC address of the PC connected has a Destination Switch ID which is the virtual ID configured statically on the vPC domain. That way the other FabricPath switches can still use Equal Cost Multi-Pathing towards the host just like any other destination in the FabricPath network without causing any loops.

The configuration is fairly simple. The task asks you to configure the FabricPath network first after which it should be made ready for the vPC configuration.

As there is no host connected to the switches, you just need to prepare the configuration for it. After this pre-configuration there is nothing different from a standard vPC configuration.



The topology that will be used for this lab is laid out in Drawing 3 in the workbook and for your convenience showed below:

Now on to the configuration. We enable the FabricPath network on all interfaces.

SW2

SW2(config)# SW2(config)# feature-set fabricpath SW2(config)# vlan 666 SW2(config-vlan)# mode fabricpath SW2(config)# interface Ethernet1/1 SW2(config-if)# switchport mode fabricpath SW2(config)# interface Ethernet1/5 SW2(config-if)# switchport mode fabricpath

SW3

SW3(config)# SW3(config)# feature-set fabricpath SW3(config)# vlan 666 SW3(config-vlan)# mode fabricpath SW3(config)# interface Ethernet1/1 SW3(config-if)# switchport mode fabricpath SW3(config)# interface Ethernet1/5 SW3(config-if)# switchport mode fabricpath



SW1-1

SW1-1(config)# SW1-1(config)# feature-set fabricpath SW1-1(config)# vlan 666 SW1-1(config-vlan)# mode fabricpath SW1-1(config)# interface Ethernet2/9 SW1-1(config-if)# switchport mode fabricpath SW1-1(config)# interface Ethernet2/11 SW1-1(config-if)# switchport mode fabricpath

SW1-2

SW1-2(config)# SW1-2(config)# feature-set fabricpath SW1-2(config)# vlan 666 SW1-2(config-vlan)# mode fabricpath SW1-2(config)# interface Ethernet2/13 SW1-2(config-if)# switchport mode fabricpath SW1-2(config)# interface Ethernet2/15 SW1-2(config-if)# switchport mode fabricpath

Now to finish the chapter we need to configure the static FabricPath Switch ID under the vPC domain configuration.

SW2

SW2(config)# SW2(config)# vpc domain 100 SW2(config-vpc-domain)# fabricpath switch-id 100

SW3

SW3(config)# SW3(config)# vpc domain 100 SW3(config-vpc-domain)# fabricpath switch-id 100

You finished Chapter 4 successfully! We also finished all Networking chapters at this moment and we will continue with the Fibre-Channel features of both the Cisco MDS switches and the Nexus 5000 switches.



Chapter 5: Data Center Storage Networking

Chapter 5: Data Center Storage networking is intended to let you be familiar with the Storage Networking features on the Cisco MDS switches. Configuring traditional Fibre Channel networks and basic Fibre Channel features.

We highly recommend creating your own diagram at the beginning of each lab so you are able to draw on your own diagram, making it much easier when you step into the real lab.

Multiple topology drawings are available for this chapter.



General Rules Try to diagram out the task. Draw your own connections the way you like it

Create a checklist to aid as you work thru the lab

Take a very close read of the tasks to ensure you don’t miss any points during grading!

Take your time. This is not a Mock Lab, so no time constraints are in place for finishing this particular chapter




Solutions

Task 1: Initial set-up The first task of this storage networking chapter is to set-up the initial topology. We first configure hostnames for the storage equipment and we configure management IP addressing. Our topology in this exercise is fairly simple. This is because most of the storage networking (Fibre Channel) features are tested in a separate CCIE Storage track. Therefore the CCIE Data Center doesn’t have a lot of focus on the SAN part.

However, the Fibre Channel world is completely different from the Ethernet world and therefore we require some extensive configuration to have a good understanding of the Fibre Channel protocol and the many differences that there are with Ethernet switching.

The Fibre Channel protocol has been around for a long time already and is used widely in the world. It makes it possible for servers to extend the SCSI commands to hard disks over a data network. This is the initial purpose of the Fibre Channel network. Fibre Channel works a little different in the OSI model than Ethernet works.

You can say that the Fibre Channel protocol is much more intelligent than Ethernet. Most of the routing protocols that we need to implement in the Ethernet and IP world are by default enabled in the Fibre Channel world and do not require any configuration to work properly. Most of the protocols can come online by default and are automatically converging fine.

The Fibre Channel protocol uses a special ‘ID’ of disks and hosts (or targets and initiators) that identify the host towards the network. These ID’s are called World Wide Names (WWNs). A device always has 2 of these ID’s, these are a World Wide Port Name (WWPN) and a World Wide Node Name (WWNN). It makes sense that the WWPN is assigned for every port on the device and the WWNN is assigned to a single device. A special WWN is assigned to every MDS switch. These are called Switch World Wide Names (SWWNs). These ID’s are used to identify any MDS switch in a topology and can be used to reference a switch. The ID is also used by the switch in many of the control-plane protocols that are running in the FC control-plane.

These IDs are most comparable to MAC addresses in the Ethernet world, except that the WWNs are never used in the data communication! This is a common misconception that, like in Ethernet, the physical addresses are used as Layer 2 protocol. They are just for identifying the host or port towards the Fibre Channel environment.

After the identification the FC switch will hand out the logical addressing (like IP addresses in the Ethernet/IP world). These addresses are called FCIDs. They are comprised of a 3-byte / 24-bit number represented as 6 hexadecimal characters: 0x890ABC. The first byte of the FCID is the so-called FC Domain ID of the MDS switch. The second byte is the area code and the third byte is the address of the host. The Domain ID is an ID used by the Fibre Channel environment between FC switches for routing!



The first byte of an FCID is used and a routing lookup is done against it to determine the destination FC switch in the network. When the destination switch is reached, the area code is used to determine the exit interface of this packet. Usually in an FC environment a whole area code is handed out to one interface on the switch. After that each host/disk that is connected behind that interface, in case it’s an NPV port or a Loop port, different devices get an ID from that area handed out.

The model looks a bit like a DHCP environment, except it’s distributed and each FC switch is participating in this handing out of addresses. There is one switch in the network responsible for handing out Domain IDs to FC switches and this is the Principal switch. This Principal switch is chosen according to the highest priority number of as a tie-breaker the highest SWWN.

This model is supported by some control-plane protocols that each host on the network needs to perform when it is connected to a switch. This process is called an FLOGI or Fabric Login process. After the FLOGI a Port Login or PLOGI is done to identify all capabilities of the just connected host.

In the configuration tasks more is explained about the Fibre Channel protocol.

The topology we use in this lab is simple. We have 2 native MDS switches. In a later chapter we will include the Nexus switches in the Fibre Channel topology as well, but we are currently only focusing on the native Fibre Channel topologies.



Connected to the 2 MDS switches are 2 JBODS. These devices are nothing more than containers for SCSI disks that are capable of logging in to a Fibre Channel switch. Fibre Channel attached servers are then able to connect to this physical hard-disk through the Fibre Channel network.

In the real world these disks would communicate to a Storage Filer where RAID configurations and other access methods are applied.

Our topology will use the disks as dual-attached devices where we can simulate having a lot of devices on the Fibre Channel network.

During this chapter the detailed solution guide will explain Fibre Channel (FC) concepts. This will not be an in-depth explanation, but a basic explanation to ensure you understand the concept and how it’s related to our CCIE Data Center topology.

The MDS switches will boot up with an empty configuration where we need to apply all configuration. The MDS switches run the same code as the Nexus switches. The old SAN-OS has been rebranded to NX-OS since version 4.0. The CLI is exactly the same, except that the FC protocol all has their own configuration parameters which are not compatible with their Ethernet equivalents.

Initially we need to give those hostnames and configure the mgmt0 management Ethernet interfaces.

MDS1

Invalid admin password. Please try again. Enter the password for "admin": ipexpert Confirm the password for "admin":ipexpert SETUP: Admin password config failed. Password is not strong enough:it is should be atleast 8 characters

NX-OS enforces you to use a strong password!

Error(s) while applying config. Enter the password for "admin": IPexpert123 Confirm the password for "admin": IPexpert123 ---- Basic System Configuration Dialog ---- This setup utility will guide you through the basic configuration of the system. Setup configures only enough connectivity for management of the system. Please register Cisco MDS 9000 Family devices promptly with your supplier. Failure to register may affect response times for initial service calls. MDS devices must be registered to receive entitled support services. Press Enter at anytime to skip a dialog. Use ctrl-c at anytime to skip the remaining dialogs.



Would you like to enter the basic configuration dialog (yes/no): no MDS Switch 10.2.8.74 login: admin Password: IPexpert123 Cisco Storage Area Networking Operating System (SAN-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license. Some parts of this software may be covered under the GNU Public License or the GNU Lesser General Public License. A copy of each such license is available at http://www.gnu.org/licenses/gpl.html and http://www.gnu.org/licenses/lgpl.html switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# switchname MDS1 MDS1(config)# interface mgmt0 MDS1(config-if)# ip address 172.16.100.10/24 MDS1(config-if)# no shutdown MDS1(config-if)# ip default-gateway 172.16.100.254 MDS1(config)# ntp server 172.16.100.254

MDS2

switch(config)# switch(config)# switchname MDS2 MDS2(config)# interface mgmt0 MDS2(config-if)# ip address 172.16.100.11/24 MDS2(config-if)# no shutdown MDS2(config-if)# ip default-gateway 172.16.100.254 MDS2(config)# ntp server 172.16.100.254

Verify that time is synchronized. Keep in mind that this might take up to 5 minutes before a full synchronization is completed.

The lab states that we shouldn’t use any automatic selection for interface speeds. This means that we need to specify static link speeds on every link that connects a host to our MDS switches. Besides that we shouldn’t use a feature of the MDS switches that let’s the switch automatically select the type of interface that should be selected according to the connected device.

The type of interface is a crucial point in configuring interfaces in a Fibre Channel network. This changes the behavior of the interface and the FC protocol. The Cisco MDS switches have a feature that they can auto detect what kind of interface is connected on the other side and set the interface type according to that.


http://www.gnu.org/licenses/gpl.html

http://www.gnu.org/licenses/lgpl.html



The following table shows all the various interface types, which are configurable on the MDS switch and what they do:

Mode Description

E The E-port is a so-called Fabric Extension port. This port ensures the extension of the Fibre Channel fabric. A connection between 2 switches is always configured as an E-port. Pay attention to oversubscription on E-ports. You might need more buffer credits on these ports as a lot of traffic might go across them.

TE The TE-port is equal to the E-port in positioning, except that this port has the Trunking protocol enabled, which enables the use of VSANs between Cisco MDS switches.

F The Fabric Port is the most common interface. You can attach Hosts and JBODs to these interfaces. On the Host or JBOD itself it’s called an N-port. So an N-port always connects to an F-port.

FL A Fabric Loop port is used to connect Public Arbitrated Loops to the MDS switch. Arbitrated Loops are self-supporting FC rings without switches between devices. This is a way to connect them to an FC infrastructure

Fx The Fx port can be either an F or an FL port. It will automatically detect if it’s a normal N-port connected behind of it or an Arbitrated Loop. This mode can be the default mode on switches.

TF The TF port is equal to an F-port, but supports the Trunking protocol, to connect VSANs towards hosts or NPV switches

NP NP ports are equal to N-ports on a device, but these ports are on FC switches that are set to N-Port Virtualization mode.

TNP These ports are equal to NP ports, but support the Trunking (EISL) protocol, to transfer VSAN information to NPV switches

SD SD is the SPAN Destination port where a FC analyzer can be connected, like the Cisco FC to Ethernet converter. This traffic receives traffic from other ports for analyzation

ST SPAN Tunnel ports are used as ‘reflector’ ports for Remote SPAN sessions. Traffic that needs to be copied to another device is using this tunnel port for encapsulation.



Mode Description

TL This port connects Translative Loops to the MDS switches. It enables communication between TL, FL and other devices in the FC fabric

Auto This is the default mode on most MDS switches. This mode auto detects the mode in which it needs to operate in. It can become a E, F, FL, TE or TF port. Other port modes need to be manually configured.

In this exercise we shouldn’t use the automatic interface type selection so we need to configure it statically. This is so you practice with the various port types that we need.

In the CCIE Data Center lab it’s typical that you will be stated what kind of JBODs are attached to the switches. These can be anything, from an N port to any kind of Private or Public loop. Be very careful as you need to select the correct interface type, because when this is not properly configured you will not see any FLOGI’s from the attached devices.

There are a few more requirements that we need to comply with. We need to use 200MBps connections towards the JBODs. Pay attention to the capital B in this word. Fibre Channel can be configured to work in 1Gbps, 2Gbps, 4Gbps or 8Gbps speeds. However, the protocol uses an 8B/10B encoding mechanism, which means that for every 8 bits of data, 10 bits are transferred over the link. This comes down to a perfect match in throughput that 1Gbps FC traffic is 100MBps of data traffic and so on.

Now we start configuring the interfaces. You might need to detect how the JBODs are connected. Just try a few interface types until you see FLOGIs coming in. In our example the JBODs are using Public Arbitrated Loops which means we need FL ports.

The ISLs between the switches should be using a standard based solution. Which means they should use a normal ISL protocol and not the Cisco specific (although also standardized now, nobody implements it) EISL encapsulation. We can use the maximum interface bandwidth, which is 4Gbps on our interfaces between the switches.

Pay attention that you need to use a dedicated rate-mode on certain switches for configuring E -ports. This has to do with the way that the ASIC is used. On E-ports all B2B credits are assigned to this interface as it’s

MDS1(config-if)# int fc1/12 MDS1(config-if)# sw mode e fc1/12: (error) Auto/E mode is not allowed in shared rate-mode

On E-ports all B2B credits located on the ASIC are assigned to this interface as it’s required for high bandwidth and longer links.



Finally, all interfaces on the MDS switches are by default set to shutdown (unless specifically configured).

MDS1

MDS1(config)# int fc1/5 MDS1(config-if)# sw mode fl MDS1(config-if)# sw speed 2000 MDS1(config-if)# no shut MDS1(config-if)# int fc1/6 MDS1(config-if)# sw mode fl MDS1(config-if)# sw speed 2000 MDS1(config-if)# no shut MDS1(config-if)# int fc1/1 MDS1(config-if)# sw mode e MDS1(config-if)# sw trunk mode off MDS1(config-if)# sw speed 4000 MDS1(config-if)# no shut MDS1(config-if)# int fc1/2 MDS1(config-if)# sw mode e MDS1(config-if)# sw trunk mode off MDS1(config-if)# sw speed 4000 MDS1(config-if)# no shut MDS1(config-if)# int fc1/3 MDS1(config-if)# sw mode e MDS1(config-if)# sw trunk mode off MDS1(config-if)# sw speed 4000 MDS1(config-if)# no shut MDS1(config-if)# int fc1/4 MDS1(config-if)# sw mode e MDS1(config-if)# sw trunk mode off MDS1(config-if)# sw speed 4000 MDS1(config-if)# no shut

Keep in mind that by default the EISL protocol is enabled on E ports, so ports between MDS switches automatically become TE ports, when nothing is changed.

MDS1(config-if)# do sh int fc1/12 fc1/12 is down (Link failure or not-connected) Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:0c:00:0d:ec:6f:4f:40 Admin port mode is E, trunk mode is on snmp link state traps are enabled Port vsan is 1 Receive data field Size is 2112 Beacon is turned off 5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 1 frames input, 172 bytes 0 discards, 0 errors 0 CRC, 0 unknown class 0 too long, 0 too short 1 frames output, 172 bytes 0 discards, 0 errors 0 input OLS, 0 LRR, 0 NOS, 0 loop inits 0 output OLS, 0 LRR, 0 NOS, 0 loop inits



MDS1(config-if)#

We are now using standard based ISLs and we configured a few JBOD’s to the MDS switches. If everything is working correctly. We should see FLOGI’s being done to the switch:

MDS1(config-if)# do sh flogi database --------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME --------------------------------------------------------------------------- fc1/5 1 0x1b02e8 c6:80:00:11:0d:07:fa:ce c5:9a:00:06:2b:01:00:07 fc1/5 1 0x1b02ef c6:80:00:11:0d:01:01:07 c5:9a:00:06:2b:01:00:07 Total number of flogi = 2. MDS1(config-if)#

And it’s working! We see disks from the JBOD logging in to the MDS switch successfully!

Now we need to configure the second switch. The task here is that the JBOD interfaces should use automatic selection of speeds, but still should comply with the previous task of 200MBps transfer rates. This is done by specifying a maximum speed of the automatic selection.

MDS2

MDS2(config)# int fc1/5 MDS2(config-if)# sw mode fl MDS2(config-if)# sw speed auto max 2000 MDS2(config-if)# no shut MDS2(config-if)# int fc1/6 MDS2(config-if)# sw mode fl MDS2(config-if)# sw speed auto max 2000 MDS2(config-if)# no shut MDS2(config-if)# int fc1/1 MDS2(config-if)# sw mode e MDS2(config-if)# sw trunk mode off MDS2(config-if)# sw speed 4000 MDS2(config-if)# no shut MDS2(config-if)# int fc1/2 MDS2(config-if)# sw mode e MDS2(config-if)# sw trunk mode off MDS2(config-if)# sw speed 4000 MDS2(config-if)# no shut MDS2(config-if)# int fc1/3 MDS2(config-if)# sw mode e MDS2(config-if)# sw trunk mode off MDS2(config-if)# sw speed 4000 MDS2(config-if)# no shut MDS2(config-if)# int fc1/4 MDS2(config-if)# sw mode e



MDS2(config-if)# sw trunk mode off MDS2(config-if)# sw speed 4000 MDS2(config-if)# no shut

Next up is that we need to give descriptive names to the ports in the fabric, but we can’t use the command that we are used to use. The MDS switches know besides the description field another field that can be used for describing the use of the interface. This feature is known as the port owner. Same as for the description this field can exist of 80 characters describing the owner or the function of the switch port.

MDS1

MDS1(config)# int fc1/5 MDS1(config-if)# sw owner JBOD1A MDS1(config-if)# int fc1/6 MDS1(config-if)# sw owner JBOD2A MDS1(config-if)# int fc1/1 MDS1(config-if)# sw owner MDS2 port 1 MDS1(config-if)# int fc1/2 MDS1(config-if)# sw owner MDS2 port 2 MDS1(config-if)# int fc1/3 MDS1(config-if)# sw owner MDS2 port 3 MDS1(config-if)# int fc1/4 MDS1(config-if)# sw owner MDS2 port 4

MDS2

MDS2(config)# int fc1/5 MDS2(config-if)# sw owner JBOD1B MDS2(config-if)# int fc1/6 MDS2(config-if)# sw owner JBOD2B MDS2(config-if)# int fc1/1 MDS2(config-if)# sw owner MDS1 port 1 MDS2(config-if)# int fc1/2 MDS2(config-if)# sw owner MDS1 port 2 MDS2(config-if)# int fc1/3 MDS2(config-if)# sw owner MDS1 port 3 MDS2(config-if)# int fc1/4 MDS2(config-if)# sw owner MDS1 port 4

One of the ports on MDS1 should be easily located. This is done using a blinking switchport LED known as the ‘beacon’. This is a single command which enables you to physically find interfaces in high port density switches.

MDS1

MDS1(config)# int fc1/5 MDS1(config-if)# sw beacon

The next feature is a feature used for long distance links. Those links are typically using lot’s of interconnects and it could occur that errors are risen during the transport of a frame across this fiber. The MDS switches use certain thresholds for these errors and after a threshold is reached the interface is put in err-disabled mode because of this.



The thresholds are reached when 15 error bursts occur in a 5 minute period. The errors might occur because of one of the following reasons:

• Bad fiber

• Bad SFP

• Interface is specified to operate at 1 Gbps, but is used at 2 Gbps

• Interface is specified to operate at 2 Gbps, but is used at 4 Gbps

• Short haul cable is used for long haul or long haul cable is used for short haul

• Momentary sync loss

• Loose cable connection at one or both ends

• Improper SFP connection at one or both ends

• We can disable this if we know the link is fine and we want to overrule the thresholds. As there is no MDS specified in this task we will configure both.

MDS1

MDS1(config)# int fc1/10 MDS1(config-if)# sw ignore bit-errors

MDS2

MDS2(config)# int fc1/10 MDS2(config-if)# sw ignore bit-errors

Next we need to ensure that all interfaces are shutdown without any configuration. This is where we are configuring defaults.

MDS1

MDS1(config)# system default ? switchport Configure default values for switchport attributes zone Configure default values for zone MDS1(config)# system default switchport ? mode Enter the port mode shutdown Disable/enable switchports by default trunk Configure trunking parameters as a default MDS1(config)# system default switchport shutdown MDS1(config)#

MDS2

MDS2(config)# system default switchport shutdown MDS2(config)#



The next task is regarding giving specific devices within the Fibre Channel an easily remembered name or alias. This alias should be known throughout the fabric. We can do this using 2 ways. One of them is by using a standard based solution called ‘FCalias’. This is a method of giving names to certain characteristics of a connected host (WWPN, Interface, FCID, etc.). The only problem is that FCalias is dependent on the VSAN. You cannot create a VSAN overlapping FCalias. Besides that it’s not possible to distribute the FCalias information throughout the Fibre Channel fabric.

The solution we are looking for here is a Cisco proprietary solution called device-alias. These device-aliases can be distributed throughout the FC Fabric by using the Cisco Fabric Services (CFS). You can create a device-alias using a PWWN of the device. By default when you the use this device-alias in any kind of configuration, for example Zoning, it will automatically convert back to the PWWN of the device.

What you really want is that the device-alias remains the connection between the zone and the host, so that when changes are done to the device-alias these are leveraged back into the zoning database automatically. This is done using so-called Enhanced device-alias. Then the device-alias name is kept in the zoning configuration, just like our task asks for.

Do not forget to enable the distribution of device-aliases, because this is not enabled by default. You can use the FLOGI database for filling the device-alias database.

MDS1

MDS1(config)# device-alias distribute MDS1(config)# device-alias mode enhanced MDS1(config)# device-alias database MDS1(config-device-alias-db)# device-alias name J1D1 pwwn c6:83:00:11:0d:18:fa:ce MDS1(config-device-alias-db)# device-alias name J1D2 pwwn c6:83:00:11:0d:18:fb:ce MDS1(config-device-alias-db)# device-alias name J1D3 pwwn c6:83:00:11:0d:18:fc:ce MDS1(config-device-alias-db)# device-alias name J1D4 pwwn c6:83:00:11:0d:18:fd:ce MDS1(config-device-alias-db)# device-alias name J2D1 pwwn 21:00:00:11:0d:18:fe:ce MDS1(config-device-alias-db)# device-alias name J2D2 pwwn 21:00:00:11:0d:18:ff:ce MDS1(config-device-alias-db)# device-alias name J2D3 pwwn 21:00:00:11:0d:18:a0:ce MDS1(config-device-alias-db)# device-alias name J2D4 pwwn 21:00:00:11:0d:18:a1:ce MDS1(config-device-alias-db)# device-alias commit

Now we need to verify if we see all the aliases on the other MDS as well, without configuring it. Be aware that your ISLs are up before this works!



MDS1

MDS1(config)# do sh device-alias database device-alias name J1D1 pwwn c6:83:00:11:0d:18:fa:ce device-alias name J1D2 pwwn c6:83:00:11:0d:18:fb:ce device-alias name J1D3 pwwn c6:83:00:11:0d:18:fc:ce device-alias name J1D4 pwwn c6:83:00:11:0d:18:fd:ce device-alias name J2D1 pwwn 21:00:00:11:0d:18:fe:ce device-alias name J2D2 pwwn 21:00:00:11:0d:18:ff:ce device-alias name J2D3 pwwn 21:00:00:11:0d:18:a0:ce device-alias name J2D4 pwwn 21:00:00:11:0d:18:a1:ce Total number of entries = 8 MDS1(config)#

MDS2

MDS2(config)# do sh device-alias status Fabric Distribution: Enabled Database:- Device Aliases 8 Mode: Enhanced Checksum: 0x617152b263ae5b4ebb547a6fb633b7fb MDS2(config)# MDS2(config)# do sh device-alias database device-alias name J1D1 pwwn c6:83:00:11:0d:18:fa:ce device-alias name J1D2 pwwn c6:83:00:11:0d:18:fb:ce device-alias name J1D3 pwwn c6:83:00:11:0d:18:fc:ce device-alias name J1D4 pwwn c6:83:00:11:0d:18:fd:ce device-alias name J2D1 pwwn 21:00:00:11:0d:18:fe:ce device-alias name J2D2 pwwn 21:00:00:11:0d:18:ff:ce device-alias name J2D3 pwwn 21:00:00:11:0d:18:a0:ce device-alias name J2D4 pwwn 21:00:00:11:0d:18:a1:ce Total number of entries = 8 MDS2(config)#

B2B credit buffers are assigned to an interface. The buffers are used for the fact of the no-drop policy that is used in FC fabrics. When a packet is sent, it costs 1 credit on the sending side. This credit is returned to the pool when an acknowledgement is received from the other side. When the switch runs out of buffer credits, it will not send any new packets on the interface until it receives acknowledgements.

The maximum number of credits is very much dependent on the switch type and the line card that is used for the interface. There are many configuration options for sharing credits in an ASIC between different ports.

The recovery that is asked for has to do with the fact that sometimes the acknowledgements might get lost on longer interfaces due to bit errors. This means that in time, the amount of buffer credits get’s less on the interface, while this is not right, as the receiving switch has space enough. This is called the recovery of credits to get them back periodically. This process is disabled by default.



Be aware that the credits that you configure are transmitted to the other end of the interface. The buffer you configure is the receive buffer, this receive buffer is then communicated so the other side knows how much buffer space it can fill up by sending frames.

We can verify this by checking the hardware options of the interface and in which port-group it’s located:

MDS1# show port-resources module 1 Module 1 Available dedicated buffers are 3765 Port-Group 1 Total bandwidth is 12.8 Gbps Total shared bandwidth is 8.8 Gbps Allocated dedicated bandwidth is 4.0 Gbps -------------------------------------------------------------------- Interfaces in the Port-Group B2B Credit Bandwidth Rate Mode Buffers (Gbps) -------------------------------------------------------------------- fc1/1 16 4.0 shared fc1/2 16 4.0 shared fc1/3 16 4.0 shared fc1/4 16 4.0 shared fc1/5 250 4.0 dedicated fc1/6 16 4.0 shared Port-Group 2 Total bandwidth is 12.8 Gbps Total shared bandwidth is 12.8 Gbps Allocated dedicated bandwidth is 0.0 Gbps MDS1#

Interface fc1/1 is located in port-group one. In this case, the amount of buffer credits is limited to the module. Some other modules have limitations per port-group.

In this case we have 3765 credits to spare on an interface. The problem is that we can only use up to 250 credits for E-ports. We can however use Extended B2B credit buffers for this use. When the regular credits run out, we are able to use the extended buffers up to the maximum. Finally we disable the credit recovery.

You require the Enterprise license to activate the Extended B2B credit buffers. Remind that you need to enable the feature separately.

MDS1

MDS1(config)# feature fcrxbbcredit extended MDS1(config)# int fc1/1 MDS1(config-if)# switch fcrx fcrxbbcredit fcrxbufsize MDS1(config-if)# switch fcrxbbcredit ? <1-500> Enter receive BB_credit default Default receive BB_credit



extended Configure extended BB_credit for the port performance-buffers Configure performance buffers for receive BB_credit MDS1(config-if)# switch fcrxbbcredit extended ? <256-4095> Enter extended credit receive BB_credit MDS1(config-if)# switch fcrxbbcredit extended 4095 fc1/5: (error) Receive bbcredit unavailable

We can’t configure that much buffers as we just don’t have them available. We configure the maximum that remains after subtracting all other interface credits.

MDS1(config-if)# switch fcrxbbcredit extended 4012 MDS1(config-if)# switchport fcbbscn

As the final step we enable credit recovery, by enabling the BB_SC_N field towards the other side.

Now we verify that we are indeed using that much credits on the interface. As we can see the other end communicated 250 credits to our interface.

MDS1(config-if)# do sh run int fc1/1 !Command: show running-config interface fc1/1 !Time: Wed Oct 17 16:43:04 2012 version 5.2(6) interface fc1/1 switchport rate-mode dedicated switchport fcrxbbcredit extended 4012 switchport mode E no shutdown MDS1(config-if)# sh int fc1/1 bbcredit fc1/1 is trunking Transmit B2B Credit is 32 Receive B2B Credit is 4012 4012 receive B2B credit remaining 32 transmit B2B credit remaining 32 low priority transmit B2B credit remaining MDS1(config-if)# do sh port-resources module 1 Module 1 Available dedicated buffers are 0 Port-Group 1 Total bandwidth is 12.8 Gbps Total shared bandwidth is 4.8 Gbps Allocated dedicated bandwidth is 8.0 Gbps -------------------------------------------------------------------- Interfaces in the Port-Group B2B Credit Bandwidth Rate Mode Buffers (Gbps) -------------------------------------------------------------------- fc1/1 4012 4.0 dedicated fc1/2 16 4.0 shared fc1/3 16 4.0 shared



fc1/4 16 4.0 shared fc1/5 16 4.0 dedicated fc1/6 16 4.0 shared Port-Group 2 Total bandwidth is 12.8 Gbps Total shared bandwidth is 12.8 Gbps Allocated dedicated bandwidth is 0.0 Gbps MDS1(config-if)#

The Maximum Transmission size is also a configuration item that is communicated to the other end. Meaning that you do not configure the Maximum Transmission Size on the interface, but you configure the Maximum Receive Size of the packet, because you (as in the switch) know how much buffer space you have to receive Fibre Channel frames. The ‘MTU’ size is by default configured to the maximum supported in FC environments which is 2112 bytes.

MDS1

MDS1(config-if)# int fc1/5 MDS1(config-if)# switch fcrxbufsize ? <256-2112> Enter receive buffer size MDS1(config-if)# switch fcrxbufsize 2000

As a final task we need to configure State Change Numbers on all JBOD interfaces. This field is carried in the same BB_SC_N field as the credit recovery is handled. So we need to enable this on all the JBOD connected interfaces.

MDS1

MDS1(config-if)# int fc1/5 MDS1(config-if)# switch fcbbscn MDS1(config-if)# int fc1/6 MDS1(config-if)# switch fcbbscn

MDS2

MDS2(config-if)# int fc1/5 MDS2(config-if)# switch fcbbscn MDS2(config-if)# int fc1/6 MDS2(config-if)# switch fcbbscn

We completed our first Fibre Channel task!



Task 2: VSANs The next task is regarding the virtualization feature that is supported on Cisco’s MDS switches. This feature enables the use of different Fibre Channel SANs on the same physical hardware.

This is done using a tagging mechanism which is very similar to 802.1Q tagging in the Ethernet world. The technical similarity to a VLAN + a VRF in the Ethernet world. As there is a full Layer 2 separation between the different Fibre Channel Fabrics. In Ethernet terms we would also say that there is a Layer 3 separation as the routing protocol (FSPF) is also separated.

During this task we will be using the VSANs and the routing protocol FSPF to steer traffic across different ways through our fabric. Another topic in this task is port-channeling of ISL links.

All settings on the MDS are VSAN dependent. Therefore there is a total separation between control-planes of the VSANs. This introduces a way that you need to specify the VSAN in each of the show and configuration commands that you type in.

This also means that Zoning in one VSAN has absolutely no impact on Zoning in other VSANs.

The drawing below illustrates the use of VSANs, by using the trunking between MDS switches, you can connect devices in any VSAN on any MDS switch and transport the FC frames between switches by using the EISL tagging (Enhanced ISL) on interfaces.



Throughout this task explanation, topics will be explained during the tasks that are discussed.

The first tasks are regarding the creation of VSANs and we should place some interfaces into the VSANS. Just like VLANs, interfaces are placed inside a VSAN.

MDS1

MDS1(config)# vsan database MDS1(config-vsan)# vsan 10 name IPX_VSAN_10 MDS1(config-vsan)# vsan 10 interface fc1/5 MDS1(config-vsan)# vsan 20 name IPX_VSAN_20 MDS1(config-vsan)# vsan 20 interface fc1/6 MDS1(config-vsan)# vsan 30 name IPX_VSAN_30 MDS1(config-vsan)# vsan 40 name IPX_VSAN_40

MDS2

MDS2(config)# vsan database MDS2(config-vsan)# vsan 10 name IPX_VSAN_10 MDS2(config-vsan)# vsan 10 interface fc1/6 MDS2(config-vsan)# vsan 20 name IPX_VSAN_20 MDS2(config-vsan)# vsan 20 interface fc1/5 MDS2(config-vsan)# vsan 30 name IPX_VSAN_30 MDS2(config-vsan)# vsan 40 name IPX_VSAN_40

By default interfaces are placed into a VSAN. When you want to support the moving of certain hosts throughout the fabric, you can use a technology called Dynamic Port VSAN Membership (DPVM). This technology will place a port into a VSAN when a certain PWWN is seen.

The DPVM technology can also be distributed throughout the fabric, which is enabled by default, so that it’s independent of the switch where it comes online and it’s automatically placed into the correct VSAN.

You can also enable the auto-learn feature which enables the automatic learning of all devices coming online on interfaces. These are then placed into the DPVM database and the next time they come online on another interface, the interface is placed into that new VSAN.

The DPVM configuration overrules the interface VSAN configuration, so the DPVM database is leading in this case.

Keep in mind that the only port where this works on is F-ports.

We start with the 2 tasks that place either a WWPN or a Device-Alias in the database for DPVM.

MDS2

MDS2(config)# feature dpvm

Enable DPVM first on the second switch. If you commit the changes on one switch and enable it later on another switch, you need to commit again. So pay attention to when you configure things that are distributed. This feature and the distribution of it, needs to be enabled on all switches first, before information is actually synchronized.



MDS1

MDS1(config)# feature dpvm MDS1(config)# dpvm distribute MDS1(config)# dpvm database MDS1(config-dpvm-db)# pwwn 20:11:00:0a:31:00:aa:de vsan 30 MDS1(config-dpvm-db)# device-alias J1D1 vsan 40 MDS1(config-dpvm-db)# dpvm activate MDS1(config)# dpvm commit

Now we can see that all information is synchronized between the switches. By using the ‘status’ command you can always check the configuration of the CFS services running for this particular feature. Very useful when troubleshooting problems with the CFS distribution.

MDS2

MDS2(config)# do sh dpvm status No active DB, auto-learn is off, distribution is enabled, Duplicated pwwn will be Rejected. MDS2(config)#

After configuring MDS1, we see that everything is synchronized between the switches

MDS2(config)# do sh dpvm database pwwn 20:11:00:0a:31:00:aa:de vsan 30 device-alias J1D1 [20:11:00:0a:31:00:aa:df] vsan 40 [Total 2 entries] MDS2(config)# do sh dpvm status DB is activated successfully, auto-learn is off, distribution is enabled, Duplicated pwwn will be Rejected. MDS2(config)#

The next questions are related to the load balancing that is used within a VSAN to distribute information across the fabric. This load balancing mechanism is used for both Equal Cost Multi Pathing (ECMP) when 2 equal costs exist in the FSPF database or when a Port-channel is used with multiple links bundled as one logical interface.

The load-balancing is configured per VSAN. It can be configured in 2 ways.

Source and Destination FC-ID

This method ensures that flows between a source and destination pair is always using the same link

Source and Destination FC-ID and the Originator Exchange ID OX-ID

The OX-ID is a value in the Fibre Channel header that is used to identify a traffic flow. Every ‘exchange’, or every ‘action’, done between 2 devices gets a new value attached. All these values are separated across different hashes so this mechanism achieves a much more granular load balancing.



In this task VSAN 10 should use the basic Source and Destination FCID where as VSAN 20 should use exchange based load balancing. This is configured using 2 relatively easy commands on both switches:

MDS1

MDS1(config)# vsan database MDS1(config-vsan)# vsan 10 loadbalancing src-dst-id MDS1(config-vsan)# vsan 20 loadbalancing src-dst-ox-id

MDS2

MDS2(config)# vsan database MDS2(config-vsan)# vsan 10 loadbalancing src-dst-id MDS2(config-vsan)# vsan 20 loadbalancing src-dst-ox-id

All ISLs should then be made possible to transport all the different VSANs that we created. This means we now need to enable the trunking on the ISLs to enable this.

MDS1

MDS1(config-if)# interface fc1/1 MDS1(config-if)# switchport trunk mode on MDS1(config-if)# interface fc1/2 MDS1(config-if)# switchport trunk mode on MDS1(config-if)# interface fc1/3 MDS1(config-if)# switchport trunk mode on MDS1(config-if)# interface fc1/4 MDS1(config-if)# switchport trunk mode on

MDS2

MDS2(config-if)# interface fc1/1 MDS2(config-if)# switchport trunk mode on MDS2(config-if)# interface fc1/2 MDS2(config-if)# switchport trunk mode on MDS2(config-if)# interface fc1/3 MDS2(config-if)# switchport trunk mode on MDS2(config-if)# interface fc1/4 MDS2(config-if)# switchport trunk mode on

To verify the working of the trunks, we can issue a similar command to what we use in the Ethernet world.

MDS1(config)# do sh int trunk vsan fc1/1 is trunking Vsan 1 is up (None) Vsan 10 is up (None) Vsan 20 is up (None) Vsan 30 is up (None) Vsan 40 is up (None) fc1/2 is trunking Vsan 1 is up (None) Vsan 10 is up (None)



Vsan 20 is up (None) Vsan 30 is up (None) Vsan 40 is up (None) fc1/3 is trunking Vsan 1 is up (None) Vsan 10 is up (None) Vsan 20 is up (None) Vsan 30 is up (None) Vsan 40 is up (None) fc1/4 is trunking Vsan 1 is up (None) Vsan 10 is up (None) Vsan 20 is up (None) Vsan 30 is up (None) Vsan 40 is up (None) MDS1(config)#

The next questions are regarding the bundling of interfaces in single logical bundles. This is where the load balancing is used between the different members of port-channels.

Port-channeling on the MDS switches is also a unique feature of the Cisco MDS platform. These port-channels have equal options as the standard FC interfaces.

The port-channels also support a protocol to negotiate there possibilities, just like LACP in the 802.3ad protocol for Ether Channels. Almost the same rules are used for including interfaces in a bundle. The speed, ASIC rate-mode, BB credits, Trunking or Non-Trunking mode and allowed VSANs need to be equal.

It’s possible to channel both E (TE) and F (TF) ports, when the platforms support it. One place where this is commonly seen is in environments that uses N-Port Virtualization (NPV).

We start by configuring the first port-channel between the 2 MDS switches. This is done in a relatively equal way to Ethernet port-channels on the Nexus switches.

MDS1

MDS1(config-if)# interface port-channel 101 MDS1(config-if)# switchport mode e MDS1(config-if)# switchport trunk mode on MDS1(config-if)# no shutdown MDS1(config-if)# interface fc1/1 MDS1(config-if)# channel-group 101 fc1/1 added to port-channel 101 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring it up MDS1(config-if)# int fc1/1 MDS1(config-if)# no shut MDS1(config-if)# interface fc1/3 MDS1(config-if)# channel-group 101 fc1/3 added to port-channel 101 and disabled please do the same operation on the switch at the other end of the port-channel,



then do "no shutdown" at both ends to bring it up MDS1(config-if)# int fc1/3 MDS1(config-if)# no shut

MDS2

MDS2(config-if)# interface port-channel 101 MDS2(config-if)# switchport mode e MDS2(config-if)# switchport trunk mode on MDS2(config-if)# no shutdown MDS2(config-if)# interface fc1/1 MDS2(config-if)# channel-group 101 fc1/1 added to port-channel 101 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring it up MDS2(config-if)# int fc1/1 MDS2(config-if)# no shut MDS2(config-if)# interface fc1/3 MDS2(config-if)# channel-group 101 fc1/3 added to port-channel 101 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring it up MDS2(config-if)# int fc1/3 MDS2(config-if)# no shut

Now we brought up our first port-channel. Keep in mind that you might need to shut and no-shut certain interfaces as the ports might get suspended when you are configuring the other switch for the port-channeling.

MDS1(config-if)# do sh port-channel summ ------------------------------------------------------------------------------ Interface Total Ports Oper Ports First Oper Port ------------------------------------------------------------------------------ port-channel 101 2 2 fc1/1 MDS1(config-if)# MDS1(config-if)# do sh int port-channel 101 port-channel 101 is trunking (Not all VSANs UP on the trunk) Hardware is Fibre Channel Port WWN is 24:65:00:05:9b:70:5a:40 Admin port mode is E, trunk mode is on snmp link state traps are enabled Port mode is TE Port vsan is 1 Speed is 8 Gbps Trunk vsans (admin allowed and active) (1,10,20,30,40) Trunk vsans (up) () Trunk vsans (isolated) () Trunk vsans (initializing) (1,10,20,30,40) 5 minutes input rate 600 bits/sec, 75 bytes/sec, 0 frames/sec 5 minutes output rate 576 bits/sec, 72 bytes/sec, 0 frames/sec



134 frames input, 10392 bytes 0 discards, 0 errors 0 CRC, 0 unknown class 0 too long, 0 too short 134 frames output, 11288 bytes 0 discards, 0 errors 4 input OLS, 4 LRR, 10 NOS, 0 loop inits 6 output OLS, 4 LRR, 6 NOS, 0 loop inits Member[1] : fc1/1 Member[2] : fc1/3 Interface last changed at Wed Oct 17 22:07:32 2012 MDS1(config-if)#

Now we also need to make them auto negotiate their bundling capabilities. These capabilities activate a protocol similar to LACP. Instead of configuring it under the member interfaces, this mode is changed under the port-channel configuration.

MDS2

MDS2(config-if)# int port-channel 101 MDS2(config-if)# channel ? mode Set the channel mode for the port-channel interface MDS2(config-if)# channel mode ? active Configure ACTIVE port-channel MDS2(config-if)# channel mode active MDS2(config-if)#

MDS2

MDS1(config-if)# do sh port-channel database port-channel 101 Administrative channel mode is on Operational channel mode is on Last membership update succeeded First operational port is fc1/1 2 ports in total, 2 ports up Ports: fc1/1 [up] fc1/3 [up] * MDS1(config-if)# channel mode active MDS1(config-if)# do sh port-channel database port-channel 101 Administrative channel mode is active Operational channel mode is active Last membership update succeeded First operational port is fc1/1 2 ports in total, 2 ports up Ports: fc1/1 [up] fc1/3 [up] * MDS1(config-if)#



The next task is asking about another channel. In the past the MDS switches knew a mechanism which allowed the auto creation of port-channels. You could enable a single command on a range of interfaces and the switch then automatically detects which ports are eligible for bundling towards the same other MDS switch on the other end.

This creates a very easy way to configure link bundles. You could also configure the interface after it’s been created with changes to the whole bundle. The number that was used for the channel depends on the maximum number of supported port-channels on that particular platform, as it uses the last available number.

However, as of NX-OS 4.1(1b), this feature was removed as it also introduced issues in that you didn’t know where the port-channels were pointing to and you really had to know what was going on the switches.

We configure the remaining 2 ISLs in another port-channel.

MDS1

MDS1(config-if)# interface port-channel 127 MDS1(config-if)# switchport mode e MDS1(config-if)# switchport trunk mode on MDS1(config-if)# no shutdown MDS1(config-if)# interface fc1/2 MDS1(config-if)# channel-group 127 fc1/2 added to port-channel 127 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring it up MDS1(config-if)# int fc1/2 MDS1(config-if)# no shut MDS1(config-if)# interface fc1/4 MDS1(config-if)# channel-group 127 fc1/4 added to port-channel 101 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring it up MDS1(config-if)# int fc1/4 MDS1(config-if)# no shut

MDS2

MDS2(config-if)# interface port-channel 127 MDS2(config-if)# switchport mode e MDS2(config-if)# switchport trunk mode on MDS2(config-if)# no shutdown MDS2(config-if)# interface fc1/2 MDS2(config-if)# channel-group 101 fc1/2 added to port-channel 101 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring it up MDS2(config-if)# int fc1/2



MDS2(config-if)# no shut MDS2(config-if)# interface fc1/4 MDS2(config-if)# channel-group 127 fc1/4 added to port-channel 127 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring it up MDS2(config-if)# int fc1/4 MDS2(config-if)# no shut

Now what remains is that we need to change our trunks to be using different links. The first 2 questions are related to the allowed VSAN command, which enables which VSANs are allowed to transfer across the links.

MDS1

MDS1(config-if)# interface port-channel 101 MDS1(config-if)# sw trunk allowed vsan 10 MDS1(config-if)# sw trunk allowed vsan add 20 MDS1(config-if)# sw trunk allowed vsan add 40 MDS1(config-if)# interface port-channel 127 MDS1(config-if)# sw trunk allowed vsan 10 MDS1(config-if)# sw trunk allowed vsan add 20 MDS1(config-if)# sw trunk allowed vsan add 30

MDS2

MDS2(config-if)# interface port-channel 101 MDS2(config-if)# sw trunk allowed vsan 10 MDS2(config-if)# sw trunk allowed vsan add 20 MDS2(config-if)# sw trunk allowed vsan add 40 MDS2(config-if)# interface port-channel 127 MDS2(config-if)# sw trunk allowed vsan 10 MDS2(config-if)# sw trunk allowed vsan add 20 MDS2(config-if)# sw trunk allowed vsan add 30

Now to verify we only see the VSANs that we configured under the interfaces to be trunked to the other MDS.

MDS2(config-if)# do sh int trunk vsan port-channel101 is trunking Vsan 10 is up (None) Vsan 20 is down (Initializing) Vsan 40 is up (None) port-channel127 is trunking Vsan 10 is up (None) Vsan 20 is down (Initializing) Vsan 30 is down (Initializing) MDS2(config-if)#



The other 2 questions are related to the steering of traffic across the fibre channel fabric. The focus in this question is about the FSPF protocol that is working on all FC switches. This protocol is very closely connected to the OSPF protocol and works basically the same. The easy part is that it works fully automatically. There is no configuration necessary and will immediately start creating neighbors and exchange information. The only information that is exchanged in the FSPF protocol is the Domain ID of the switch in the specific VSAN. The Fibre Channel routing uses those FC Domain IDs to steer traffic to the correct switch. After it reaches the destination switch it will perform a look-up to find the egress interface of the traffic where it can send its traffic to. This is all bound to a VSAN basis. Therefore a VSAN can also be called a Layer 3 VRF in the IP world, as the routing protocol is also fully separated.

We can steer the traffic by changing the FSPF link cost between the switches. FSPF link cost is dependent on the bandwidth on the link. Therefore also for port-channels the amount of bandwidth of all member interfaces is counted up to a high bandwidth number.

The following list shows the default FSPF link cost:

• 1Gbps: 1000

• 2Gbps: 500

• 4Gbps: 250

• 8Gbps: 125

• 16Gbps (port-channel of 2 x 8gbps links): 62

As we are working with 8Gbps links we have a default cost of 125. Therefore we will lower the cost on the links we want to make active for a certain VSAN.

We need to make VSAN10 to use port-channel101 and VSAN20 to use port-channel 102 as their primary links.

MDS1

MDS1(config-if)# interface port-channel 101 MDS1(config-if)# fspf cost 99 vsan 10 MDS1(config-if)# interface port-channel 127 MDS1(config-if)# fspf cost 99 vsan 20

MDS2

MDS2(config-if)# interface port-channel 101 MDS2(config-if)# fspf cost 99 vsan 10 MDS2(config-if)# interface port-channel 127 MDS2(config-if)# fspf cost 99 vsan 20



To verify we need to use a little strange command to find out which interface is actually used for the traffic. By default the command doesn’t show which interface is used. We do see the correct FSPF cost already being in place.

MDS2# sh fcroute unicast D:direct R:remote P:permanent V:volatile A:active N:non-active # Next Protocol VSAN FC ID/Mask RCtl/Mask Flags Hops Cost -------- ---- -------- -------- ---- ---- ----- ---- ---- local 1 0x290000 0xffffff 0x00 0x00 D P A 1 1 fspf 10 0x880000 0xff0000 0x00 0x00 D P A 1 99 fspf 20 0x3a0000 0xff0000 0x00 0x00 D P A 1 99 fspf 30 0x260000 0xff0000 0x00 0x00 D P A 1 250 fspf 40 0x4f0000 0xff0000 0x00 0x00 D P A 1 250 MDS2# MDS2#

When we look a little deeper, by specifying the destination FC Domain ID, we can see the actual interface that is used for the next-hop. There we see that we are having a properly configured FSPF network!

MDS2# sh fcroute unicast host 0x880000 0xff0000 vsan 10 D:direct R:remote P:permanent V:volatile A:active N:non-active # Next Protocol VSAN FC ID/Mask RCtl/Mask Flags Hops Cost -------- ---- -------- -------- ---- ---- ----- ---- ---- fspf 10 0x880000 0xff0000 0x00 0x00 D P A 1 99 port-channel101 Domain 0x88(136) MDS2# MDS2# sh fcroute unicast host 0x3a0000 0xff0000 vsan 20 D:direct R:remote P:permanent V:volatile A:active N:non-active # Next Protocol VSAN FC ID/Mask RCtl/Mask Flags Hops Cost -------- ---- -------- -------- ---- ---- ----- ---- ---- fspf 20 0x3a0000 0xff0000 0x00 0x00 D P A 1 99 port-channel127 Domain 0x3a(58) MDS2#

The next question is regarding the possibility of the FC Fabric to deliver packets in-order between hosts. Some hosts that are connected to the fabric require this, for example HP/UX hosts or FICON hosts assume that the fabric will deliver packets in-order and will have various issues when traffic is not delivered in the same order as it was sent.

Therefore you can enable the in-order delivery of packets per VSAN.

MDS1

MDS1(config)# in-order-guarantee vsan 30



MDS2

MDS2(config)# in-order-guarantee vsan 30

It’s even possible to create static routes in the FC fabric network by using the ‘fcroute’ command. We will need this kind of traffic engineering for the next question to send traffic between two specific disks across another link than the default traffic is going.

In this question you should be aware that it’s talking about 2 ways of traffic, so the steering needs to be set back to disk as well (bidirectional traffic flow).

Pay close attention to the FC Domain ID of the switch and the port. These can change when the host is re-connected or the host is rebooted and the Domain ID is not statically configured. In this question it’s not that relevant, but in the lab, keep this in mind. You will want to have static FCID’s configured when you need to create a configuration like this.

MDS2

MDS2(config)# fcroute 0x8800e8 0xffffff interface port-channel127 domain 136 ? aggregate Aggregable route metric Cost of route remote Destination switch remotely connected vsan VSAN id

When you specify a destination Domain ID of anything behind the next-hop switch, you should specify the remote keyword here. As otherwise the switch will always assume that the domain ID specified is also the next-hop switch. Look up this domain ID using the ‘show fcdomain vsan’ command.

MDS1(config-vsan-db)# do sh fcdomain vsan 10 The local switch is the Principal Switch. Local switch run time information: State: Stable Local switch WWN: 20:0a:00:05:9b:70:5a:41 Running fabric name: 20:0a:00:05:9b:70:5a:41 Running priority: 2 Current domain ID: 0x88(136) Local switch configuration information: State: Enabled FCID persistence: Enabled Auto-reconfiguration: Disabled Contiguous-allocation: Disabled Configured fabric name: 20:01:00:05:30:00:28:df Optimize Mode: Disabled Configured priority: 128 Configured domain ID: 0x00(0) (preferred) Principal switch run time information: Running priority: 2



MDS1(config-vsan-db)#

MDS2

MDS2(config)# fcroute 0x8800e8 0xffffff interface port-channel127 domain 136 vsan 10

MDS1

MDS1(config)# fcroute 0x7f0000 0xffffff interface port-channel127 domain 127 vsan 10

The final verification is again by using the fcroute show command, so we can check specific destinations, not only for the Domain ID mask, but also for the host mask.

MDS2(config)# show fcroute unicast host 0x8800e8 0xffffff vsan 10 D:direct R:remote P:permanent V:volatile A:active N:non-active # Next Protocol VSAN FC ID/Mask RCtl/Mask Flags Hops Cost -------- ---- -------- -------- ---- ---- ----- ---- ---- static 10 0x8800e8 0xffffff 0x00 0x00 D P A 1 10 port-channel127 Domain 0x88(136) fspf 10 0x880000 0xff0000 0x00 0x00 D P A 1 99 port-channel101 Domain 0x88(136) MDS2(config)#

In this show output we see that all traffic is sent using the FSPF route across port-channel 101 and that traffic specific for the disk in the JBOD is sent across the other port-channel. Just what the question stated!

In the Fibre Channel network there is also a root switch for all the Multicast and Broadcast traffic that is sent throughout the fabric. By default the root switch is also the Principal switch of the fabric. In case of VSAN 20 we need to change this to be the lowest domain ID switch

This is done using a single command, again VSAN specific.

MDS1

MDS1(config)# mcast root lowest vsan 20

MDS2

MDS2(config)# mcast root lowest vsan 20

To verify we see that by default the Principal switch is chosen as the default multicast root. Now for VSAN 20 we changed this to be the lowest Domain ID switch.

MDS2(config)# show mcast vsan 10 Multicast root for VSAN 10 Configured root mode : Principal switch Operational root mode : Principal switch Root Domain ID : 0x88(136)



MDS2(config)# show mcast vsan 20 Multicast root for VSAN 20 Configured root mode : Lowest domain switch Operational root mode : Lowest domain switch Root Domain ID : 0x3a(58) MDS2(config)#

The next task is regarding so-called iSPF calculations. With these incremental changes the SPF calculations of the FSPF protocol are done only on the changes.

MDS2

MDS2(config)# fspf config vsan 30 MDS2(config-(fspf-config))# ? min-ls-arrival Configure Min LS Arrival min-ls-interval Configure Min LS interval no Negate a command or set its defaults region Configure the autonomous region for FSPF spf Configure parameters related to SPF route computation end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in MDS2(config-(fspf-config))# spf ? hold-time Configure the hold time between route computations static Force static spf computation MDS2(config-(fspf-config))# spf static

MDS1

MDS1(config)# fspf config vsan 30 MDS1(config-(fspf-config))# spf static

The final question in this task is that we want unused ports not to use the Default VSAN. The Default VSAN is always VSAN 1. Now unused ports are by default configured into VSAN 1.

This cannot be changes easily, but we can ensure that there is no traffic being sent within VSAN 1. This is done by making it ‘suspended’, but pay close attention! This makes the process difficult as all interfaces start from VSAN 1 including all ISLs.

To overcome this limitation is to configure the ports in another VSAN just to get them coming online. Keep this also in mind in later chapters where you are configuring additional links in the FC fabric, like FCIP tunnels, etc.

MDS1

MDS1(config)# vsan database MDS1(config-vsan-db)# vsan 1 suspend



MDS2

MDS2(config)# vsan database MDS2(config-vsan-db)# vsan 1 suspend

Now initially everything looks fine and the trunks are still up and running.

MDS2(config-vsan-db)# do sh int trunk vsan fc1/5 is trunking Vsan 10 is up (None) Vsan 20 is up (None) Vsan 40 is up (None) fc1/6 is trunking Vsan 10 is up (None) Vsan 20 is up (None) Vsan 30 is up (None) MDS2(config-vsan-db)#

However when the links are re-enabled, they will not come back online and you will get into a very tricky and stressful situation when you are far down the lab!

MDS2(config-if)# do sh int fc1/5 fc1/5 is down (Inactive) Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:05:54:7f:ee:01:2b:80 Admin port mode is E, trunk mode is on snmp link state traps are enabled Port vsan is 1 Receive data field Size is 2112 Beacon is turned off 5 minutes input rate 200 bits/sec, 25 bytes/sec, 0 frames/sec 5 minutes output rate 384 bits/sec, 48 bytes/sec, 0 frames/sec 5603 frames input, 424228 bytes 0 discards, 0 errors 0 CRC, 0 unknown class 0 too long, 0 too short 4888 frames output, 308536 bytes 0 discards, 0 errors 3 input OLS, 4 LRR, 0 NOS, 0 loop inits 7 output OLS, 7 LRR, 3 NOS, 0 loop inits Interface last changed at Wed Oct 17 13:58:22 2012

This problem is overcome by giving the ISL links another VSAN in the VSAN database so they are made online within another VSAN, before the trunking protocol is initialized.

MDS1

MDS1(config)# vsan database MDS1(config-vsan-db)# vsan 10 interface port-channel101 MDS1(config-vsan-db)# vsan 10 interface port-channel127 MDS1(config-vsan-db)# interface port-channel101 MDS1(config-if)# shut MDS1(config-if)# no shutdown MDS1(config-if)# interface port-channel127 MDS1(config-if)# shut



MDS1(config-if)# no shutdown

MDS2

MDS2(config)# vsan database MDS2(config-vsan-db)# vsan 10 interface port-channel101 MDS2(config-vsan-db)# vsan 10 interface port-channel127 MDS2(config-vsan-db)# interface port-channel101 MDS2(config-if)# shut MDS2(config-if)# no shutdown MDS2(config-if)# interface port-channel127 MDS2(config-if)# shut MDS2(config-if)# no shutdown

After the re-configuration of the ISL links, the port-channel interfaces are coming online just as they were with the correct trunking protocol and we are up and running again!

MDS1(config-vsan-db)# do sh int fc1/5 fc1/5 is trunking (Not all VSANs UP on the trunk) Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:05:00:05:9b:70:5a:40 Peer port WWN is 20:05:54:7f:ee:01:2b:80 Admin port mode is E, trunk mode is on snmp link state traps are enabled Port mode is TE Port vsan is 10 Speed is 4 Gbps Rate mode is dedicated Transmit B2B Credit is 32 Receive B2B Credit is 250 B2B State Change Number is 14 Receive data field Size is 2112 Beacon is turned off Trunk vsans (admin allowed and active) (10,20,30,40) Trunk vsans (up) (10,20) Trunk vsans (isolated) (30) Trunk vsans (initializing) (40) 5 minutes input rate 216 bits/sec, 27 bytes/sec, 0 frames/sec 5 minutes output rate 184 bits/sec, 23 bytes/sec, 0 frames/sec 4994 frames input, 354756 bytes 0 discards, 0 errors 0 CRC, 0 unknown class 0 too long, 0 too short 5697 frames output, 363640 bytes 0 discards, 0 errors 9 input OLS, 9 LRR, 11 NOS, 0 loop inits 9 output OLS, 4 LRR, 8 NOS, 0 loop inits 250 receive B2B credit remaining 32 transmit B2B credit remaining 32 low priority transmit B2B credit remaining Interface last changed at Wed Oct 17 23:50:04 2012

Again this error is not spotted when you are still working on the switches and they are not flapped. Once they are flapped or the switch is rebooted the links will remain down unless they are given another interface in the VSAN database like we just changed.



The final question of this task is for using another protocol in our Fibre Channel Fabric.

This protocol is called the IP over Fibre Channel protocol (IPFC), which is very different from the FCIP protocol!

This protocol is to create IP interfaces across the FC infrastructure. You can use different protocols for this use, one of them is management of the FC fabric network or for using CFS over IP services.

The configuration is very similar to creating Switched Virtual Interfaces (SVI’s) in the Ethernet world. Don’t forget to name the VSAN as this is a requirement from the beginning of this task.

MDS1

MDS1(config)# vsan data MDS1(config-vsan-db)# vsan 50 name IPX_VSAN_50 MDS1(config-vsan-db)# interface port-channel101 MDS1(config-if)# sw trunk allowed vsan add 50 MDS1(config-if)# int port-channel127 MDS1(config-if)# sw trunk allowed vsan add 50 MDS1(config-if)# interface vsan 50 MDS1(config-if)# ip add 198.18.50.1 255.255.255.0 MDS1(config-if)# no shut

MDS2

MDS2(config)# vsan data MDS2(config-vsan-db)# vsan 50 name IPX_VSAN_50 MDS2(config-vsan-db)# interface port-channel101 MDS2(config-if)# sw trunk allowed vsan add 50 MDS2(config-if)# int port-channel127 MDS2(config-if)# sw trunk allowed vsan add 50 MDS2(config-if)# interface vsan 50 MDS2(config-if)# ip add 198.18.50.2 255.255.255.0 MDS2(config-if)# no shut

After the configuration, we can see that the new VSAN is allowed on the Trunk interfaces and we can ping the other MDS switch! This means that the IPFC connection is working.

MDS2(config-if)# do sh int trunk vsan port-channel101 is trunking Vsan 10 is up (None) Vsan 20 is up (None) Vsan 40 is up (None) Vsan 50 is up (None) port-channel127 is trunking Vsan 10 is up (None) Vsan 20 is up (None) Vsan 30 is up (None) Vsan 50 is up (None) MDS2(config-if)# do ping 198.18.50.1 PING 198.18.50.1 (198.18.50.1) 56(84) bytes of data. 64 bytes from 198.18.50.1: icmp_seq=1 ttl=64 time=5.16 ms 64 bytes from 198.18.50.1: icmp_seq=2 ttl=64 time=2.04 ms 64 bytes from 198.18.50.1: icmp_seq=3 ttl=64 time=2.03 ms



64 bytes from 198.18.50.1: icmp_seq=4 ttl=64 time=2.02 ms --- 198.18.50.1 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 5050ms rtt min/avg/max/mdev = 2.014/2.552/5.162/1.167 ms MDS2(config-if)#

Now task 2 is finished and we continue with the next task about Zoning!

Task 3: Zoning This third task will be about configuring a typical Fibre Channel feature. By default nobody in the FC network can communicate to each other! This is a very important topic about Fibre Channel. You can compare Zoning to Access Lists in the IP world. By default there is a deny rule on all the interfaces.

The zoning needs to be configured per VSAN. All devices that are configured under a zone are able to communicate to each other. Devices between zones can’t talk to each other. Zones are combined under a Zoneset. This zoneset is activated on a specific VSAN and spread throughout the FC Fabric.

When devices are not specified within a zone, they are part of the Default Zone. This Default Zone will not process any traffic unless specifically configured in the CLI. Devices can be configured under multiple Zones at the same time. For example a certain server needs access to various JBODs, but the JBODs can not communicate to each other. This would result in a number of zones each specifying the same server and 1 JBOD.

The standard zoning was not sufficient for larger environments. This is why Enhanced Zoning was created. With Enhanced Zoning it becomes possible to perform zoning configuration from a single switch at the same time. With basic zoning, administrators can override each others changes easily by just applying a different zoneset to the VSAN. With Enhanced zoning, all information is distributed within the fabric and the fabric is locked for changes when one administrator starts configuring zoning.

We will start by configuring relatively easy zoning and will build up to more complex zoning including advanced features.

One important thing to remember with zoning is that you should never forget to change the zoning when this is required. When we will be configuring the UCS system later on, we might require some additional zoning to be done, while this might not be specifically mentioned in the lab task, however you can not access your storage without any configuration about the zoning.

The first question in this task is by creating a task by using the device-alias of the devices. Since we already enabled Enhanced Device-Aliases, we will keep seeing these names in the zoning configurations.



Pick easy to remember names for your zones. You could create zone names with the devices in it, but usually what’s best to just use a simple number in the names to make it clear. You can have overlapping zone names between VSANs, as they are never created as VSAN independent.

MDS1

MDS1(config)# zone name ZONE1 vsan 10 MDS1(config-zone)# member device-alias J1D2 MDS1(config-zone)# member device-alias J1D3 MDS1(config-zone)# member device-alias J1D4 MDS1(config-zone)# zoneset name ZONESET1 vsan 10 MDS1(config-zoneset)# member ZONE1 MDS1(config-zoneset)# zoneset activate name ZONESET1 vsan 10 Zoneset activation initiated. check zone status MDS1(config-zone)#

MDS2

MDS2(config)# do sh zoneset active vsan 10 zoneset name ZONESET1 vsan 10 zone name ZONE1 vsan 10 device-alias J1D3 device-alias J1D4 device-alias J1D2 MDS2(config)#

On MDS2 the only thing we need to do is verify, as the zones are distributed throughout the fabric. Pay attention that only the active zones are distributed! MDS2 does not have the configuration for the zones in its running-configuration now!

The next task requires you to use something different than de device-aliases. It’s possible to create zones in many different ways, using various options to include devices.

MDS1

MDS1(config)# zone name ZONE2 vsan 10 MDS1(config-zone)# member ? device-alias Add device-alias member to zone domain-id Add member based on domain-id,port-number fcalias Add fcalias to zone fcid Add FCID member to zone fwwn Add Fabric Port WWN member to zone interface Add member based on interface pwwn Add Port WWN member to zone symbolic-nodename Add Symbolic Node Name member to zone MDS1(config-zone)# member pwwn 21:00:00:11:0d:18:fe:ce MDS1(config-zone)# member pwwn 21:00:00:11:0d:18:a1:ce MDS1(config-zone)# zoneset name ZONESET1 vsan 10 MDS1(config-zoneset)# member ZONE2 MDS1(config-zoneset)# zoneset activate name ZONESET1 vsan 10 Zoneset activation initiated. check zone status MDS1(config)#



Remember to always activate the zoneset. Without activation the configuration is not working and devices can still not communicate to each other!

Now when we verify we see that MDS2 can automatically map the configured PWWNs to the device-alias that is also configured for this JBOD disk.

MDS2(config)# do sh zoneset active vsan 10 zoneset name ZONESET1 vsan 10 zone name ZONE1 vsan 10 device-alias J1D3 device-alias J1D4 device-alias J1D2 zone name ZONE2 vsan 10 pwwn 21:00:00:11:0d:18:fe:ce [J1D5] pwwn 21:00:00:11:0d:18:a1:ce [J1D6] MDS2(config)#

The next task is to add an interface of another MDS switch to the zoning configuration. This can be done by using 2 factors. Either the Domain ID or the SWWN of the switch. The SWWN is a given fact of the switch that is not possible to change, so this would be the preferred way to configure this type of zoning.

MDS2

MDS2(config)# sh wwn switch Switch WWN is 20:00:00:05:9b:73:57:40

We first do a lookup of the SWWN of the switch before we can configure the new zone.

MDS1

MDS1(config)# zone name ZONE3 vsan 10 MDS1(config-zone)# member interface ? fc Fiber Channel interface san-port-channel SAN Port Channel interface MDS1(config-zone)# member interface fc1/6 ? <CR> , Multi range separator - Range separator domain-id Specify domain-id swwn Specify switch-wwn for the interface MDS1(config-zone)# member interface fc1/6 swwn 20:00:00:05:9b:73:57:40 MDS1(config-zone)# zoneset name ZONESET1 vsan 10 MDS1(config-zoneset)# member ZONE3 MDS1(config-zoneset)# zoneset activate name ZONESET1 vsan 10 Zoneset activation initiated. check zone status MDS1(config)#

Then we verify the successful addition of the interface to the zoning.



MDS2(config)# do sh zoneset active vsan 10 zoneset name ZONESET1 vsan 10 zone name ZONE1 vsan 10 device-alias J1D3 device-alias J1D4 device-alias J1D2 zone name ZONE2 vsan 10 pwwn 21:00:00:11:0d:18:fe:ce [J2D1] pwwn 21:00:00:11:0d:18:a1:ce [J2D4] zone name ZONE3 vsan 10 interface fc1/6 swwn 20:00:00:05:9b:73:57:40 MDS2(config)#

Next question is regarding the frames that are sent to the Fibre Channel Broadcast address (0xFFFFFF). By default broadcast frames arrive to all hosts that are configured in zones. You can limit this by using ‘broadcast zoning’. This broadcast zoning is enabled by using the ‘broadcast’ keyword in the zones. When this keyword is configured in one of the zones in the VSAN, the broadcasting is immediately limited to only the zones that have the configuration applied.

This would solve our question! Do not forget to enable broadcast zoning first.

MDS1

MDS1(config-zone)# zone broadcast enable vsan 10 MDS1(config)# zone name ZONE2 vsan 10 MDS1(config-zone)# attribute broadcast MDS1(config-zone)# zoneset activate name ZONESET1 vsan 10 MDS1(config)#

The next question is required to activate your zoning. Best way to do it, would be to activate the zoneset every time you perform a change. It’s difficult to verify what has been enabled and what hasn’t. By just activating it after each change, you will never forget to enable zoning. Again you will loose points on the lab when this hasn’t been done.

The next tasks will let you add another zone to a zoneset, but not change the active zoneset as well. We still want all of the zonesets and zones to be visible on all switches, which is by default not done. The process that performs these changes is known as Full Zoneset Distribution.

We can do this in 2 ways, every time a zone distribution is done, this can be performed. In that case the configuration should be done in the configuration mode, or you can do it only one time, when the command is issued in the operational mode. To be sure we will do it via the configuration this time, as the question did not state if it needs to be done only one time or multiple times.

As we can not change any configuration on MDS1 we first need to be sure to have the configuration in our running-configuration on MDS2. This requires a copy action from the operational CLI first,



before starting to configure zones. Otherwise all configuration on the other MDS switch is overwritten with the new configuration.

MDS2

MDS2# zone copy active-zoneset full-zoneset vsan 10 WARNING: This command may overwrite common zones in the full zoneset. Do you want to continue? (y/n) [n] y MDS2# sh run zone vsan 10 !Command: show running-config zone vsan 10 !Time: Thu Oct 18 09:06:25 2012 version 5.0(3)N1(1a) !Full Zone Database Section for vsan 10 zone name ZONE1 vsan 10 member device-alias J1D3 member device-alias J1D4 member device-alias J1D2 zone name ZONE2 vsan 10 member pwwn 21:00:00:11:0d:18:fe:ce member pwwn 21:00:00:11:0d:18:a1:ce zone name ZONE3 vsan 10 member interface fc1/6 swwn 20:00:00:05:9b:73:57:40 zoneset name ZONESET1 vsan 10 member ZONE1 member ZONE2 member ZONE3 zoneset activate name ZONESET1 vsan 10 MDS2#

Now all zones are configured in the running-configuration of MDS2. Here we can change the zoning as required for the question.

MDS2

MDS2(config)# zone name ZONE4 vsan 10 MDS2(config-zone)# member fcid ? <0x0-0xffffff> Enter FCID MDS2(config-zone)# member fcid 0x8800aa MDS2(config-zone)# member fcid 0x8800bb MDS2(config-zone)# member fcid 0x8800cc MDS2(config-zone)# member fcid 0x8800dd MDS2(config-zone)# member fcid 0x8800ee MDS2(config-zone)# member fcid 0x8800ff MDS2(config-zone)# zoneset name ZONESET2 vsan 10 MDS2(config-zoneset)# member ZONE1 MDS2(config-zoneset)# member ZONE2 MDS2(config-zoneset)# member ZONE4



MDS2(config-zoneset)# exit MDS2(config)# zoneset distribute full vsan 10 MDS2(config)# zoneset activate name ZONESET1 vsan 10 Zoneset activation initiated. check zone status MDS2(config)#

MDS1

MDS1(config)# zoneset distribute full vsan 10

Now when we applied the zoneset, we can check on MDS1 and see that indeed, the non-active zoneset is transferred as well as is the new zone, which is not active in the active zoneset! We do need to activate the full zoneset distribution on MDS1 as well, as this setting is not distributed with the zones.

MDS1# sh zoneset active vsan 10 zoneset name ZONESET1 vsan 10 zone name ZONE1 vsan 10 device-alias J1D3 device-alias J1D4 device-alias J1D2 zone name ZONE2 vsan 10 pwwn 21:00:00:11:0d:18:fe:ce [J2D1] pwwn 21:00:00:11:0d:18:a1:ce [J2D4] zone name ZONE3 vsan 10 interface fc1/6 swwn 20:00:00:05:9b:73:57:40 MDS1# sh run zone vsan 10 !Command: show running-config zone vsan 10 !Time: Thu Oct 18 08:59:58 2012 version 5.0(3)N1(1a) !Full Zone Database Section for vsan 10 zone name ZONE1 vsan 10 member device-alias J1D3 member device-alias J1D4 member device-alias J1D2 zone name ZONE2 vsan 10 member pwwn 21:00:00:11:0d:18:fe:ce member pwwn 21:00:00:11:0d:18:a1:ce zone name ZONE3 vsan 10 member interface fc1/6 swwn 20:00:00:05:9b:73:57:40 zone name ZONE4 vsan 10 member fcid 0x8800aa member fcid 0x8800bb member fcid 0x8800cc member fcid 0x8800dd member fcid 0x8800ee member fcid 0x8800ff



zoneset name ZONESET1 vsan 10 member ZONE1 member ZONE2 member ZONE3 zoneset name ZONESET2 vsan 10 member ZONE1 member ZONE2 member ZONE4 zoneset activate name ZONESET1 vsan 10 MDS1#

And we completed all zoning for VSAN 10!

For zoning the next VSAN, we will require to use a new type of zoning. Like previously mentioned a new way of zoning has been introduced in a newer version of the Fibre Channel standards. This type of zoning is commonly referred to as ‘Enhanced Zoning’. The standards that describe this are FC-GS-4 and FC-SW-3. This big advantage of enhanced zoning is that all changes to the zones are immediately distributed throughout the fabric. You do still need to perform a manual activation of the zoning after the configuration. A commit of all changes is not enough to make them active! We also need to use inline zone creation. This is a way that you can configure zones right out of the zoneset configuration prompt, making it very easy to use and to configure multiple zones at once.

We start the configuration by making the first zone, which is also asking for some read-only zoning. Instead of using the attributes right in the zone, Enhanced Zoning requires to use so-called attribute-groups that can be applied to zones.

MDS1

MDS1(config)# zone mode enhanced vsan 20 WARNING: This command would distribute the zoning database of this switch throughout the fabric. Do you want to continue? (y/n) [n] y Set zoning mode command initiated. Check zone status MDS1(config)#

After the enhanced zoning has been configured, it’s seen on all other switches. This change is immediately distributed.

MDS2(config)# sh zone status vsan 20 VSAN: 20 default-zone: deny distribute: active only Interop: default mode: enhanced merge-control: allow session: none hard-zoning: enabled broadcast: enabled Default zone: qos: none broadcast: disabled ronly: unsupported Full Zoning Database : DB size: 44 bytes Zonesets:0 Zones:0 Aliases: 0 Attribute-groups: 1



Active Zoning Database : Database Not Available Status: Set zoning mode success at 10:03:41 UTC Oct 18 2012 MDS2(config)#

Now the zones can be created inline. We are configuring the first zone.

MDS1

MDS1(config)# zone-attribute-group name RO vsan 20 MDS1(config-attribute-group)# read-only MDS1(config-attribute-group)# exit MDS1(config)# zoneset name ZONESET1 vsan 20 Enhanced zone session has been created. Please 'commit' the changes when done. MDS1(config-zoneset)# zone ? name Zone name MDS1(config-zoneset)# zone name ZONE1 MDS1(config-zoneset-zone)# member device-alias J2D1 MDS1(config-zoneset-zone)# member device-alias J2D2 MDS1(config-zoneset-zone)# member device-alias J2D3 MDS1(config-zoneset-zone)# attribute-group RO MDS1(config-zoneset-zone)# exit MDS1(config-zoneset)# zoneset activate name ZONESET1 vsan 20 MDS1(config)# zone commit vsan 20 Commit operation initiated. Check zone status

Now we verify that everything is applied correctly

MDS1(config)# sh run zone vsan 20 !Command: show running-config zone vsan 20 !Time: Thu Oct 18 09:59:28 2012 version 5.0(3)N1(1a) zone mode enhanced vsan 20 !Active Zone Database Section for vsan 20 zone name ZONE1 vsan 20 member device-alias J2D1 member device-alias J2D2 member device-alias J2D3 zoneset name ZONESET1 vsan 20 member ZONE1 zoneset activate name ZONESET1 vsan 20 do clear zone database vsan 20 !Full Zone Database Section for vsan 20 zone-attribute-group name RO vsan 20 read-only zone name ZONE1 vsan 20 attribute-group RO member device-alias J2D1 member device-alias J2D2



member device-alias J2D3 zoneset name ZONESET1 vsan 20 member ZONE1 zone commit vsan 20 MDS1(config)#

The next zoning is using a feature which is dedicated to Cisco MDS switches due to the fact that it requires Trunk E ports between switches to transport a specific field.

Within Cisco MDS switches it’s possible to use Quality of Service features within zones. By using these features it’s possible to give certain hosts or JBODs a higher priority over other traffic. In case of congestion on the TE-ports between the switches, the traffic marked with a high priority gets served first, after that the medium priority traffic is served and finally the low priority traffic. The QoS settings are applied per zone. The marking of the traffic is carried in a special field in the EISL header.

We will configure the next zone with the QoS settings and the FWWNs of the disks instead of their device-alias or PWWN. You can easily find the FWWN with the show fcns database or show flogi database on the switch where the JBODs are connected.

Don’t forget to use inline zone creation as this was a requirement for the whole of VSAN 20.

MDS1

MDS1(config)# zone-attribute-group name HIGH vsan 20 Enhanced zone session has been created. Please 'commit' the changes when done. MDS1(config-attribute-group)# qos ? priority Priority to be used for frames matching zone MDS1(config-attribute-group)# qos prio ? high Frames matching zone get high priority low Frames matching zone get low priority medium Frames matching zone get medium priority MDS1(config-attribute-group)# qos prio high MDS1(config-attribute-group)# zoneset name ZONESET1 vsan 20 MDS1(config-zoneset)# zone name ZONE2 MDS1(config-zoneset-zone)# attribute-group HIGH MDS1(config-zoneset-zone)# member fwwn 22:00:00:11:0d:18:a1:ce MDS1(config-zoneset-zone)# member fwwn 22:00:00:11:0d:18:a1:de MDS1(config-zoneset-zone)# zoneset activate name ZONESET1 vsan 20 MDS1(config)# zone commit vsan 20 Commit operation initiated. Check zone status

Now to verify the zones have been applied correctly:

MDS1(config)# sh run zone vsan 20 !Command: show running-config zone vsan 20



!Time: Thu Oct 18 10:17:46 2012 version 5.0(3)N1(1a) zone mode enhanced vsan 20 !Active Zone Database Section for vsan 20 zone name ZONE1 vsan 20 member device-alias J2D1 member device-alias J2D2 member device-alias J2D3 zone name ZONE2 vsan 20 member fwwn 22:00:00:11:0d:18:a1:ce member fwwn 22:00:00:11:0d:18:a1:de zoneset name ZONESET1 vsan 20 member ZONE1 member ZONE2 zoneset activate name ZONESET1 vsan 20 do clear zone database vsan 20 !Full Zone Database Section for vsan 20 zone-attribute-group name RO vsan 20 read-only zone-attribute-group name HIGH vsan 20 qos priority high zone name ZONE1 vsan 20 attribute-group RO member device-alias J2D1 member device-alias J2D2 member device-alias J2D3 zone name ZONE2 vsan 20 attribute-group HIGH member fwwn 22:00:00:11:0d:18:a1:ce member fwwn 22:00:00:11:0d:18:a1:de zoneset name ZONESET1 vsan 20 member ZONE1 member ZONE2 zone commit vsan 20

The next question asks that when devices in VSAN 20 are not configured for zoning, they should still be able to read data from each other. The only way we can solve this is by allowing traffic between the hosts that are in the default zone. Besides that they are then immediately able to write data to each other as well. To stop this, we can change the attributes of the default zone so we can apply the read-only zoning option as well.

MDS1

MDS1(config)# zone default-zone permit vsan 20 Enhanced zone session has been created. Please 'commit' the changes when done. MDS1(config)# zone default-zone vsan 20



MDS1(config-default-zone)# read-only MDS1(config)# zone commit vsan 20 Commit operation initiated. Check zone status

The next topic is regarding a special feature called LUN zoning. This means that when you specify the LUN ID which needs to be reached behind the zoning, only this LUN is reachable. In this task you are asked to configure this.

Remember for the LUN numbers, that they are in hexadecimal notation!

MDS1

MDS1(config)# zoneset name ZONESET1 vsan 20 MDS1(config-zoneset)# zone name ZONE3 Enhanced zone session has been created. Please 'commit' the changes when done. MDS1(config-zoneset-zone)# member device-alias J2D5 lun 0x13 MDS1(config-zoneset-zone)# member device-alias J1D6 lun 0x74 MDS1(config-zoneset-zone)# zoneset activate name ZONESET1 vsan 20 MDS1(config)# zone commit vsan 20 Commit operation initiated. Check zone status MDS1(config)#

Finally we make the zoning active and we are done!

Task 4: FC Domain The next task is regarding a technology called FC Domain. The FC Domain process is used within the FC Fabric to ensure unique ID’s are assigned to switches. This Domain ID is then used for routing the Fibre Channel traffic to other switches. Therefore the Domain ID is a crucial part of the FC Fabric configuration.

Within a fabric there is a switch chosen to be the Principal switch. This principal switch is then responsible for handing out the Domain IDs to other switches in the fabric, when they are set to automatically configuring the Domain ID. The Principal switch is selected based on a configured priority value and as a tie-breaker the lowest SWWN is used.

The following steps occur when a new FC Fabric is converging:

Principal Switch Election

By using priority values or lowest SWWN a Principal switch of the entire Fibre Channel Fabric (per VSAN) is chosen.

Domain ID distribution



The principal switch ensures that there are only unique Domain IDs in the fabric. Switches can statically configure their own Domain ID, but they always need to ask the Principal switch if this is still a unique value in the fabric. When there are duplicate Domain ID’s the network will not work!

When switches are configured to automatically get a Domain ID assigned, the Principal switch will do this.

FC ID allocation

When Domain IDs are allocated to the switches. The switches can start by assigning FCID blocks (areas) to the connected devices. They base this on their allocated domain ID. The first byte of the FCID is the Switch Domain ID. The second byte is the Area code and the last byte is the host address

Fabric reconfiguration

A domain restart is crucial when new FC Domain IDs are selected for the FC Fabric. This restart is disruptive for the traffic. All traffic is stopped until the FC Domain and FSPF protocol are fully converged.

The first tasks are all regarding VSAN 10. MDS1 should be the principal switch and have a static ID configured. The second switch should prefer a Domain ID.

Pay attention to Decimal and Hexadecimal numbers in these tasks. It depends on the show command if the Domain ID is presented in decimal or in hexadecimal notation.

By default the FC Domain priority is set to 128. When a principal switch is elected, it changes it’s ‘running’ priority to 2. This is to ensure that it remains the principal switch, even when a new switch comes into the fabric with a lower SWWN. When we are configuring priority values, we only need to go under the configured priority of the switches as those are the priorities that are used in times of convergence of the fabric in the VSAN.

MDS2

MDS2(config)# fcdomain domain ? <0-239> Configure the domain id in decimal, 0 not allowed for static dom <0x0-0xef> Configure the domain id in hexadecimal MDS2(config)# fcdomain domain 0x34 preferred vsan 10

MDS1

MDS1(config)# fcdomain priority 99 vsan 10 MDS1(config)# fcdomain domain 34 static vsan 10 MDS1(config)# fcdomain restart disruptive vsan 10 MDS1(config)#



We require a Disruptive restart before we will be seeing any results of our configuration changes.

MDS1

MDS1(config)# sh fcdomain vsan 10 The local switch is the Principal Switch. Local switch run time information: State: Stable Local switch WWN: 20:0a:00:05:9b:73:1b:c1 Running fabric name: 20:0a:00:05:9b:73:1b:c1 Running priority: 2 Current domain ID: 0x22(34) Local switch configuration information: State: Enabled FCID persistence: Enabled Auto-reconfiguration: Disabled Contiguous-allocation: Disabled Configured fabric name: 20:01:00:05:30:00:28:df Optimize Mode: Disabled Configured priority: 99 Configured domain ID: 0x22(34) (static) Principal switch run time information: Running priority: 2 Interface Role RCF-reject ------------------ ------------- ------------ port-channel101 Downstream Disabled port-channel127 Downstream Disabled ------------------ ------------- ------------ MDS1(config)#

MDS2

MDS2(config)# sh fcdomain vsan 10 The local switch is a Subordinated Switch. Local switch run time information: State: Stable Local switch WWN: 20:0a:00:05:9b:73:57:41 Running fabric name: 20:0a:00:05:9b:73:1b:c1 Running priority: 128 Current domain ID: 0x34(52) Local switch configuration information: State: Enabled FCID persistence: Enabled Auto-reconfiguration: Disabled Contiguous-allocation: Disabled Configured fabric name: 20:01:00:05:30:00:28:df Optimize Mode: Disabled Configured priority: 128 Configured domain ID: 0x34(52) (preferred)



Principal switch run time information: Running priority: 2 Interface Role RCF-reject ------------------ ------------- ------------ port-channel101 Upstream Disabled port-channel127 Upstream Disabled ------------------ ------------- ------------ MDS2(config)#

We can see that MDS1 is the principal switch of the fabric with the correct priority configured. The Domain ID’s also match with what the question described. It’s also shown that the Domain ID’s can be configured in either decimal or hexadecimal format, like they are shown in the show command as well.

Next we need to ensure that Domain IDs are handed out in a sequential order. By default Domain ID’s are handed out in any order. This change is only necessary on the Principal switch as this switch is handing out the Domain ID’s.

MDS1

MDS1(config)# fcdomain contiguous-allocation vsan 10 MDS1(config)# fcdomain disruptive restart vsan 10

The disruptive restart is a very tough restart. All traffic in the VSAN in the Fabric is interrupted when this occurs. Therefore there has been made a security mechanism to prevent this from happening from each switch in the network. This feature enables that only a few switches in the network can trigger the disruptive restart.

When you configure different Domain IDs in your network a disruptive restart is necessary to make them active in the fabric. This is because the Domain IDs are used for allocating FCIDs to the connected hosts. When the Domain ID changes, these have to change as well.

Therefore we limit the disruptive restarts on MDS1. This means that MDS2 can never issue a disruptive restart for the whole FC Fabric in VSAN 10.

MDS1

MDS1(config)# fcdomain rcf-reject vsan 10 MDS1(config)# fcdomain disruptive restart vsan 10

The next task is regarding a fixed allocation of a certain device connected to MDS1. JBOD1 is connected to MDS1 in VSAN 10, where this can get assigned static FCIDs.

The ID’s that need to be assigned are in the range of the Domain ID that is running on the switch. The only way to statically configure the FCID’s is when they match with the running Domain ID of course.



We will configure the static allocation of the FCID:

MDS1

MDS1(config)# fcdomain fcid database MDS1(config-fcid-db)# vsan 10 wwn c6:83:00:11:0d:18:fa:ce fcid 0x222200 area

Now by specifying the WWN of J1D1 and the FCID that we want we can specify the area keyword behind it. This means that it will assign this entire area to that particular WWN and allocate all FCID’s just for this disk.

When this disk will perform a new FLOGI to the switch, it will get this FCID assigned!

The next question will assign Domain IDs to the switches in VSAN 20. Again pay attention to the hexadecimal notation of the allowed FC Domain IDs that can be handed out by the Principal Switch. In this case MDS2 should be the Principal switch. MDS1 should be requesting a domain ID.

MDS1

MDS1(config)# fcdomain domain 214 preferred vsan 20

MDS2

MDS2(config)# fcdomain priority 99 vsan 20 MDS2(config)# fcdomain allowed 176-206 vsan 20 Error: attempt to manually set an allowed domain list that would not include all the currently assigned and locally configured domains or that would conflict with existing allowed domain lists. MDS2(config)# fcdomain domain 176 static vsan 20 MDS2(config)# fcdomain restart disruptive vsan 20 MDS2(config)# fcdomain allowed 176-206 vsan 20 Error: the VSAN is not stable. Please try again after it stabilizes. MDS2(config)# fcdomain allowed 176-206 vsan 20 Error: attempt to manually set an allowed domain list that would not include all the currently assigned and locally configured domains or that would conflict with existing allowed domain lists.

What happens here is that the allowed list can’t be set in the VSAN unless all Domain IDs are currently compliant with the configuration. This means that we need to change the Domain ID of MDS1 temporarily because this introduces the issue.

MDS1

MDS1(config)# fcdomain domain 205 static vsan 20 MDS1(config)# fcdomain restart disruptive vsan 20 MDS1(config)# fcdomain domain 214 preferred vsan 20

Once we change the Domain ID of MDS1 back to something that falls within the range that we want to configure. After the allowed list is configured a final restart of the VSAN is done.



MDS2

MDS2(config)# fcdomain allowed 176-206 vsan 20 MDS2(config)# fcdomain restart disruptive vsan 20

Now we can verify if everything works like we expect it to work.

MDS2(config)# sh fcdomain vsan 20 The local switch is the Principal Switch. Local switch run time information: State: Stable Local switch WWN: 20:14:00:05:9b:73:57:41 Running fabric name: 20:14:00:05:9b:73:57:41 Running priority: 2 Current domain ID: 0xb0(176) Local switch configuration information: State: Enabled FCID persistence: Enabled Auto-reconfiguration: Disabled Contiguous-allocation: Disabled Configured fabric name: 20:01:00:05:30:00:28:df Optimize Mode: Disabled Configured priority: 99 Configured domain ID: 0xb0(176) (static) Principal switch run time information: Running priority: 2 Interface Role RCF-reject ------------------ ------------- ------------ port-channel101 Downstream Disabled port-channel127 Downstream Disabled ------------------ ------------- ------------ MDS2(config)#

MDS2 looks fine. We see that it is elected as the Principal Switch of the VSAN and that it is running a Domain ID that falls within the range.

Now how is MDS 1 working, as we have configured a preferred Domain ID which is outside of the range that is allowed.

MDS1(config)# sh fcdomain vsan 20 The local switch is a Subordinated Switch. Local switch run time information: State: Stable Local switch WWN: 20:14:00:05:9b:73:1b:c1 Running fabric name: 20:14:00:05:9b:73:57:41 Running priority: 128 Current domain ID: 0xce(206) Local switch configuration information: State: Enabled



FCID persistence: Enabled Auto-reconfiguration: Disabled Contiguous-allocation: Disabled Configured fabric name: 20:01:00:05:30:00:28:df Optimize Mode: Disabled Configured priority: 128 Configured domain ID: 0xd6(214) (preferred) Principal switch run time information: Running priority: 2 Interface Role RCF-reject ------------------ ------------- ------------ port-channel101 Upstream Disabled port-channel127 Upstream Disabled ------------------ ------------- ------------ MDS1(config)#

The switch is running another Domain ID as the preferred one was not allowed. Therefore the Principal switch allocated another ID to this switch, which it is using now.

The final question is related to a feature called fast-restart, which has been recently available on the MDS switches. We need to turn this feature on in VSAN 30. This is one of those features that is easily looked up in the Documentation on Cisco.com.

MDS1

MDS1(config)# fcdomain optimize fast-restart vsan 30 MDS1(config)#

MDS2

MDS2(config)# fcdomain optimize fast-restart vsan 30 MDS2(config)# fcdomain restart disruptive vsan 30 MDS2(config)#

Now task 4 is finished and we have a good working FC Domain configuration.

Task 5: Fibre Channel Security Features This task will focus on the security features that are available in the Fibre Channel network. This task focuses on 3 major security features:

Port Security

Port-security ensures that there are no rogue hosts getting connected to the Fibre Channel Fabric. The feature checks if the device connected has an equal PWWN, FWWN or SWWN otherwise it is rejected.



Port Security can also protect that devices are not connected on different ports. So devices are only allowed to come online on certain interfaces. This can be one or more.

Fabric Binding

Fabric Binding is a technology used to protect the FC switches in a Fabric. You will need to have the SWWNs configured of all switches before it may come online in the fabric. You can choose to even protect the Domain IDs configured on these SWWNs

FC-SP / DH-CHAP

FC-SP is a protocol which is also called DH-CHAP. Configuration commands all start with DHCHAP. This feature ensures that all connected devices on the enabled interfaces are authenticated on the switch before they are allowed access. They need to use a pre-defined password (either local or via RADIUS) that needs to match before the log-in is allowed on the switch.

The port-security feature is the first feature that is configured. The initial task is using a specific feature called auto-learning on port-security. This means that all devices that are connected to the VSAN are automatically added to the port-security database and saved.

The port-security database can be distributed throughout the Fibre Channel fabric as well, so all auto-learned entries are saved to all switches.

In the initial task you are not aware of the WWNs in the fabric so you will require using the auto-learning feature.

MDS1

MDS1(config)# port-security enable MDS1(config)# port-security auto-learn vsan 10 Error for VSAN 10: No Active database MDS1(config)# port-security activate vsan 10 MDS1(config)# port-security auto-learn vsan 10

Now we enabled the port-security in VSAN 10 with auto-learning enabled. The next step is to ensure that it is seen in the configuration of the device. This requires a command on the operational prompt to copy the auto-learnt entries into the configuration.

MDS1

MDS1(config-if)# do sh port-security data v 10 -------------------------------------------------------------------------------- VSAN Logging-in Entity Logging-in Point (Interface) -------------------------------------------------------------------------------- MDS1(config-if)# end MDS1# port-security database copy vsan 10 MDS1# sh port-sec data



-------------------------------------------------------------------------------- VSAN Logging-in Entity Logging-in Point (Interface) -------------------------------------------------------------------------------- 10 c6:81:00:11:0d:17:fa:ce(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)* 10 c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)* 10 c6:80:00:11:0d:07:fa:ce(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)* 10 c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)* 10 20:01:54:7f:ee:05:08:79(swwn) 20:05:00:0d:ec:6f:4f:40(fc1/5)* 10 20:01:54:7f:ee:05:08:79(swwn) 20:06:00:0d:ec:6f:4f:40(fc1/6)* [Total 6 entries] MDS1#

In the configuration you will find all entries now.

MDS1# sh run vsan 10 version 3.3(1c) vsan database vsan 10 fcdomain fcid database vsan 10 wwn 20:03:00:0d:ec:6f:4f:40 fcid 0x200000 area dynamic vsan 10 wwn 20:04:00:0d:ec:6f:4f:40 fcid 0x200100 area dynamic vsan database vsan 10 interface fc1/3 vsan 10 interface fc1/4 port-security database vsan 10 pwwn c6:81:00:11:0d:17:fa:ce interface fc1/4 pwwn c6:81:00:11:0d:02:01:07 interface fc1/4 pwwn c6:80:00:11:0d:07:fa:ce interface fc1/3 pwwn c6:80:00:11:0d:01:01:07 interface fc1/3 swwn 20:01:54:7f:ee:05:08:79 interface fc1/5 swwn 20:01:54:7f:ee:05:08:79 interface fc1/6 port-security activate vsan 10

This copy action is a manual action. It makes it very easy for the initial roll-out of port-security to have it enabled first and then disabled after the initial learning. Then you can add and remove things in the database manually.

In this case there is no reason to disable the auto-learning so we leave it enabled.

Don’t forget to perform this same action on MDS2, as there is no distribution enabled in this question.

MDS2

MDS2(config)# port-security enable MDS2(config)# port-security activate vsan 10 MDS2(config)# port-security auto-learn vsan 10 MDS2(config)# end MDS2# port-security database copy vsan 10

After it’s enabled we see the auto learned entries showing up in our database.



MDS2# sh port-sec data -------------------------------------------------------------------------------- VSAN Logging-in Entity Logging-in Point (Interface) -------------------------------------------------------------------------------- 10 21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78(fc1/1)* 10 20:00:00:0d:ec:6f:4f:40(swwn) 20:05:54:7f:ee:05:08:78(fc1/5)* 10 20:00:00:0d:ec:6f:4f:40(swwn) 20:06:54:7f:ee:05:08:78(fc1/6)* [Total 3 entries] MDS2#

The next task will prepare you for new devices that will be connected into the fabric. The port-security feature also knows the concept of wild-cards. You can enable certain characteristics of a host in the database where you are unsure on which interface it will be connected. This means you can leave it completely blank or you can add only a switch where it is added.

As we don’t have any distribution of the database entries between the MDS switches, we do have an overlap in this question as one of the PWWNs can be connected to either MDS switch.

MDS1

MDS1(config)# port-sec database vsan 10 MDS1(config-port-security)# ? Port Security Lists: any-wwn Any Node/Port/Switch WWNs device-alias N_Port Device_Alias do EXEC command end Exit from configure mode exit Exit from this submode no Negate a command or set its defaults nwwn N_Port Node-WWN pwwn N_Port Port-WWN swwn Switch-WWN MDS1(config-port-security)# pwwn 20:00:00:a3:bf:33:11:33 ? fwwn Swith-Port WWN interface Switch Interface name swwn Switch WWN <cr> Carriage return. MDS1(config-port-security)# pwwn 20:00:00:a3:bf:33:11:33 interface fc1/11 MDS1(config-port-security)# pwwn 20:00:00:a3:fe:00:98:32

After the 2 entries have been added we see in the database that the entries area added and that for the entry which has an interface configured the SWWN of MDS1 is added to it.

MDS1(config)# do sh port-sec database -------------------------------------------------------------------------------- VSAN Logging-in Entity Logging-in Point (Interface) -------------------------------------------------------------------------------- 10 c6:81:00:11:0d:17:fa:ce(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)* 10 c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)* 10 c6:80:00:11:0d:07:fa:ce(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*



10 c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)* 10 20:01:54:7f:ee:05:08:79(swwn) 20:05:00:0d:ec:6f:4f:40(fc1/5)* 10 20:01:54:7f:ee:05:08:79(swwn) 20:06:00:0d:ec:6f:4f:40(fc1/6)* 10 20:00:00:a3:bf:33:11:33(pwwn) 20:0b:00:0d:ec:6f:4f:40(fc1/11)* 10 20:00:00:a3:fe:00:98:32(pwwn) Any [Total 8 entries] MDS1(config)# do sh wwn sw Switch WWN is 20:00:00:0d:ec:6f:4f:40 MDS1(config)#

Now we need to configure MDS2 for the remaining PWWNs.

MDS2

MDS2(config)# port-security data vsan 10 MDS2(config-port-security)# pwwn 20:00:00:a3:fe:00:98:32 MDS2(config-port-security)# do sh wwn sw Switch WWN is 20:00:54:7f:ee:05:08:78 MDS2(config-port-security)# pwwn 20:00:00:A3:DE:11:66:2B swwn 20:00:54:7f:ee:05:08:78

Verification about the just configured parameters. It’s shown that one of the PWWNs can only be connected to the SWWN of MDS2 and the other entry can be connected anywhere in the fabric.

MDS2(config-port-security)# do sh port-sec database -------------------------------------------------------------------------------- VSAN Logging-in Entity Logging-in Point (Interface) -------------------------------------------------------------------------------- 10 21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78(fc1/1)* 10 20:00:00:0d:ec:6f:4f:40(swwn) 20:05:54:7f:ee:05:08:78(fc1/5)* 10 20:00:00:0d:ec:6f:4f:40(swwn) 20:06:54:7f:ee:05:08:78(fc1/6)* 10 20:00:00:a3:de:11:66:2b(pwwn) 20:00:54:7f:ee:05:08:78(swwn) 10 20:00:00:a3:fe:00:98:32(pwwn) Any [Total 5 entries] MDS2(config-port-security)#

The next question in this task is to secure VSAN 20 and to manually configure all port-security parameters. This time all parameters need to be distributed between the switches, because we can only change MDS1 to finish this task. When we need to be as specific as possible, it means that we need to connect PWWNs to their respective interfaces. This is the most specific we can get in a port-security environment.

We first need the FLOGI information of both of the switches so we can create a database using this information.

MDS1

MDS1(config-port-security)# do sh flogi database vsan 20 --------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME



--------------------------------------------------------------------------- fc1/3 20 0x0a00e8 c6:80:00:11:0d:07:fa:ce c5:9a:00:06:2b:01:00:07 fc1/3 20 0x0a00ef c6:80:00:11:0d:01:01:07 c5:9a:00:06:2b:01:00:07 fc1/4 20 0x0a01e8 c6:81:00:11:0d:17:fa:ce c5:9a:00:06:2b:02:00:07 fc1/4 20 0x0a01ef c6:81:00:11:0d:02:01:07 c5:9a:00:06:2b:02:00:07 Total number of flogi = 4.

MDS2

MDS2(config)# sh flogi data v 20 -------------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME -------------------------------------------------------------------------------- fc1/1 20 0x0c0000 21:01:00:e0:8b:3c:c0:b5 20:01:00:e0:8b:3c:c0:b5 Total number of flogi = 1.

Now onto the configuration of our database.

MDS1

MDS1(config-port-security)# port-security database vsan 20 MDS1(config-port-security)# pwwn c6:80:00:11:0d:07:fa:ce interface fc1/3 MDS1(config-port-security)# pwwn c6:80:00:11:0d:01:01:07 interface fc1/3 MDS1(config-port-security)# pwwn c6:81:00:11:0d:17:fa:ce interface fc1/4 MDS1(config-port-security)# pwwn c6:81:00:11:0d:02:01:07 interface fc1/4 MDS1(config-port-security)# pwwn 21:01:00:e0:8b:3c:c0:b5 ? fwwn Swith-Port WWN interface Switch Interface name swwn Switch WWN <cr> Carriage return. MDS1(config-port-security)# pwwn 21:01:00:e0:8b:3c:c0:b5 swwn 20:00:54:7f:ee:05:08:78 interface fc1/1 MDS1(config)# port-security activate vsan 20 MDS1(config)# port-security commit v 20

After activating the database on the MDS1, we can see an activated database on both MDS switches with all necessary information for activating the security in VSAN 20.

MDS1

MDS1(config)# do sh port-security data v 20 -------------------------------------------------------------------------------- VSAN Logging-in Entity Logging-in Point (Interface)



-------------------------------------------------------------------------------- 20 c6:80:00:11:0d:07:fa:ce(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)* 20 c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)* 20 c6:81:00:11:0d:17:fa:ce(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)* 20 c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)* 20 21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78* [Total 5 entries] MDS1(config)#

MDS2

MDS2(config)# do sh port-security data v 20 -------------------------------------------------------------------------------- VSAN Logging-in Entity Logging-in Point (Interface) -------------------------------------------------------------------------------- 20 c6:80:00:11:0d:07:fa:ce(pwwn) 20:03:00:0d:ec:6f:4f:40* 20 c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40* 20 c6:81:00:11:0d:17:fa:ce(pwwn) 20:04:00:0d:ec:6f:4f:40* 20 c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40* 20 21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78*(fc1/1) [Total 5 entries] MDS2(config)#

The last entry on MDS2 shows that it sees the interface that is configured on MDS1 back on MDS2. Our port-security with distribution works!

Next up is Fabric Binding! This technology is used to protect Fibre Channel fabrics of rogue switches to log-in to the fabric. It’s also possible to create this kind of protection by using Port Security. You can specify all of the SWWNs that are allowed access to the Fibre Channel fabric. However the fabric-binding technology is more useful for this, because it can also use the Domain ID for securing the fabric.

The initial task is to secure VSAN 30 so only MDS1 and MDS2 are allowed. Because we should also add the Domain IDs in this task, we should make these static as a reboot might change the Domain ID of a switch, which might cause the fabric to reject the switches and you might end up with loosing points for this section.

The configuration must be done from both switches as fabric-binding cannot be distributed between switches.

MDS1

MDS1(config)# sh wwn sw Switch WWN is 20:00:00:05:9b:70:5a:00 MDS1(config)# fabric-binding data vsan 30 MDS1(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00 domain 1 MDS1(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8 domain 2 MDS1(config-fabric-binding)# exit MDS1(config)# fabric-binding activate v 30



MDS2

MDS2(config)# sh wwn sw Switch WWN is 20:00:54:7f:ee:09:3e:c8 MDS2(config)# fabric-binding data vsan 30 MDS2(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00 domain 1 MDS2(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8 domain 2 MDS2(config-fabric-binding)# exit MDS2(config)# fabric-binding activate v 30

After activating the fabric-binding database we can verify the changes.

MDS1

MDS1(config)# sh fabric-binding data -------------------------------------------------- Vsan Logging-in Switch WWN Domain-id -------------------------------------------------- 30 20:00:00:05:9b:70:5a:00 0x1(1) [Local] 30 20:00:54:7f:ee:09:3e:c8 0x2(2) [Total 2 entries]

MDS2

MDS2(config)# sh fabric-binding data -------------------------------------------------- Vsan Logging-in Switch WWN Domain-id -------------------------------------------------- 30 20:00:54:7f:ee:09:3e:c8 0x2(2) [Local] 30 20:00:00:05:9b:70:5a:00 0x1(1) [Total 2 entries] MDS2(config)# sh int trunk vsan port-channel101 is trunking Vsan 1 is up (None) Vsan 10 is up (None) Vsan 20 is up (None) Vsan 30 is up (None) Vsan 40 is up (None) port-channel127 is trunking Vsan 1 is up (None) Vsan 10 is up (None) Vsan 20 is up (None) Vsan 30 is up (None) Vsan 40 is up (None) MDS2(config)#

When the VSAN is still up on the trunk interfaces we know that the fabric-binding is properly configured!

The final topic in this security task is the FC-SP feature. This feature enables the authentication of host-switch and switch-switch interfaces. It combines a CHAP authentication with a Diffie-Hellman exchange. The steps for configuring FC-SP can become quite complex as a lot of different combinations are possible.



An important topic is the combination of interface modes. You can configure an interface with different modes as it might be required that you configure an interface with optional authentication. The table below demonstrates the different modes and what will happen when these 2 interfaces are connected to each other.

On Auto-Active Auto-Passive Off

On Yes Yes Yes Link down

Auto-Active Yes Yes Yes No authentication

Auto-Passive Yes Yes No authentication No authentication

Off Link down No authentication No authentication No authentication

The On and Off modes are simple modes that do what they tell. They enforce authentication or reject it. The auto modes are there for optional authentication. There should be at least 1 active link when both sides are on auto as the passive link will wait for an authentication request before starting any negotiation.

Behind the mode you have the option to configure the re-authentication timeout, so the authentication is performed a couple of times and is not done only once in the lifetime of the interface.

We start by configuring the most powerful Diffie-Hellman group exchange.

MDS1

MDS1(config)# feature fcsp MDS1(config)# fcsp dhchap dhgroup 4

MDS2

MDS2(config)# feature fcsp MDS2(config)# fcsp dhchap dhgroup 4

The next step is to configure the password that you want to accept on the switch for authentication requests. This can be configured as one key per peer device.

This means that when you have several devices connected to the switch you can configure 1 key per connected device, by specifying the WWN of the switch.

In our example we need to be as specific as possible, although we only have 2 devices, we will still configure that this key is only accepted for that particular MDS switch that is connected.



MDS1

MDS1(config)# fcsp dhchap ? devicename Configure password of the remote devices dhgroup Configure DHCHAP Diffie-Hellman Group List (In preference) hash Configure DHCHAP Hash Algorithm List (In order of preference) password Configure local DHCHAP password MDS1(config)# fcsp dhchap password IPexpertMDS1 ? <CR> <hh:hh:hh:hh:hh:hh:hh:hh> WWN of the device with which to use this local password MDS1(config)# fcsp dhchap password IPexpertMDS1 20:00:54:7f:ee:09:3e:c8

MDS2

MDS2(config)# fcsp dhchap password IPexpertMDS2 20:00:00:05:9b:70:5a:00

The password command is used for the accepted password on the local switch, whereas the devicename command will configure the password that is used to authenticate the neighbor, so the sending password.

Of course we need to configure this as well.

MDS1

MDS1(config)# fcsp dhchap device 20:00:54:7f:ee:09:3e:c8 password IPexpertMDS2

MDS2

MDS2(config)# fcsp dhchap devicename 20:00:00:05:9b:70:5a:00 password IPexpertMDS1

Now the passwords are configured and we can start the first authentication. MDS2 should not initiate any request, so we will make this a passive link. MDS1 should set the link down when the switch doesn’t respond. As described in the table above, the only option that makes a link go down is the ‘on/off’ modes. So we are going to use this in this first task.

MDS1

MDS1(config)# interface fc1/1 MDS1(config-if)# fcsp on ? <CR> <0-100000> Provide reauthentication interval in minutes MDS1(config-if)# fcsp on 15

MDS2

MDS2(config-if)# int fc1/1



MDS2(config-if)# fcsp auto-passive MDS2(config-if)#

When the FC-SP protocol is configured and the link is re-initialized, the authentication is working!

MDS1

MDS1(config-if)# sh fcsp interface fc1/1 fc1/1: fcsp authentication mode:SEC_MODE_ON reauthentication timeout (in minutes):15 Status:Successfully authenticated on Thu Oct 18 11:57:28 2012 Authenticated using local password database

MDS2

MDS2(config-if)# sh fcsp interface fc1/1 fc1/1: fcsp authentication mode:SEC_MODE_AUTO_PASSIVE Status:Successfully authenticated on Thu Oct 18 11:56:20 2012 Authenticated using local password database

The next link to authenticate needs to use SHA1 hashes with a fall-back to MD5. By default this is the other way around. MD5 hashes are used by default with a fall-back to SHA1.

These are changed for the entire switch at the same time. We will configure both switches to respond actively to authentication requests.

MDS1

MDS1(config-if)# fcsp dhchap hash sha1 md5 MDS1(config)# interface fc1/2 MDS1(config-if)# fcsp auto-active

MDS2

MDS2(config-if)# fcsp dhchap hash sha1 md5 MDS2(config-if)# int fc1/2 MDS2(config-if)# fcsp auto-active MDS2(config-if)#

On the third link, which is the second member of port-channel 101 (fc1/3) we will completely disable FC-SP, which is an easy thing to do. Keep in mind that ports have FC-SP auto-passive enabled by default!

MDS1

MDS1(config)# interface fc1/3 MDS1(config-if)# fcsp off

MDS2

MDS2(config-if)# int fc1/3 MDS2(config-if)# fcsp off



MDS2(config-if)#

And finally we configure authentication on the last link between the switches. One side can be configured passive and one side as active. They will still come online when neither link authenticates, but we do want authentication to happen. Therefore we need at least one end to be active.

MDS1

MDS1(config)# interface fc1/4 MDS1(config-if)# fcsp auto-active

MDS2

MDS2(config-if)# int fc1/4 MDS2(config-if)# fcsp auto-passive MDS2(config-if)#

FC-SP is always configured under a port-channel member interface and never on the port-channel itself!

Other places where FC-SP can be configured is on FCIP interfaces. FC-SP is never VSAN dependent.

Task 6: Advanced Features The final task of this chapter is regarding some advanced Fibre Channel features. The CCIE Data Center test is not made to test on a lot of advanced Fibre Channel MDS related features, but still you should be aware with some of the core features.

The initial task is about configuring the distribution of features between switches. This distribution is done using a Cisco proprietary protocol called Cisco Fabric Services (CFS), which has been talked about earlier. It’s possible with CFS to configure so-called ‘regions’. The regions that are configured mean that you can create some MDS switches that have different configuration sets than other MDS switches.

This is done by configuring the features in their own matching region number. By default every feature resides in region 1. Up to 200 different regions can be configured.

In this case we want MDS2 to have it’s own call-home features. We can easily accomplish this by configuring the call-home feature in a different region. When we then configure call-home (or any other feature that uses CFS) on MDS1 and the changes are committed, they are not accepted on MDS2 as a valid configuration and not inserted into the configuration.



The configuration for this feature is relatively easy, but it could introduce a lot of complexity in the MDS configuration.

MDS2

MDS2(config)# cfs region 100 MDS2(config-cfs-region)# callhome WARNING: If an Application is moved/assigned to a new region, its scope is restricted to that region and it ignores all other regions for distribution or merge. Are you sure? (y/n) [n] y

The next feature that we need to configure are 2 things at the same time. The MDS switches can create a list of SCSI targets that you can use in the entire FC fabric. This list can then be handed to the Storage administrators to demonstrate that the FC network is working fine.

The next feature where we want to use this SCSI target list is a Job Scheduler. The job scheduler is a unique feature of the NX-OS where you can schedule all CLI configurations to be done either at a given date and time or by configuring it on a repetitive basis, like in our case where we need to configure the scheduler to run every 24 hours.

The configuration needs to get used to, but there are no complex technologies involved in this case.

The initial result of the SCSI discovery is shown below.

MDS1# discover scsi-target local os all discovery started MDS1# show scsi-target devices -------------------------------------------------------------------------------- VSAN FCID PWWN VENDOR MODEL REV -------------------------------------------------------------------------------- 10 0xc500e8 c2:10:00:11:0d:0b:fa:ce SANBlaze VLUN FC RAMDisk 4.0 10 0xc500ef c2:10:00:11:0d:01:01:11 DGC RAID 5 4.0 10 0xc501e8 c2:11:00:11:0d:1b:fa:ce SANBlaze VLUN FC RAMDisk 4.0 10 0xc501ef c2:11:00:11:0d:02:01:11 DGC RAID 5 4.0 MDS1#

Now we want to save this to the flash disk on the MDS switch. We can do this by using a Linux like command.

MDS1# show scsi-target devices > VSAN10hosts.txt MDS1# dir 557 Oct 18 12:22:05 2012 VSAN10hosts.txt 25 Jan 28 00:41:03 2012 cpu_logfile



20676608 Oct 02 19:06:38 2012 kickstart-latest 49152 Oct 02 19:51:56 2012 lost+found/ 15668059 Jan 28 00:43:53 2012 m9000-ek9-ssi-mz.5.2.2a.bin 5596002 Jan 28 00:48:17 2012 m9000-epld-5.2.1.img 20576768 Jan 28 00:45:33 2012 m9200-s2ek9-kickstart-mz.5.2.2a.bin 20676608 Oct 02 17:07:43 2012 m9200-s2ek9-kickstart-mz.5.2.6.bin 99152951 Jan 28 00:48:01 2012 m9200-s2ek9-mz.5.2.2a.bin 18954240 Oct 02 17:11:33 2012 m9200-s2ek9-mz.5.2.6.bin 24628 Oct 18 11:06:14 2012 mts.log 101471413 Oct 02 17:13:42 2012 system-latest -* Some filesystem space might be used for internal purposes. * Usage for bootflash:// 355192832 bytes used 337154048 bytes free 692346880 bytes total MDS1# show file bootflash:VSAN10hosts.txt -------------------------------------------------------------------------------- VSAN FCID PWWN VENDOR MODEL REV -------------------------------------------------------------------------------- 10 0xc500e8 c2:10:00:11:0d:0b:fa:ce SANBlaze VLUN FC RAMDisk 4.0 10 0xc500ef c2:10:00:11:0d:01:01:11 DGC RAID 5 4.0 10 0xc501e8 c2:11:00:11:0d:1b:fa:ce SANBlaze VLUN FC RAMDisk 4.0 10 0xc501ef c2:11:00:11:0d:02:01:11 DGC RAID 5 4.0 MDS1#

Now we can schedule the task using our task scheduler.

MDS1(config)# feature scheduler MDS1(config)# scheduler job name SCSI MDS1(config-job)# discover scsi-target local os all MDS1(config-job)# show scsi-target devices > VSAN10hosts.txt MDS1(config-job)# exit MDS1(config)# scheduler schedule name SCSI_SCHED MDS1(config-schedule)# job name SCSI MDS1(config-schedule)# time ? daily Specify a daily schedule monthly Specify a monthly schedule start Specify a future time schedule weekly Specify a weekly schedule MDS1(config-schedule)# time daily 00:00 MDS1(config-schedule)# exit

To verify we can check if the schedule is configured correctly and if the commands that we configured under the job are the correct commands.



MDS1(config)# do sh scheduler job Job Name: SCSI -------------- discover scsi-target local os all show scsi-target devices > VSAN10hosts.txt ============================================================================== MDS1(config)# do sh scheduler schedule Schedule Name : SCSI_SCHED -------------------------------- User Name : admin Schedule Type : Run every day at 0 Hrs 0 Mins Last Execution Time : Yet to be executed ----------------------------------------------- Job Name Last Execution Status ----------------------------------------------- SCSI -NA- ============================================================================== MDS1(config)#

The final 2 tasks of this chapter are introducing another unique Cisco MDS feature.

We are asked to create failovers between devices where the end-user can never notice any changes in the Fibre Channel network.

The feature that supports this is SAN Device Virtualization (SDV). This feature is a hidden command to enable since NX-OS 5.x, but it could very well still be an option to configure on the CCIE Data Center test.

Once the feature is enabled, all commands are still available like they used to be.

We can create a virtual Fibre Channel device on our network that in the background has 2 devices connected to it for virtualization purposes. We can configure automatic failover and fallback for this physical device.

The virtual device is taken into the zoning. So we should create a zone for this use so the users in VSAN 10 can communicate to it.

We configure SDV with the following configuration.

MDS1

MDS1(config)# sdv virtual-device name JBOD1 vsan 10 MDS1(config-sdv-virt-dev)# device-alias J1D1 primary MDS1(config-sdv-virt-dev)# device-alias J2D1 MDS1(config-sdv-virt-dev)# attribute ? failover Configure the failover attributes MDS1(config-sdv-virt-dev)# attribute failover ? auto Enables Auto-failover for a virtual-device



MDS1(config-sdv-virt-dev)# attribute failover auto ? <CR> fallback Enables Switchback with Auto-failover MDS1(config-sdv-virt-dev)# attribute failover auto fallback

One important note is the attribute command where we configure the actual automatic failover and fallback. By default the SDV device will keep pointing to the PWWN of the primary device, unless the link command is used to change this. We can make this process automatic, by configuring the automatic failover and fallback parameters.

When the SDV configuration is committed, a device-alias is automatically created for use within the zoning.

MDS1(config)# sdv commit vsan 10 MDS1(config)# sh device-alias data device-alias name J1D1 pwwn c6:83:00:11:0d:18:fa:ce device-alias name J1D2 pwwn c6:83:00:11:0d:18:fb:ce device-alias name J1D3 pwwn c6:83:00:11:0d:18:fc:ce device-alias name J1D4 pwwn c6:83:00:11:0d:18:fd:ce device-alias name J2D1 pwwn 21:00:00:11:0d:18:fe:ce device-alias name J2D2 pwwn 21:00:00:11:0d:18:ff:ce device-alias name J2D3 pwwn 21:00:00:11:0d:18:a0:ce device-alias name J2D4 pwwn 21:00:00:11:0d:18:a1:ce device-alias name JBOD1 pwwn 50:00:53:00:01:03:d0:0a Total number of entries = 9 MDS1(config)#

You successfully completed chapter 5 regarding Cisco MDS Fibre Channel features. The next chapter will focus on Fibre Channel extension over IP connections!

Do NOT erase configurations on the MDS switches before moving on to the next Chapter!



Chapter 6: Data Center Storage Networking Extension

Chapter 6: Data Center Storage networking Extension is intended to let you be familiar with the Storage Networking features on the Cisco MDS switches. This chapter will be about configuring IP features like iSCSI, iSLB and FCIP including the relevant Security features for Fibre Channel extension across IP connections. We highly recommend to create your own diagram at the beginning of each lab so you are able to draw on your own diagram, making it much easier when you step into the real lab. Multiple topology drawings are available for this chapter.



Solutions

Task 1: Initial set-up

In this chapter we will be configuring several IP features related to the Cisco MDS platform. The MDS platform has some unique IP features that no other Fibre Channel switch vendor has. Initially Cisco offered several line-cards in the chassis based MDS switches, which supported Ethernet (1Gbps) interfaces which have these features. In the current generation platforms (MDS9222i), these interfaces are integrated in the switch.

On the IP features the MDS platform has 2 strong and prominent features, which are tested, in the CCIE Data Center:

Fibre Channel over IP (FCIP) tunnels

With FCIP tunnels you are able to extend the Fibre Channel network across a normal Layer 3 IP network. This overcomes the distance limitations that you have when expanding Fibre Channel between data centers, where you could easily run into issues with B2B credits. Besides that you need a Dark Fiber or DWDM transparent system to transport Fibre Channel. FCIP overcomes these limitations and narrows down your requirements for transporting FC between Data Centers to just an IP connection

iSCSI

The iSCSI feature on the Cisco MDS platform is not widely used, but still a great feature and most important of all tested on the CCIE Data Center lab exam. This feature makes it able for you to present a Fibre Channel disk to an non-FC host. Meaning that the Server can talk over it’s IP connection with the iSCSI protocol to the MDS switch which will present itself as an iSCSI LUN, while in the background you are actually talking to a Fibre Channel disk.

The iSLB feature is a extension of the iSCSI feature where redundancy, distribution and load balancing are integrated. You could say iSLB is actually iSCSI 2.0 on the MDS platform

Be aware that you need the SAN Extension over IP license on the MDS switches to enable FCIP. For iSCSI you do not require a license. On the MDS 9222i the SAN Extension license is included!

For this task we will be required to use a Ethernet switch to transport our traffic between our MDS switches. Our physical topology shows that we have our MDS IP ports connected to the Nexus 5500 switches. This means that we first need to ensure that we transport several VLANs between the switches.



We will be using several VLANs during this chapter, which is not strictly necessary, but do let you practice with the 802.1Q features of the IP interfaces on the line cards. The IP interfaces of the MDS switch are different than a standard router or switch interface. Keep in mind that these interfaces are actually just ‘host’ interfaces that let you connect a host device to it. This means that you can just configure an IP address in the same subnet on multiple interfaces. As they are just hosts on the network and will never start routing between these ports.

The MDS also support port-channeling of these interfaces in standard Ethernet port-channels. In this chapter we will not be configuring Ethernet Port-channels on the MDS switch, but configuration is equal to how that is done on the Nexus switches.

Drawing 1 shows you the physical topology we are using in this chapter.

In the first task we are required to ensure that the MDS switches can communicate over a number of VLANs. You might still have configuration from a previous task on the Nexus switches, but this is not conflicting with this lab.

The first task is to configure the VLANs and let the MDS switches have access to it. This means we need to make sure that both GigabitEthernet interfaces on the MDS switches are configured as Trunk interfaces on the Nexus 5500s.

Configuration is a piece of cake, as we have already finished extensive chapters regarding the configuration of NX-OS with Ethernet and IP features.



Configuration in this case will only comprise of the VLANs and Trunk configurations. We do not need to configure IP routing, where this would be an option for FCIP and iSCSI as well. In this case it would not add a lot to our complexity and we just want to separate traffic across different VLANs.

We use Drawing 2 to determine the VLANs that we need.

In this drawing we can see that we should be using VLAN 69 through 72 for the IP features of the MDS switches. In this chapter we will not be using any ISL links between the MDS switches. We have pre-configuration from the previous chapter on there, but this is not conflicting with the tasks that we are about to configure.

We re-configure the interfaces between the MDS switches as we require some other traffic than FabricPath from the last Chapter where we configured the Nexus switches.

SW2

SW2(config)# vlan 69,70,71,72 SW2(config-vlan)# exit SW2(config)# interface e1/4 SW2(config-if)# switchport mode trunk SW2(config-if)# no shutdown SW2(config-if)# exit SW2(config)# default interface e1/9 SW2(config)# interface e1/9 SW2(config-if)# switchport mode trunk SW2(config-if)# switchport trunk allowed vlan 69-72 SW2(config-if)# no shutdown



SW2(config-if)# exit SW2(config)# default interface e1/10 SW2(config)# interface e1/10 SW2(config-if)# switchport mode trunk SW2(config-if)# switchport trunk allowed vlan 69-72 SW2(config-if)# no shutdown SW2(config-if)# exit

SW3

SW3(config)# vlan 69,70,71,72 SW3(config-vlan)# exit SW3(config)# interface e1/4 SW3(config-if)# switchport mode trunk SW3(config-if)# no shutdown SW3(config-if)# exit SW3(config)# default interface e1/9 SW3(config)# interface e1/9 SW3(config-if)# switchport mode trunk SW3(config-if)# switchport trunk allowed vlan 69-72 SW3(config-if)# no shutdown SW3(config-if)# exit SW3(config)# default interface e1/10 SW3(config)# interface e1/10 SW3(config-if)# switchport mode trunk SW3(config-if)# switchport trunk allowed vlan 69-72 SW3(config-if)# no shutdown SW3(config-if)# exit

MDS1

MDS1(config)# interface gig1/1 MDS1(config-if)# no shutdown MDS1(config-if)# exit MDS1(config)# interface gig1/2 MDS1(config-if)# no shutdown MDS1(config-if)# exit

MDS2

MDS2(config)# interface gig1/1 MDS2(config-if)# no shutdown MDS2(config-if)# exit MDS2(config)# interface gig1/2 MDS2(config-if)# no shutdown MDS2(config-if)# exit

We do not need to configure any IP addressing right now. The final statement in this task is regarding the future configurations of IP addressing, where we need to pay attention to the format that we are going to use. Drawing 2 will illustrate which host IP addresses need to be used and which VLANs belong to which feature.

So we finished the first task!



Task 2: FCIP

In the second task we will start configuring our FCIP tunnels between the switches. In this chapter you will be configuring both the FCIP tunnels and their High Availability features that are available on the Cisco MDS platform.

During your CCIE Data Center lab exam you might see questions about the expansion of your Fibre Channel network to the other Data Center in the lab topology.

Keep in mind that it’s not possible to transport FCoE across the data centers when you are using FabricPath and/or OTV. Therefore you will see MDS switches with FCIP features being used in Cisco best practice designs regarding multiple Data Centers.

The FCIP technology is a standard based solution (FC-BB-2) and compatible with other vendors FC switches.

The goal of the technology is to extend the Fibre Channel fabric across geographically dispersed locations.

The only traffic that you are able to transport is E-port type traffic, meaning you can only transport ISL traffic. The Virtual E-port (VE) that is created has the same features as a normal E-port. Meaning that you can also enable the Trunking protocol to transport multiple VSANs.

The FCIP interface is the same interface as a normal FC interface on the MDS switch, meaning that you have an equal amount of configuration commands available.

The drawing below illustrates the positioning of the FCIP tunnel in the Fibre Channel fabric.



The Host and the JBOD in this drawing do not know that they are interconnected through a Layer 3 IP network as they just talk plain Fibre Channel throughout the FC fabric. All 4 FC switches are participating in the same FC Domain and FSPF network. Traffic is routed in the same way, except that the traffic gets an IP encapsulation instead of an FC encapsulation between the data centers.

The FCIP tunnel consists of 2 TCP sessions:

Control

The control session is used for all Fibre Channel control packets. This means that all Class F fibre channel frames are transmitted across this TCP session. This ensures that control traffic can receive a different QoS treatment throughout the IP network to ensure a low latency transmission of all control frames.

Data

The second TCP session is used for all other data (Class 1, 2, 3) frames that are sent across the FC fabric. These data packets can get another QoS marking so they fall in different classes when required.

You can choose to have only 1 TCP session for the FCIP connection. By default there will be 2 sessions.

Configuration of the FCIP connection consists of 2 sections:

FCIP Profile

The FCIP profile configured the TCP session parameters and the local interface parameters. This configures the source interface of the FCIP tunnel. A single profile can be used by multiple tunnels.

FCIP Interface

The FCIP interface configurations is the actual tunnel interface. On this interface all Fibre Channel related features are configured, a profile is attached, the peer IP address is configured and the number of TCP sessions used for this tunnel.

To create a FCIP interface you need to have at least 1 differentiation out of these 4 parameters:

• Local IP address

• Local Listener Port

• Peer IP address

• Peer port



You can not create multiple interfaces where all these parameters are equal, as that would result in the same interface. When one of these 4 is different, the interface can come up, but it might require some engineering regarding the initiation of the interface.

Our first question of the task is to configure a FCIP connection between the 2 MDS switches. We will transport only 2 already active VSANs across this interface. The FCIP tunnel is using only a single interface and using only 1 TCP session between the switches.

We can not use the default TCP port, which is TCP 3225, so we need to change this as well. As this is a TCP session parameter we need to configure this under the profile configuration.

MDS1

MDS1(config-if)# interface gig1/1.69 MDS1(config-if)# ip address 198.18.69.9 255.255.255.0 MDS1(config-if)# no shutdown MDS1(config-if)# MDS1(config-if)# feature fcip MDS1(config)# fcip profile 1 MDS1(config-profile)# ip address 198.18.69.9 MDS1(config-profile)# port ? <1-65535> Profile TCP Port MDS1(config-profile)# port 3226 MDS1(config-profile)# tcp ? cwm Enable congestion window monitoring keepalive-timeout Set keep alive timeout in sec max-bandwidth-kbps Configure maximum available path bandwidth in Kbps max-bandwidth-mbps Configure maximum available path bandwidth in Mbps max-retransmissions Maximum number of retransmissions min-retransmit-time Set minimum retransmit time in millisec pmtu-enable Enable PMTU Discovery sack-enable Enable SACK option for TCP send-buffer-size Send buffer size in KBytes MDS1(config-profile)#

The profile configuration now has the source IP address configured and the non-default port that we are required to use. Other TCP parameters that can be set are traffic-shaping parameters and some TCP features. All the advanced TCP features like Path MTU Discovery (PMTU) and Selective Acknowledgements (SACK) are by default enabled.



The interface that we configure has a sub-interface number of 69. This number is equal to the VLAN number, because this is immediately the VLAN configuration. There is no encapsulation command like on normal router/switch interfaces. So keep in mind which sub-interface number you use for this case as it’s crucial for the VLAN numbering.

The amount of TCP connections is still supported to be configured, except that it’s not available through the question mark anymore. Therefore you should know that this command is configured under the FCIP interface with the tcp-connections command. You may specify a value of 1 or 2 behind it, where 2 is the default.

Now the FCIP interface should be configured.

MDS1

MDS1(config-profile)# interface fcip1 MDS1(config-if)# use-profile 1 MDS1(config-if)# peer-info ipaddr 198.18.69.11 port 3226 MDS1(config-if)# tcp-connections 1 MDS1(config-if)# switchport ? description Enter description of maximum 80 characters fcrxbufsize Configure receive data field size for the port mode Enter the port mode trunk Configure trunking parameters on an interface MDS1(config-if)# switchport mode ? E E port mode auto Autosense mode MDS1(config-if)# switchport mode e MDS1(config-if)# sw trunk mode on MDS1(config-if)# sw trunk allowed vsan 10 MDS1(config-if)# sw trunk allowed vsan add 20 MDS1(config-if)# ? channel-group Add to/remove from a port-channel fcdomain Configure fcdomain parameters ficon-tape-accelerator Configure the FCIP FICON tape accelerator ficon-tape-read-accelerator Configure FCIP FICON Read accelerator ficon-xrc-accelerator Configure FCIP FICON XRC accelerator ficon-xrc-logging Configure Ficon XRC Logging fspf Configure FSPF parameters ip-compression Configure the IP compression for the FCIP tunnel link-state-trap Enable/disable link state change traps no Negate a command or set its defaults passive-mode Fcip set passive connect mode peer-info Configure fcip peer information qos Configure qos for interface shutdown Enable/disable an interface switchport Configure switchport parameters use-profile Fcip use profile for tunnel write-accelerator Configure the FCIP write accelerator end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in



MDS1(config-if)# no shut MDS1(config-if)# vsan database MDS1(config-vsan-db)# vsan 10 interface fcip1

The final configuration that is done in this case, adding the FCIP interface to a VSAN in the VSAN database, has to do with the fact that one of the tasks in the previous chapter was to deactivate the Default VSAN (VSAN 1). By making it suspended, every interface that has it’s registration in VSAN 1 will stay down unless something else is configured. In the case of Trunk interfaces, the port first needs to come online before the trunking is negotiated with the other side. When the trunk port is configured in the Default VSAN it will first come online in this VSAN before it will start trunking. Therefore we need to configure the FCIP interface, as it’s treated as just another FC interface, inside the VSAN database into another VSAN. The VSAN that you configure the tunnel in, is not that significant, except that it should be anything else than VSAN 1.

The FCIP connection is now configured on one side. There is one additional requirement for this initial few questions of this task, which is that MDS1 should always initiate the connection.

This is achieved using the passive-mode command, where the switch where it’s configured will never initiate the connection and will always wait on the other side for setting up the tunnel.

This passive-mode is required when you are configuring multiple FCIP tunnels that each should terminate on the same FCIP profile, meaning you have only 1 destination IP/port for the remote side. When this switch with only 1 listening port, would initiate the connections there is no problem as the switch will initiate the connection correctly to the right switch. When it receives traffic on the FCIP port, it will not know to which FCIP tunnel the traffic actually belongs.

In this case it has no special use, except that you know how to configure this.

What remains is to configure the other side.

MDS2

MDS2(config-if)# interface gig1/1.69 MDS2(config-if)# ip address 198.18.69.11 255.255.255.0 MDS2(config-if)# no shutdown MDS2(config-if)# MDS2(config-if)# feature fcip MDS2(config)# fcip profile 1 MDS2(config-profile)# ip address 198.18.69.11 MDS2(config-profile)# port 3226 MDS2(config-profile)# interface fcip1 MDS2(config-if)# use-profile 1 MDS2(config-if)# peer-info ipaddr 198.18.69.9 port 3226 MDS2(config-if)# tcp-connections 1 MDS2(config-if)# switchport mode e MDS2(config-if)# sw trunk mode on MDS2(config-if)# sw trunk allowed vsan 10 MDS2(config-if)# sw trunk allowed vsan add 20 MDS2(config-if)# passive-mode



MDS2(config-if)# no shut MDS2(config-if)# vsan database MDS2(config-vsan-db)# vsan 10 interface fcip1

Now we need to verify our configuration.

First we verify our FCIP profile configuration.

MDS1(config-if)# sh fcip profile all FCIP Profile 1 Internet Address is 198.18.69.6 (interface GigabitEthernet1/1.69) Tunnels Using this Profile: fcip1 Listen Port is 3226 TCP parameters SACK is enabled PMTU discovery is enabled, reset timeout is 3600 sec Keep alive is 60 sec Minimum retransmission timeout is 200 ms Maximum number of re-transmissions is 4 Send buffer size is 0 KB Maximum allowed bandwidth is 1000000 kbps Minimum available bandwidth is 500000 kbps Configured round trip time is 1000 usec Congestion window monitoring is enabled, burst size is 50 KB Auto jitter detection is enabled MDS1(config-if)#

As mentioned the advanced features like SACK and PMTU are by default enabled. The best command to verify the working of all your FCIP interfaces is the following command. It shows you a summary of all important parameters of the interfaces.

MDS1(config-if)# show fcip summary ------------------------------------------------------------------------------- Tun prof Eth-if peer-ip Status T W T Enc Comp Bandwidth rtt E A A max/min (us) ------------------------------------------------------------------------------- 1 1 GE1/1.69 198.18.69.11 UP Y N N N N 1000M/500M 1000 MDS1(config-if)#

The detailed information for each FCIP tunnel is found in a normal show interface.

MDS1(config-if)# show interface fcip1 fcip1 is up (Trunking) Hardware is GigabitEthernet Port WWN is 20:14:00:0d:ec:54:a9:40 Admin port mode is E, trunk mode is on snmp link state traps are enabled Port vsan is 1 Using Profile id 1 (interface GigabitEthernet1/1.69) Peer Information Peer Internet address is 198.18.69.11 and port is 3226 Write acceleration mode is configured off Tape acceleration mode is configured off Tape Accelerator flow control buffer size is automatic FICON XRC Accelerator is configured off Ficon Tape acceleration configured off for all vsans IP Compression is disabled Maximum number of TCP connections is 1



QOS control code point is 0 QOS data code point is 0 TCP Connection Information 1 Active TCP connections Local 198.18.69.9:65523, Remote 198.18.69.11:3226 0 host table full 0 target entries in use 6 Attempts for active connections, 3 close of connections TCP Parameters Path MTU 1500 bytes Current retransmission timeout is 200 ms Round trip time: Smoothed 8 ms, Variance: 4 Jitter: 157 us Advertized window: Current: 34 KB, Maximum: 24580 KB, Scale: 5 Peer receive window: Current: 2047 KB, Maximum: 2047 KB, Scale: 5 Congestion window: Current: 14 KB, Slow start threshold: 112 KB Current Send Buffer Size: 34 KB, Requested Send Buffer Size: 0 KB CWM Burst Size: 50 KB Measured RTT : 115 us Min RTT: 500000 us Max RTT: 143 us 5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 18 frames input, 2736 bytes 18 Class F frames input, 2736 bytes 0 Class 2/3 frames input, 0 bytes 0 Reass frames 0 Error frames timestamp error 0 23 frames output, 2916 bytes 23 Class F frames output, 2916 bytes 0 Class 2/3 frames output, 0 bytes 0 Error frames MDS1(config-if)#

Now multiple things are shown in this output. You can see the specifics of this particular FCIP interface. This is separated in the information related to the TCP parameters of the session and the Fibre Channel parameters of this interface. The FCIP interface is just a normal FC interface when looked at from a Fabric perspective.

We can see in this output that we are indeed using 1 TCP connection and a non-default port, which is initiated by MDS1, because it uses a destination port of 3226. When the switch would’ve received the connection the source port would be 3226.

Now we have our first FCIP connection successfully up and running!



The next question in this task asks you to configure high availability for this FCIP tunnel. We can solve this using 4 solutions.

Redundant FCIP tunnel

We could create an additional FCIP tunnel where the FCIP 1 tunnel can failover to the other using a FSPF re-calculation. However we can not change the FCIP configuration for this task.

Fibre Channel port-channel

As the FCIP tunnel is treated as a regular Fibre Channel interface, it’s also possible to bundle the FCIP interfaces together in a FC port-channel which would be a seamless switch over. However we can not change the FCIP configuration for this task and we can not use port-channels.

Ethernet port-channels

We have the option to configure Gig1/1 and Gig1/2 in an Ethernet port-channel which would solve our question with a seamless failover for the single FCIP 1 connection. However we can not use port-channels for this task.

VRRP

The last and only correct option is the use of the Layer 3 redundancy protocol VRRP. This protocol is the only FHRP protocol available on the MDS switches. With the use of this protocol we can still have a single IP address for both the source interface and the peer IP address. When something happens when the first GigabitEthernet interface the VRRP configuration will issue a failover to the other. As the GigabitEthernet interfaces on the MDS switches are just treated as Host interfaces, it’s perfectly possible to run a local VRRP on the same MDS switch.

We will perform the VRRP configuration in a way that we do not need to change the FCIP configuration as we are not allowed to do this according to the question. This means that we need to re-use our interface IP addressing in the VRRP group configuration.

When we need to create additional IP addresses in the subnet this would be perfectly allowed according to the current question requirements!

Let’s start by configuring the VRRP group using the correct IP addressing so we do not need to change the FCIP interfaces for this change.

MDS1

MDS1(config-if)# interface gig1/1.69 MDS1(config-if)# ip address 198.18.69.99 255.255.255.0 MDS1(config-if)# vrrp 69 MDS1(config-if-vrrp)# address 198.18.69.9 MDS1(config-if-vrrp)# priority 120 MDS1(config-if-vrrp)# no shutdown MDS1(config-if-vrrp)# exit



MDS1(config-if)# interface gig1/2.69 MDS1(config-if)# ip address 198.18.69.199 255.255.255.0 MDS1(config-if)# no shutdown MDS1(config-if)# vrrp 69 MDS1(config-if-vrrp)# address 198.18.69.9 MDS1(config-if-vrrp)# no shutdown MDS1(config-if-vrrp)# exit

MDS2

MDS2(config-if)# interface gig1/1.69 MDS2(config-if)# ip address 198.18.69.111 255.255.255.0 MDS2(config-if)# vrrp 96 MDS2(config-if-vrrp)# address 198.18.69.11 MDS2(config-if-vrrp)# priority 120 MDS2(config-if-vrrp)# no shutdown MDS2(config-if-vrrp)# exit MDS2(config-if)# interface gig1/2.69 MDS2(config-if)# ip address 198.18.69.112 255.255.255.0 MDS2(config-if)# no shutdown MDS2(config-if)# vrrp 96 MDS2(config-if-vrrp)# address 198.18.69.11 MDS2(config-if-vrrp)# no shutdown MDS2(config-if-vrrp)# exit

Now with these VRRP groups configured we still have an active FCIP tunnel between the originally configured endpoints as our endpoints didn’t change. We did have to configure additional IP addresses, but there is no statement in the question that prohibits you from using additional IP addresses in the subnet to accomplish the question.

MDS1

MDS1(config-if-vrrp)# sh vrrp Interface VR IpVersion Pri Time Pre State VR IP addr --------------------------------------------------------------------------- GigE1/1.69 69 IPv4 120 1 s master 198.18.69.9 GigE1/2.69 69 IPv4 100 1 s backup 198.18.69.9 MDS1(config-if-vrrp)#

MDS2

MDS2(config-if-vrrp)# sh vrrp Interface VR IpVersion Pri Time Pre State VR IP addr --------------------------------------------------------------------------- GigE1/1.69 96 IPv4 120 1 s master 198.18.69.11 GigE1/2.69 96 IPv4 100 1 s backup 198.18.69.11 MDS2(config-if-vrrp)#



Keep the VRRP group numbers different on the 2 MDS switches. You are configuring 2 VRRP groups in the same VLAN. Remember that the group number is used in the virtual MAC address of this virtual default gateway. When you configure equal group numbers in the same VLAN, this will cause serious issues and an unworkable situation. The VRRP configuration on both DMS switches should work fully independent of each other. Therefore you will need to use different group numbers on both MDS switches.

We see that the local MDS switch is backup for itself. Therefore when something happens with GigabitEthernet1/1 the second interface will become the master and the FCIP connection remains up!

The next task will create the second FCIP connection between the MDS switches by using another interface. Again we need to take care of a few specifics in this FCIP connection. When you read ahead on the next number of questions you will see some specifics that need to be applied to this connection.

For example, we need to ensure that this FCIP 2 connection receives a higher QoS priority. Well we just saw in the verification of FCIP 1 that the default QoS marking is 0. So anything higher than this is good for a better treatment in the IP network, when properly configured.

The second point is that the tunnel must be brought down when there are no packets received for 90 seconds. This feature is the ‘tcp keepalive-timeout’ configured in the FCIP profile, as it’s a TCP parameter.

MDS1

MDS1(config-if)# interface gig1/2.70 MDS1(config-if)# ip address 198.18.70.9 255.255.255.0 MDS1(config-if)# no shutdown MDS1(config-if)# MDS1(config-if)# fcip profile 2 MDS1(config-profile)# ip add 198.18.70.9 MDS1(config-profile)# tcp ? cwm Enable congestion window monitoring keepalive-timeout Set keep alive timeout in sec max-bandwidth-kbps Configure maximum available path bandwidth in Kbps max-bandwidth-mbps Configure maximum available path bandwidth in Mbps max-retransmissions Maximum number of retransmissions min-retransmit-time Set minimum retransmit time in millisec pmtu-enable Enable PMTU Discovery sack-enable Enable SACK option for TCP send-buffer-size Send buffer size in KBytes MDS1(config-profile)# tcp keepalive-timeout ? <1-7200> TCP Keep Alive timeout value MDS1(config-profile)# tcp keepalive-timeout 90 MDS1(config-profile)# ? ip Config ip to profile no Negate a command or set its defaults port Config local port to profile tcp Config TCP Parameters for the Profile end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name



push Push current mode to stack or save it under name where Shows the cli context you are in MDS1(config-profile)# exit MDS1(config)# interface fcip 2 MDS1(config-if)# use-profile 2 MDS1(config-if)# peer ip 198.18.70.11 MDS1(config-if)# qos ? control Configure qos for control and data packets MDS1(config-if)# qos control ? <0-63> QOS control code point value MDS1(config-if)# qos control 15 ? data Configure qos for control and data packets MDS1(config-if)# qos control 15 data 10 MDS1(config-if)# sw mod e MDS1(config-if)# sw trunk allowed vsan 10 MDS1(config-if)# sw trunk allowed vsan add 20 MDS1(config-if)# sw trunk allowed vsan add 50 MDS1(config-if)# vsan database MDS1(config-vsan-db)# vsan 10 interface fcip2

We first create a new sub-interface on Gig1/2 with the new VLAN. We could have configured this FCIP connection in the same VLAN as we are using different port numbers in the first VLAN. So using the default port numbers would generate enough separation in the profile, as discussed at the start of this task, to create an additional FCIP connection in the same VLAN.

In this case we create a new VLAN with different IP addressing. In the FCIP profile we configure the keepalive-timeout to be 90 seconds, so the tunnel is brought down after this time out expires.

Then under the FCIP interface we configure the QoS values for this interface. The requirement is that it should be higher than the FCIP 1 interface. In this example you see that it’s possible to create a different QoS mapping for both the Control session and the Data session. Requirement for this to work is that you always use the 2 TCP connection configuration, which is the default.

Finally we need to add this second FCIP connection toe the VSAN database as our Default VSAN 1 is suspended, which means the interface will not come online when it is configured in that VSAN.

Now we need to bring the connection up on the second MDS.

MDS2

MDS2(config-if)# interface gig1/2.70 MDS2(config-if)# ip address 198.18.70.11 255.255.255.0 MDS2(config-if)# no shutdown MDS2(config-if)# MDS2(config-if)# fcip profile 2 MDS2(config-profile)# ip add 198.18.70.11 MDS2(config-profile)# tcp keepalive-timeout 90 MDS2(config-profile)# exit MDS2(config)# interface fcip 2



MDS2(config-if)# use-profile 2 MDS2(config-if)# peer ip 198.18.70.11 MDS2(config-if)# qos control 15 data 10 MDS2(config-if)# sw mod e MDS2(config-if)# sw trunk allowed vsan 10 MDS2(config-if)# sw trunk allowed vsan add 20 MDS2(config-if)# sw trunk allowed vsan add 50 MDS2(config-if)# vsan database MDS2(config-vsan-db)# vsan 10 interface fcip2

When both FCIP 1 and FCIP 2 are up and running we can configure the load balancing parameters that we need to comply with.

We need to ensure that VSAN 10 traffic uses one link and VSAN 20 the other link. From the standpoint of MDS2 this should be vice versa. This means that on MDS2 the traffic for VSAN 10 should use the FCIP 2 connection and VSAN 20 traffic should be using the FCIP 1 connection.

This means that we will be using the FSPF costs to alter the behavior of where traffic uses which link. The FCIP interfaces are by default seen as 1Gbps links. We need to use these links as the primary links for VSAN 10 and 20, as this is stated by the questions. Keep in mind to lower the costs to ensure that this happens. As said the FCIP links are seen as 1Gbps links, meaning they have a default FSPF cost of 1000.

MDS1

MDS1(config-if)# interface fcip1 MDS1(config-if)# fspf cost 10 vsan 10 MDS1(config-if)# fspf cost 20 vsan 20 MDS1(config-if)# interface fcip2 MDS1(config-if)# fspf cost 20 vsan 10 MDS1(config-if)# fspf cost 10 vsan 20

MDS2

MDS2(config-if)# interface fcip1 MDS2(config-if)# fspf cost 20 vsan 10 MDS2(config-if)# fspf cost 10 vsan 20 MDS2(config-if)# interface fcip2 MDS2(config-if)# fspf cost 10 vsan 10 MDS2(config-if)# fspf cost 20 vsan 20

With these changes the traffic will now flow across the correct FCIP links.

The next question in this task is regarding compression. FCIP connections can compress their data to ensure a more optimal way of sending traffic across the links.



In the past there where 3 modes of compression. This has been changed to only 2 modes:

Auto mode

This is enabled by default and will compress files when necessary and when the other side of the FCIP connection negotiates compression as well.

Mode 1

This mode is used for higher bandwidth links, so less compression. Don’t let yourself go blind on the description you get from the CLI. As that will tell you that mode 1 is High compression, but they actually mean it’s High Speed compression compared to Mode 2.

Mode 2

This mode enables a higher compression rate, but introduces latency due to the compression algorithm that takes time to run.

As we want the highest compression as possible in this question we need to configure Mode 2 on both sides. Don’t let yourself be confused with the Mode 1 CLI comment!

MDS1

MDS1(config-if)# interface fcip2 MDS1(config-if)# ip-compression ? <CR> auto Auto compression setting mode1 High compression mode2 Moderate compression for medium bandwidth links MDS1(config-if)# ip-compression mode2 MDS1(config-if)#

MDS2

MDS2(config-if)# interface fcip2 MDS2(config-if)# ip-compression mode2 MDS2(config-if)#

The compression modes need to match on both sides otherwise you could run into issues with the FCIP tunnels. Auto mode will solve all these issues.

The next question in the task wants that FCIP 1 sends R_RDY messages locally. Which means that the messages are sent by the local MDS where they should actually be sent by the remote host. When a host receives a confirmation a packet is arrived it will close the write action on it’s disk and assumes everything is fine. Now this responsibility comes in the hands of the MDS switch.



If the MDS switches will send confirmations for write actions this means that the disks will respond faster and will finish their actions faster.

This technology is called ‘write-accelerator’. It can be configured for both normals disks and tapes. For tape traffic a little different style of acknowledgements is sent and received, so this is adapted for by the MDS switch.

It’s easy to enable by using a single command. Ensure you have enabled this on both sides of the connection, otherwise traffic could behave strangely.

MDS1

MDS1(config-if)# interface fcip1 MDS1(config-if)# write-accelerator

MDS2

MDS2(config-if)# interface fcip1 MDS2(config-if)# write-accelerator

The final question of this task asks to configure high availability for connection FCIP 2. We should do this using a third FCIP connection and we can not use Ethernet port-channels for the high availability.

This leaves only one option which is configuring a Fibre Channel based port-channel configuration with FCIP connections.

We need to ensure that the tasks that we configured before this will be kept in the port-channel configuration so we don’t loose any points there.

The task mentions that we should keep high availability in mind when we are configuring this question. To ensure we have the most resiliency on this third FCIP connection we should use a different Ethernet interface as well, which we can easily do.

We can use the GigabitEthernet1/1 connection in VLAN 70, as we haven’t configured that interface yet. We will need to use different IP addressing in this case as we share the same VLAN. The question does not prohibit the creation of additional IP addresses in the subnet, so we can do this.

Because we are creating a Fibre Channel port-channel the FSPF protocol does not notice the change when one interface is lost in the port-channel FSPF costs will need to remain the same for the configured VSANs in the previous question.

MDS1

MDS1(config-if)# interface gig1/1.70 MDS1(config-if)# ip address 198.18.70.99 255.255.255.0 MDS1(config-if)# no shutdown MDS1(config-if)# MDS1(config-if)# fcip profile 3 MDS1(config-profile)# ip add 198.18.70.99 MDS1(config-profile)# tcp keepalive-timeout 90



MDS1(config-profile)# exit MDS1(config)# interface fcip 3 MDS1(config-if)# use-profile 3 MDS1(config-if)# peer ip 198.18.70.111 MDS1(config-if)# qos control 15 data 10 MDS1(config-if)# sw mod e MDS1(config-if)# sw trunk allowed vsan 10 MDS1(config-if)# sw trunk allowed vsan add 20 MDS1(config-if)# sw trunk allowed vsan add 50 MDS1(config-if)# MDS1(config-if)# vsan database MDS1(config-vsan-db)# vsan 10 interface fcip3

MDS2

MDS2(config-if)# interface gig1/1.70 MDS2(config-if)# ip address 198.18.70.111 255.255.255.0 MDS2(config-if)# no shutdown MDS2(config-if)# MDS2(config-if)# fcip profile 3 MDS2(config-profile)# ip add 198.18.70.111 MDS2(config-profile)# tcp keepalive-timeout 90 MDS2(config-profile)# exit MDS2(config)# interface fcip 3 MDS2(config-if)# use-profile 3 MDS2(config-if)# peer ip 198.18.70.99 MDS2(config-if)# qos control 15 data 10 MDS2(config-if)# sw mod e MDS2(config-if)# sw trunk allowed vsan 10 MDS2(config-if)# sw trunk allowed vsan add 20 MDS2(config-if)# sw trunk allowed vsan add 50 MDS2(config-if)# MDS2(config-if)# vsan database MDS2(config-vsan-db)# vsan 10 interface fcip3

Now we configured the third FCIP connection across the other GigabitEthernet interface so we have high availability for those 2 FCIP connections.

To finish we need to force the tunnels to form a port-channel. We need to ensure we keep the configuration of FCIP 2 leading for the port-channel.

MDS1

MDS1(config-if)# interface fcip2 MDS1(config-if)# channel-group 99 MDS1(config-if)# no shutdown MDS1(config-if)# interface fcip3 MDS1(config-if)# channel-group 99 MDS1(config-if)# no shutdown MDS1(config-if)# MDS1(config-if)# interface port-channel 99 MDS1(config-if)# sw mod e MDS1(config-if)# sw trunk allowed vsan 10 MDS1(config-if)# sw trunk allowed vsan add 20 MDS1(config-if)# sw trunk allowed vsan add 50 MDS1(config-if)# ip-compression mode2 MDS1(config-if)# fspf cost 20 vsan 10 MDS1(config-if)# fspf cost 10 vsan 20 MDS1(config-if)# no shutdown MDS1(config-if)# vsan database MDS1(config-vsan-db)# vsan 10 interface port-channel99



MDS2

MDS2(config-if)# interface fcip2 MDS2(config-if)# channel-group 99 MDS2(config-if)# no shutdown MDS2(config-if)# interface fcip3 MDS2(config-if)# channel-group 99 MDS2(config-if)# no shutdown MDS2(config-if)# MDS2(config-if)# interface port-channel 99 MDS2(config-if)# sw mod e MDS2(config-if)# sw trunk allowed vsan 10 MDS2(config-if)# sw trunk allowed vsan add 20 MDS2(config-if)# sw trunk allowed vsan add 50 MDS2(config-if)# ip-compression mode2 MDS2(config-if)# fspf cost 10 vsan 10 MDS2(config-if)# fspf cost 20 vsan 20 MDS2(config-if)# no shutdown MDS2(config-if)# vsan database MDS2(config-vsan-db)# vsan 10 interface port-channel99

Now the port-channel has formed and we have 2 members configured in this port-channel. It transports all the VSANs that FCIP 2 does and it has the same interface properties like FSPF costs for the VSANs and we have configured IP-Compression like we did on FCIP 2.

Finally don’t forget to add this connection to the VSAN database as well, otherwise it will remain down.

We now have a high available FC port-channel connection for FCIP!

Task 3: FCIP Security

The third task is regarding the security of our FCIP connections. We will be configuring 2 different security mechanisms in this task. The second mechanism is used widely in the world for connections between different networks. Therefore it’s not strange that this technology is also found in the MDS switch to protect the IP traffic that it needs to send to different data centers in the world. This technology is IPsec together with IKE.

Not all MDS switches support this technology, but the ones used in the lab do. Security features are tested so be prepared for this configuration.

The first task asks us to secure the failover mechanism of FCIP 1. This means we have to protect our VRRP with a MD5 hash. We already did this once in the High Availability chapter so this is not a complex configuration.

The solution is just a few commands and we have a secure protocol between our local interfaces. Don’t forget to configure both MDS switches!

MDS1

MDS1(config-if)# interface gig1/1.69 MDS1(config-if)# vrrp 69



MDS1(config-if-vrrp)# authentication ? md5 Set the MD5 authentication key (16 char max) text Set the authentication password (8 char max) MDS1(config-if-vrrp)# authentication md5 SecureIPexpert ^ % Incomplete command at '^' marker. MDS1(config-if-vrrp)# authentication md5 SecureIPexpert ? spi Security parameter index MDS1(config-if-vrrp)# authentication md5 SecureIPexpert spi ? <0x0-0xffffffff> MDS1(config-if-vrrp)# authentication md5 SecureIPexpert spi 0x0 The vr configuration can only be updated when the administrative state is down and the operational state is initialize MDS1(config-if-vrrp)# shut MDS1(config-if-vrrp)# authentication md5 SecureIPexpert spi 0x0 MDS1(config-if-vrrp)# no shut MDS1(config-if-vrrp)# MDS1(config-if)# interface gig1/2.69 MDS1(config-if)# vrrp 69 MDS1(config-if-vrrp)# shut MDS1(config-if-vrrp)# authentication md5 SecureIPexpert spi 0x0 MDS1(config-if-vrrp)# no shut MDS1(config-if-vrrp)#

One thing different about configuration of authentication in VRRP on the MDS switches is that you need to configure a Security Parameter Index (SPI). This is a unique key for the authentication key that is communicated to the neighbor. This SPI should be unique for each VSAN!

MDS2

MDS2(config-if)# interface gig1/1.69 MDS2(config-if)# vrrp 96 MDS2(config-if-vrrp)# shut MDS2(config-if-vrrp)# authentication md5 SecureIPexpert spi 0x1 MDS2(config-if-vrrp)# no shut MDS2(config-if-vrrp)# MDS2(config-if)# interface gig1/2.69 MDS2(config-if)# vrrp 96 MDS2(config-if-vrrp)# shut MDS2(config-if-vrrp)# authentication md5 SecureIPexpert spi 0x1 MDS2(config-if-vrrp)# no shut MDS2(config-if-vrrp)#



After this configuration the communication of the VRRP is now authenticated between the neighbors.

The next 2 tasks are regarding the security of the FCIP 1 connection. This configuration is quite complex on the MDS switch and you should pay close attention to what you configure here.

The IPsec feature is not available on all platforms, but it is available on the platform in the CCIE Data Center lab topology, MDS 9222i. It’s not available on the IPS-4 and IPS-8 line cards. You also require a ENTERPRISE license for this feature.

Configuration is similar to the IOS configuration for IPsec VPNs, but you will find a little different commands are used here and there.

The question states that we should secure the FCIP 1 connection. Keep in mind that you use the VRRP configuration as the peer address configurations, but you should apply the IPsec tunnels to both interfaces where the tunnel could terminate. When this isn’t done, you will loose points as the FCIP 1 connection is high available and your security not!

Let’s start the configuration on MDS1.

MDS1

MDS1(config)# feature crypto ike MDS1(config)# feature crypto ipsec MDS1(config)# MDS1(config)# crypto ike domain ipsec MDS1(config-ike-ipsec)# identity address MDS1(config-ike-ipsec)# key IPexpertEncrypt address 198.18.69.11 MDS1(config-ike-ipsec)# policy 1 MDS1(config-ike-ipsec-policy)# encr aes MDS1(config-ike-ipsec-policy)# hash md5 MDS1(config-ike-ipsec-policy)# authentication pre-share MDS1(config-ike-ipsec-policy)# group 2 MDS1(config-ike-ipsec-policy)# exit MDS1(config-ike-ipsec)# MDS1(config-ike-ipsec)# ip access-list FCIP1IPSEC permit ip 198.18.69.9 0.0.0.0 198.18.69.11 0.0.0.0 MDS1(config)# crypto transform-set domain ipsec FCIP1 esp-aes ? 128 ESP transform using AES 128 bit cipher 256 ESP transform using AES 256 bit cipher MDS1(config)# crypto transform-set domain ipsec FCIP1 esp-aes 128 esp-md5-hmac MDS1(config)# crypto map domain ipsec FCIP 1 MDS1(config-crypto-map-ip)# match address FCIP1IPSEC MDS1(config-crypto-map-ip)# set peer 198.18.69.11 MDS1(config-crypto-map-ip)# set transform-set FCIP1 MDS1(config-crypto-map-ip)# interface gig1/1.69 MDS1(config-if)# crypto map domain ipsec FCIP MDS1(config-if)# interface gig1/2.69 MDS1(config-if)# crypto map domain ipsec FCIP



We configured the IPsec tunnel on both interfaces for only the traffic that is using the VRRP group addresses. Other traffic is not hit by the configured access-list and will therefore not hit the crypto map and traffic will not be encrypted. Only the IP addresses that are configured in the access-list will hit the crypto map and will be encrypted.

Now we need to finish the configuration for the second MDS.

MDS2

MDS2(config)# feature crypto ike MDS2(config)# feature crypto ipsec MDS2(config)# MDS2(config)# crypto ike domain ipsec MDS2(config-ike-ipsec)# identity address MDS2(config-ike-ipsec)# key IPexpertEncrypt address 198.18.69.9 MDS2(config-ike-ipsec)# policy 1 MDS2(config-ike-ipsec-policy)# encr aes MDS2(config-ike-ipsec-policy)# hash md5 MDS2(config-ike-ipsec-policy)# authentication pre-share MDS2(config-ike-ipsec-policy)# group 2 MDS2(config-ike-ipsec-policy)# exit MDS2(config-ike-ipsec)# MDS2(config-ike-ipsec)# ip access-list FCIP1IPSEC permit ip 198.18.69.11 0.0.0.0 198.18.69.9 0.0.0.0 MDS2(config)# crypto transform-set domain ipsec FCIP1 esp-aes 128 esp-md5-hmac MDS2(config)# crypto map domain ipsec FCIP 1 MDS2(config-crypto-map-ip)# match address FCIP1IPSEC MDS2(config-crypto-map-ip)# set peer 198.18.69.11 MDS2(config-crypto-map-ip)# set transform-set FCIP1 MDS2(config-crypto-map-ip)# interface gig1/1.69 MDS2(config-if)# crypto map domain ipsec FCIP MDS2(config-if)# interface gig1/2.69 MDS2(config-if)# crypto map domain ipsec FCIP

Now all FCIP 1 related traffic is secured and we have finished task 3!



Task 4: SAN Extension Tuner

The next task will let you configure a specific MDS feature that will let you generate test traffic to tune your FCIP configurations. This feature is called the SAN Extension Tuner.

The test traffic that is generated so you are able to optimize your FCIP throughput according to the IP network latency, jitter and throughput.

You will create a virtual initiator and a virtual target at each end of an FCIP connection. Between this initiator and target you will create a stream of I/O’s just like normal FC traffic. By using this data you are able to optimize the FCIP connection.

There are a number of requirements and best practices to follow when you want to use this feature.

The virtual ports do not register with the name-server

The virtual ports will not show up for other initiators and targets in the Fibre Channel fabric.

Login requests are rejected

When a request is sent to the FCID the request is rejected

Virtual hosts can only communicate to each other

The virtual initiator and virtual target are only able to communicate to each other and not to other hosts or targets in the Fibre Channel fabric

Use a different GigabitEthernet interface than the FCIP interface

As you are using this feature to test the ethernet interface that is used for the FCIP connection, you should not use the same GigabitEthernet interface as the tunnel itself because it impacts the amount of traffic in the link

This interface does not need to be up, it is just used to generate the traffic on

iSCSI needs to be enabled

Separate VSAN is recommended

The use of this feature in a separate VSAN is not strictly necessary, because all traffic towards the virtual hosts is rejected, but it’s still recommended to configure this feature in a separate VSAN.

Configuration can be extensive. You are required to use manually configured nWWNs and pWWNs. Besides that it’s important to remember that you first need to configure the san-extension-tuner configuration about the virtual initiator/target on each switch before you start configuring the commands. Otherwise the command for the read or write action is not accepted when the virtual host is not found.



Be aware that you require zoning changes before the hosts are able to communicate to each other. As we have no zoning configured for VSAN 50, which is the VSAN we should use in this task. We can just allow traffic to be sent in this VSAN in the Default Zone.

We require to enable a couple of things before we start the san-extension-tuner configuration.

MDS1

MDS1(config)# feature san-ext-tuner MDS1(config)# feature iscsi MDS1(config)# int gig1/3 MDS1(config-if)# no shut MDS1(config-if)# int iscsi1/3 ^ Invalid range at '^' marker. MDS1(config)# iscsi enable module 1 MDS1(config)# int iscsi1/3 MDS1(config-if)# no shut

We also need to enable the iSCSI feature for this task. When you enable iSCSI you also need to enable the iSCSI interface. The interface numbering is equal to the GigabitEthernet number. Before this interface exists, you also need to enable the iSCSI feature on the line card (module) that you want to enable the iSCSI interface on. In the case of a fixed chassis, like the MDS 9222i, we only have 1 module.

After it’s enabled we can enable the san-extension-tuner host in the network and we need to enable the Default Zone to allow traffic for being sent between hosts.

MDS1

MDS1(config)# zone default-zone permit vsan 50 MDS1(config)# exit MDS1(config)# san-ext-tuner ^ % Incomplete command at '^' marker.

Weird enough, the san-extension-tuner configuration is not entered using the configuration mode, but from the operational mode.

MDS1(config)# exit MDS1# san-ext-tuner MDS1(san-ext)# ? nWWN Configure san_ext_tuner nwwn no Negate a command or set its defaults nport Configure nport end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in MDS1(san-ext)# nwwn ? <hh:hh:hh:hh:hh:hh:hh:hh> Enter the N-WWN of san extension tuner MDS1(san-ext)# nwwn 10:00:00:00:00:00:00:01



MDS1(san-ext)# nport pwWN 10:00:00:00:00:00:00:02 ? vsan Configure nport vsan MDS1(san-ext)# nport pwWN 10:00:00:00:00:00:00:02 vsan 50 ? interface Specify the interface MDS1(san-ext)# nport pwWN 10:00:00:00:00:00:00:02 vsan 50 interface ? gigabitethernet Ethernet interface MDS1(san-ext)# nport pwWN 10:00:00:00:00:00:00:02 vsan 50 interface gig1/3

After the N-port is configured you can see a successful FLOGI being done in the configured VSAN with the nWWN and pWWN that you just configured.

MDS1(san-ext-nport)# sh flogi data v 50 -------------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME -------------------------------------------------------------------------------- iscsi1/3 50 0x6c0000 10:00:00:00:00:00:00:02 10:00:00:00:00:00:00:01 Total number of flogi = 1. MDS1(san-ext-nport)#

Now we need to repeat these steps on the second MDS.

MDS2

MDS2(config)# feature san-ext-tuner MDS2(config)# feature iscsi MDS2(config)# int gig1/3 MDS2(config-if)# no shut MDS2(config)# iscsi enable module 1 MDS2(config)# int iscsi1/3 MDS2(config-if)# no shut MDS2(config)# zone default-zone permit vsan 50 MDS2(config)# exit MDS2# san-ext-tuner MDS2(san-ext)# nwwn 20:00:00:00:00:00:00:01 MDS2(san-ext)# nport pwWN 20:00:00:00:00:00:00:02 vsan 50 interface gig1/3

Now we have 2 virtual hosts in VSAN 50 that we can use to create our data communication flows on for our san-extension-tuner feature.

Now we configure the read flow towards 1 direction.

MDS1

MDS1# san-ext-tuner MDS1(san-ext)# nport pwWN 10:00:00:00:00:00:00:02 vsan 50 interface gig1/3 MDS1(san-ext-nport)# read command-id 1 target 20:00:00:00:00:00:00:02 transfer-size 512000 outstanding-ios 2 ? <CR> continuous Do transactions forever



num-transactions Configure num transactions MDS1(san-ext-nport)# read command-id 1 target 20:00:00:00:00:00:00:02 transfer-size 512000 outstanding-ios 2 continuous

Do not forget to enable the continuous keyword at the end of the configuration. The question clearly states that you should start a continuous flow of traffic

We currently have traffic in one direction. The task states that we should enable the read commands in both directions. So we need to configure the same configuration on the other MDS switch as well.

MDS2

MDS2# san-ext-tuner MDS2(san-ext)# nport pwWN 10:00:00:00:00:00:00:02 vsan 50 interface gig1/3 MDS2(san-ext-nport)# read command-id 1 target 20:00:00:00:00:00:00:02 transfer-size 512000 outstanding-ios 2 continuous

After this configuration is applied the read command immediately starts and traffic is flowing across the FCIP 2 connection, because this is the only FCIP connection where VSAN 50 is allowed.

Task 5: iSCSI

The next task is all about the iSCSI feature on the Cisco MDS platform. This feature has been available since a long time and is still available, however a more newer version is also available, which is discussed in the next task.

Using the iSCSI feature on the MDS platform you are able to extend IP connected hosts (initiators) on to the Fibre Channel network. This means that the Fibre Channel fabric will have an entry in the name-server for this IP host just like if it was a Fibre Channel connected server.



On the other side, you can create iSCSI targets based on Fibre Channel targets. You can take any target in the Fibre Channel fabric and make this accessible through iSCSI.

The iSCSI feature is available on the same IP interfaces as the FCIP feature is available.

The MDS switch makes the conversion between the IP traffic and Fibre Channel traffic. Hosts that log-in using iSCSI do not know they are actually communicating with a Fibre Channel switch and a Fibre Channel disk. The iSCSI header is stripped and a Fibre Channel header is placed on top of it. Of course the return traffic works exactly vice versa.

The drawing below illustrates the working of the iSCSI portal on the MDS switch.

The most easy option for configuring iSCSI is to use the dynamic mapping feature.

With this feature it’s possible to present every target available on the Fibre Channel network as an iSCSI target. As the first question states in this task this is prohibited!

This leaves you with the option to configure static targets. Here you have options to configure targets either on a target basis or you can even specify which LUN can be accesses through the iSCSI network.

As with targets you can also dynamically create virtual initiators. Meaning that every device that tries a log-in on the MDS switch IP address on the iSCSI portal, will automatically get a virtual initiator assigned. Again we do not want this and we will be configuring static initiators so we have full control over which host can access which target, as the initiators will need to be included in zoning!

The 2 options that you can use to define a iSCSI initiator are:

iSCSI Qualified Name (iQN)

This IQN is a clear-text name that the iSCSI device uses to identify itself. When an initiator has multiple IP addresses it will still be just a single Virtual N-Port in the Fibre Channel fabric



IP address

The Virtual N-port can also be created according to the IP address that the iSCSI initiator uses to access the MDS. Each IP address is another virtual N-port and another host in the Fibre Channel fabric

In the initial configuration we will be using the Transparent Initiator mode, which means that every iSCSI initiator will appear as an Initiator in the FC fabric network (Virtual N-port).

During this task we will also be configuring the other mode, which is the Proxy Initiator mode. With the use of this mode. The MDS switch will act as a single virtual N-port and all iSCSI initiators are hidden behind this ‘proxy’ feature. This means that you can not use zoning to determine which iSCSI initiator can access which target.

However there are multiple ways in the iSCSI configuration where you can specify which target can access which initiator.

Now we start the configuration of our first couple questions in this iSCSI task.

We call an iSCSI interface on the MDS switch a iSCSI Portal, as it’s a single portal where you can log-in and access all targets that you have the right to.

We start with some infrastructure configuration to create the iSCSI portal.

The iSCSI interface that will be configured here is a virtual interface that is an overlay on top of the GigabitEthernet interface. The numbering of the iSCSI interface is equal to the GigabitEthernet numbering. This interface is required to be enabled, to really activate the iSCSI portal on the MDS. Besides that, which we already did in the previous task, is that we need to enable the iSCSI feature globally and per line-card where the feature needs to be enabled.

We configure the VLAN according to Drawing 2. There we find that we should be using VLAN 71. Therefore we need to create another sub-interface for the VLAN. It’s not possible on the Cisco MDS switch to configure the iSCSI portal on a sub-interface. Meaning that all iSCSI interface configuration that you do will be used on all sub-interfaces that are configured.

This means that you can not create different iSCSI portals on different sub-interfaces, but only a single portal per physical interface.

MDS1

MDS1(config)# feature iscsi MDS1(config)# iscsi enable module 1 MDS1(config)# interface gig1/1 MDS1(config-if)# no shut MDS1(config-if)# int gi1/1.70 MDS1(config-if)# ip add 198.18.70.9 255.255.255.0 MDS1(config-if)# no shut



MDS1(config-if)# int iscsi1/1.70 ^ Invalid interface format at '^' marker

Like previously mentioned. You can not create an iSCSI sub-interface only a global interface.

MDS1(config-if)# int iscsi1/1 MDS1(config-if)# port 3261 MDS1(config-if)# tcp ? cwm Enable congestion window monitoring keepalive-timeout Set keep alive timeout in sec max-bandwidth-kbps Configure maximum available path bandwidth in Kbps max-bandwidth-mbps Configure maximum bandwidth max-jitter Configure jitter for iSCSI in micro seconds max-retransmissions Maximum number of retransmissions min-retransmit-time Set minimum retransmit time in millisec pmtu-enable Enable PMTU Discovery sack-enable Enable SACK option for TCP send-buffer-size Send buffer size in KBytes

The TCP parameters that can be configured are similar to the options that are available for configuring FCIP connections.

MDS1(config-if)# qos ? <0-63> QOS code point value MDS1(config-if)# qos 22 MDS1(config-if)# no shut MDS1(config-if)# vsan data MDS1(config-vsan-db)# vsan 10 interface iscsi1/1

Finally the last configuration that needs to be done is by putting the iSCSI interface into another VSAN, because the Default VSAN is suspended. When this is not configured, no matter what kind of VSAN configuration is done under the virtual initiators or virtual targets the interface will remain down, because the Default VSAN is suspended.

The different topics in this infrastructure task is that the portal should use a non-default port. By default iSCSI uses TCP 3260 for all communications. We now change this to TCP 3261. The configuration is done under the iSCSI interface. Under the iSCSI interface the characteristics of the traffic, like TCP parameters, port number, etc.

MDS1(config-if)# do sh int iscsi1/1 iscsi1/1 is up Hardware is GigabitEthernet Port WWN is 20:13:00:0d:ec:54:a9:80 Admin port mode is ISCSI snmp link state traps are enabled Port mode is ISCSI Port vsan is 10 Speed is 1 Gbps Interface last changed at Tue Oct 23 14:13:46 2012 iSCSI initiator is identified by name Number of iSCSI session: 0 (discovery session: 0) Number of TCP connection: 0 Configured TCP parameters Local Port is 3261



PMTU discover is enabled, reset timeout is 3600 sec Keepalive-timeout is 60 sec Minimum-retransmit-time is 300 ms Max-retransmissions 4 Sack is enabled QOS code point is 22 Maximum allowed bandwidth is 1000000 kbps Minimum available bandwidth is 70000 kbps Configured round trip time is 1000 usec Send buffer size is 4096 KB Congestion window monitoring is enabled, burst size is 50 KB Configured maximum jitter is 500 us Forwarding mode: store-and-forward TMF Queuing Mode: enabled Proxy Initiator Mode: disabled 5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec iSCSI statistics Input 0 packets, 0 bytes Command 0 pdus, Data-out 0 pdus, 0 bytes, Unsolicited 0 bytes Output 0 packets, 0 bytes Response 0 pdus (with sense 0), R2T 0 pdus Data-in 0 pdus, 0 bytes

Now our iSCSI portal is up and running, but since we are using no dynamic targets or initiators nothing will be able to log-in to it yet.

As shown in this output, you can see that by default initiators are recognized by their name (iQN). This needs to be changed for this portal, as we want to use the IP address of the initiator as we look at the following question. This can be changed per iSCSI interface and therefore per iSCSI portal configured on the MDS switch.

MDS1

MDS1(config)# int iscsi1/1 MDS1(config-if)# switchport ? description Enter description of maximum 80 characters initiator Configure iSCSI initiator id mode proxy-initiator Configure iSCSI proxy initiator mode timestamp-check Enable timestamp check MDS1(config-if)# switchport initiator ? id Configure iSCSI initiator id mode MDS1(config-if)# switchport initiator id ? ip-address Configure iSCSI initiator id mode based on ip-address name Configure iSCSI initiator id mode based on init name MDS1(config-if)# switchport initiator id ip-address



The Cisco MDS switch allows you to configure both manual nWWN and pWWN’s to the initiators and you can chose for the feature that the MDS will select pWWNs or a nWWN automatically based on a pool.

In this case we will use statically configured WWNs.

MDS1

MDS1(config)# iscsi initiator ip-address 198.18.71.100 MDS1(config-iscsi-init)# ? mutual-chap Assign username for initiator's challenge (mutual chap) no Negate a command or set its defaults static Create static nWWN or pWWN for the initiator username Assign username for iSCSI login authentication vsan Assign VSAN membership for the initiator end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in MDS1(config-iscsi-init)# static nWWN 11:00:00:00:00:00:00:01 MDS1(config-iscsi-init)# static pwwn ? <hh:hh:hh:hh:hh:hh:hh:hh> Enter pWWN system-assign Ask the system to assign static pWWN(s) for this initiator MDS1(config-iscsi-init)# static pwwn 22:00:00:00:00:00:00:01

We need to ensure that the initiator is participating in VSAN 20, but we can not configure this under the initiator by using the vsan command in this case. With the use of this command you are able to configure different initiators in different VSANs. This creates a lot of flexibility in the deployment of the initiators.

We just configured the iSCSI interface under the VSAN database. So we need to change this to VSAN 20 to support our question and let the initiator have access to resources in VSAN 20.

MDS1

MDS1(config-if)# vsan data MDS1(config-vsan-db)# vsan 20 interface iscsi1/1



Now that we have finished our configuration of our initiator we can verify it’s configuration. Don’t get scared when you issue the first command and you don’t see any results.

This has to do with the fact that the show iscsi initiator command will only show you the currently active iSCSI sessions. With the configured keyword behind it, it will show you the configuration of your iSCSI initiators.

You will also not see any FLOGI’s or Name-Server entries until the iSCSI session is active and working.

MDS1

MDS1(config-vsan-db)# sh iscsi initiator MDS1(config-vsan-db)# sh iscsi initiator ? *** No matching command found in current mode, matching in (exec) mode *** <CR> > Redirect it to a file >> Redirect it to a file in append mode WORD Enter initiator node name (Max Size 223) configured Show iSCSI initiator configured information detail Show more detailed information fcp-session Show FC session details iscsi-session Show iSCSI session details summary Show iSCSI initiator summary information | Pipe command output to filter MDS1(config-vsan-db)# sh iscsi initiator configured iSCSI Node name is 198.18.71.100 Node WWN is 11:00:00:00:00:00:00:01 No. of PWWN: 1 Port WWN is 22:00:00:00:00:00:00:01 Configured node (iSCSI) MDS1(config-vsan-db)#



Now that the initiator part is configured we can start with configuring targets for this iSCSI host. The host should be able to access the JBOD 2 Disk 1 in VSAN 20.

The next question states that you should be configuring a virtual target for the JBOD 2 Disk 1. This is done using a similar configuration as the initiator.

Before the virtual target can be accessed you need to configure which initiators are allowed to access it. This means, the initiators that will see it when they discover the targets through iSCSI. After this configuration you still need zoning where you need to give the virtual pWWN or IP-address access to the Fibre Channel target.

Next is that you can configure the virtual-target to have it showing up on which iSCSI portal. This is done by specifying the GigabitEthernet interface (and with that the portal) where it’s available.

Now on to the configuration.

MDS1

MDS1(config)# iscsi virtual-target name iqn.iscsi-disk-JBOD2-Disk1 MDS1(config-iscsi-tgt)# ? advertise Advertise virtual-target on interfaces specified all-initiator-permit Allow all iSCSI initiator access to this target initiator Allow iSCSI initiator access to this target no Negate a command or set its defaults pWWN Enter the pWWN of the fc-target revert-primary-port Revert back to primary port when it comes back up end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in MDS1(config-iscsi-tgt)# initiator ? WORD Enter iSCSI initiator name (Max Size 223) ip Allow iSCSI initiator access to this target by ip address MDS1(config-iscsi-tgt)# initiator ip address 198.18.71.100 ? A.B.C.D Enter initiator ip netmask permit Allow iSCSI initiator access to this target MDS1(config-iscsi-tgt)# initiator ip address 198.18.71.100 permit

You can specify an entire range of IP addresses with this command, when you do not configure any netmask, it means that you only allow a single host.

MDS1

MDS1(config-iscsi-tgt)# advertise ? interface Interface eth-interface-to-advertise MDS1(config-iscsi-tgt)# advertise int gig1/1.71 MDS1(config-iscsi-tgt)# pwwn ? <hh:hh:hh:hh:hh:hh:hh:hh> Enter pWWN



Within the current iSCSI configuration it’s only possible to configure a pWWN of the target you want to make available. You will see in the next task that there are more options available, like using the device-alias instead of the pWWN, which is a much more flexible way.

MDS1

MDS1(config-iscsi-tgt)# pwwn 21:00:00:11:0d:18:fe:ce ? <CR> fc-lun FC lun secondary-pWWN Enter the secondary pWWN of the fc-target

The other options are not used in the current question.

MDS1

MDS1(config-iscsi-tgt)# pwwn 21:00:00:11:0d:18:fe:ce MDS1(config-iscsi-tgt)# exit

To finish up, we need to configure zoning so the host as access to the target. This can be based on multiple things.

MDS1

MDS1(config)# zone name iSCSI_1 vsan 20 MDS1(config-zone)# member ? device-alias Add device-alias member to zone domain-id Add member based on domain-id,port-number fcalias Add fcalias to zone fcid Add FCID member to zone fwwn Add Fabric Port WWN member to zone interface Add member based on interface ip-address Add IP address member to zone pwwn Add Port WWN member to zone symbolic-nodename Add Symbolic Node Name member to zone

You can use the following options in letting iSCSI initiators access Fibre Channel targets:

IP address

Just use the IP address of the initiator, just like we configured the virtual initiator

IQN

Use the ‘hostname’ of the initiator, called the symbolic-nodename in the zoning configuration to configure the initiator in the zone. This is a flexible way independent of the IP addressing of the initiator



pWWN

You can use the virtual pWWN that is manually assigned by you or system assigned by the MDS switch to give the initiator access to the FC Disk. This is also a very flexible option, but it makes you zoning configuration harder to read

MDS1

MDS1(config-zone)# member ip-address 198.18.71.100 MDS1(config-zone)# member device-alias J2D1 MDS1(config-zone)# zoneset name ZS1 vsan 20 MDS1(config-zoneset)# member iSCSI_1 MDS1(config-zoneset)# zoneset activate name ZS1 vsan 20 MDS1(config)# zone commit vsan 20 Zoneset activation initiated. check zone status MDS1(config)#

The next question requires you to configure authentication. It’s possible to authenticate your iSCSI session in both directions, which is what we are looking for.

You can authenticate the iSCSI users according to the local database, which is what we are going to use here, or through an RADIUS or TACACS+ server.

Be aware that when you are configuring the user, to not forget the special ‘iscsi’ keyword behind the line, otherwise it will definitely not work!

Especially when you are configuring mutual-chap, where the second authentication is configured right under the virtual initiator, you might think that it enough, but you are not done there. You need to configure the initial CHAP username/password as well!

MDS1

MDS1(config)# iscsi initiator ip-address 198.18.71.100 MDS1(config-iscsi-init)# username iSCSI1 MDS1(config-iscsi-init)# mutual-chap ? username Assign username for initiator's challenge (mutual chap) MDS1(config-iscsi-init)# mutual-chap username iSCSI1 ? password Assign password for initiator's challenge (mutual chap) MDS1(config-iscsi-init)# mutual-chap username iSCSI1 password IP3xp3rtiSCSI MDS1(config-iscsi-init)# exit MDS1(config)# username iSCSI1 password IP3xp3rtiSCSI iscsi

After this the configuration of this virtual target is done and we can verify our configuration.

MDS1(config)# show iscsi virtual-target target: iqn.iscsi-disk-jbod2-disk1 Port WWN 21:00:00:11:0d:18:fe:ce Configured node (iSCSI) No. of advertised interface: 1 GigabitEthernet 1/1.71 No. of initiators permitted: 1 initiator 198.18.71.100/32 is permitted All initiator permit is disabled Trespass support is disabled



Revert to primary support is disabled MDS1(config)#

All configuration parameters that we configured are available. Now we check our changed initiator, where we added the authentication.

MDS1(config)# show iscsi initiator configured iSCSI Node name is 198.18.71.100 Node WWN is 11:00:00:00:00:00:00:01 No. of PWWN: 1 Port WWN is 22:00:00:00:00:00:00:01 Configured node (iSCSI) User Name for login authentication: iSCSI1 User Name for Mutual CHAP: iSCSI1 MDS1(config)#

The authentication is added and we see that the same username is used for both the login and the second CHAP (mutual).

The next question asks you to configure another virtual target with a LUN specified. The iSCSI feature supports LUN masking, so you are able to convert the LUN IDs of targets to another LUN ID on the iSCSI level.

The question will also be using another great feature of the iSCSI feature of the Cisco MDS platform. The iSCSI virtual-target feature can take care of high availability. Where it will be using a sort of SDV (SAN Device Virtualization) technology to ensure this. The iSCSI virtual target will have access to a primary pWWN and a secondary pWWN that it can use to failover to as soon as the primary fails.

Configuration can generate quite a long command!

MDS1

MDS1(config)# iscsi virtual-target name iqn.iscsi-disk-jbod1-disk3 MDS1(config-iscsi-tgt)# sh device-alias data device-alias name J1D1 pwwn c6:83:00:11:0d:18:fa:ce device-alias name J1D2 pwwn c6:83:00:11:0d:18:fb:ce device-alias name J1D3 pwwn c6:83:00:11:0d:18:fc:ce device-alias name J1D4 pwwn c6:83:00:11:0d:18:fd:ce device-alias name J2D1 pwwn 21:00:00:11:0d:18:fe:ce device-alias name J2D2 pwwn 21:00:00:11:0d:18:ff:ce device-alias name J2D3 pwwn 21:00:00:11:0d:18:a0:ce device-alias name J2D4 pwwn 21:00:00:11:0d:18:a1:ce Total number of entries = 8 MDS1(config-iscsi-tgt)# pwwn c6:83:00:11:0d:18:fc:ce fc-lun 0xa ? iscsi-lun ISCSI lun MDS1(config-iscsi-tgt)# pwwn c6:83:00:11:0d:18:fc:ce fc-lun 0xa iscsi-lun 0x0 ? <CR> secondary-pWWN Enter the secondary pWWN of the fc-target MDS1(config-iscsi-tgt)# pwwn c6:83:00:11:0d:18:fc:ce fc-lun 0xa iscsi-lun 0x0 secondary-pWWN 21:00:00:11:0d:18:a0:ce ? <CR> sec-lun FC lun on the secondary port



MDS1(config-iscsi-tgt)# pwwn c6:83:00:11:0d:18:fc:ce fc-lun 0xa iscsi-lun 0x0 secondary-pWWN 21:00:00:11:0d:18:a0:ce sec-lun 0xa

Here we can see that we can even configure a different Fibre Channel LUN for both the primary and the backup target. As the question only specified LUN 10, we will continue to use this on the backup target as well.

MDS1

MDS1(config-iscsi-tgt)# all-initiator-permit

As the question stated that ‘iSCSI initiators’ should have access to the virtual-target, this indicates that we want to allow all initiators that will be logging in, to have access to this specific target. As we only have 1 initiator configured, this will be currently the only one that has access to it.

MDS1

MDS1(config-iscsi-tgt)# revert-primary-port

The question also stated that the backup disk should fall back to the primary when this is available again after repair.

MDS1

MDS1(config-iscsi-tgt)# trespass MDS1(config-iscsi-tgt)#

The trespass support that is asked for, is currently a hidden command, but still available in the configuration and a supported feature. This feature ensures that Logical Units are transferred over to the failover target.

Do not forget to enable zoning for the initiator that we configured! In this zoning it’s very important not to forget to add both the disks that you are using, so both the primary and the backup target should be mentioned in the zone, before the initiator has access to it.

MDS1

MDS1(config)# zone name iSCSI_2 vsan 20 MDS1(config-zone)# member ip-address 198.18.71.100 MDS1(config-zone)# member device-alias J1D3 MDS1(config-zone)# member device-alias J2D3 MDS1(config-zone)# zoneset name ZS1 vsan 20 MDS1(config-zoneset)# member iSCSI_2 MDS1(config-zoneset)# zoneset activate name ZS1 vsan 20 MDS1(config)# zone commit vsan 20 Zoneset activation initiated. check zone status MDS1(config)#

Now we can verify our new target.

MDS1(config-iscsi-tgt)# show iscsi virtual-target target: iqn.iscsi-disk-jbod2-disk1 Port WWN 21:00:00:11:0d:18:fe:ce Configured node (iSCSI) No. of advertised interface: 1 GigabitEthernet 1/1.71 No. of initiators permitted: 1



initiator 198.18.71.100/32 is permitted All initiator permit is disabled Trespass support is enabled Revert to primary support is disabled target: iqn.iscsi-disk-jbod1-disk3 Port WWN c6:83:00:11:0d:18:fc:ce Secondary PWWN 21:00:00:11:0d:18:a0:ce Configured node (iSCSI) No. of LU mapping: 1 iSCSI LUN: 0x0000, FC LUN: 0x000a, Secondary FC LUN: 0x000a All initiator permit is enabled Trespass support is enabled Revert to primary support is enabled MDS1(config-iscsi-tgt)#

In the second virtual target you will find the 2 pWWNs of the disks and the features that are all enabled.

The final feature that should be enabled on this iSCSI portal is that we want you to improve the read performance on the switch.

This is related to the way the iSCSI packets are treated in the MDS switch. The MDS can treat iSCSI frames just like an Ethernet switch has different modes to transfer Ethernet packets.

The modes that are available are:

Pass-Thru

The downside of this mode is that it lowers the total amount of data transfer that is possible. However the positive side is that you get lower latency and you are able to use the data-digest feature.

Store-and-Forward

This is the default and recommended mode. Data-digest can not be used, but you do get a faster data transfer performance in the iSCSI process. The entire iSCSI packet is cached inside the box before the next action is taken.

Cut-Thru

This mode improves the read performance when compared to store-and-forwards. The downside is that the improvement only works for read performance and for initial packets only.

The changing of the mode is also a hidden command, but you are still able to configure all three modes and they are supported. In this case we want to improve read performance, so we should be using the Cut-Thru switching mode.

MDS1

MDS1(config)# int iscsi1/1



MDS1(config-if)# mode ? store-and-forward Enable store-and-forward MDS1(config-if)# mode cut-thru

The different modes are not available through the CLI context help, but they are available. You can verify using the following output.

MDS1(config-if)# do sh int iscsi1/1 iscsi1/1 is down (Initializing) Hardware is GigabitEthernet Port WWN is 20:13:00:05:9b:70:5a:00 Admin port mode is ISCSI snmp link state traps are enabled Port vsan is 20 iSCSI initiator is identified by ip address Number of iSCSI session: 0 (discovery session: 0) Number of TCP connection: 0 Configured TCP parameters Local Port is 3261 PMTU discover is enabled, reset timeout is 3600 sec Keepalive-timeout is 60 sec Minimum-retransmit-time is 300 ms Max-retransmissions 4 Sack is enabled QOS code point is 0 Maximum allowed bandwidth is 1000000 kbps Minimum available bandwidth is 70000 kbps Configured round trip time is 1000 usec Send buffer size is 4096 KB Congestion window monitoring is enabled, burst size is 50 KB Configured maximum jitter is 500 us Forwarding mode: cut-thru TMF Queuing Mode: enabled Proxy Initiator Mode: disabled 5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec iSCSI statistics Input 0 packets, 0 bytes Command 0 pdus, Data-out 0 pdus, 0 bytes, Unsolicited 0 bytes Output 0 packets, 0 bytes Response 0 pdus (with sense 0), R2T 0 pdus Data-in 0 pdus, 0 bytes MDS1(config-if)#

Next we are configuring a new iSCSI portal on the other MDS switch.

This new portal is configured in the same iSCSI VLAN as we used in the previous questions, but as we are using another IP address on another MDS switch, there is no problem in any overlap.

This portal will work a little different than the previous one as the iSCSI initiators should all be hidden behind a single nWWN and pWWN. This is called the ‘proxy-initiator-mode’. With this mode the MDS switch will act as the Fibre Channel host in the network where all traffic is sent and the switch will convert it then to iSCSI packets destined to the correct initiator.



This mode introduces flexibility as you only have a single entry in your FC Name-Server for the iSCSI hosts. It hides the iSCSI hosts from the network, but it eliminates the fact that you can zone the targets and initiators together. It also creates simplicity as you have most security features to secure the combination of initiators and targets in the virtual-target configuration as you would have in zoning. Therefore the adding of each initiator and each target (including possible backup targets) will create a lot of additional zones that need to be configured.

Within the proxy-initiator-mode, the MDS will create a virtual N-port per iSCSI portal (per iSCSI interface to be exact). So when you enable this mode on multiple iSCSI interfaces, it will create additional N-ports in the FC Fabric, which you all need to add to the zones.

The following drawing illustrates how the Proxy Initiator Mode works.

We now configure the iSCSI portal on MDS2. Most configuration is equal to the other iSCSI portal on MDS1, except that the behavior is very different.

Keep in mind that when you configure proxy-initiator-mode, it’s not possible anymore to do your zoning based on IQNs or IP addresses.

MDS2

MDS2(config)# feature iscsi MDS2(config)# iscsi enable mod 1 MDS2(config)# int gig1/1.71 MDS2(config-if)# ip add 198.18.71.11 255.255.255.0 MDS2(config-if)# no shut MDS2(config-if)# int gig1/1 MDS2(config-if)# no shut MDS2(config-if)# int iscsi1/1 MDS2(config-if)# no shut



We are first required to enable iSCSI on this switch as it wasn’t enabled before.

MDS2

MDS2(config-if)# switchport proxy-initiator ? <CR> nWWN Configure nWWN for proxy initiator mode MDS2(config-if)# switchport proxy-initiator nwwn ? <hh:hh:hh:hh:hh:hh:hh:hh> Enter nWWN MDS2(config-if)# switchport proxy-initiator nwwn 11:22:33:44:55:66:77:88 ? pwwn Configure pwwn for proxy initiator mode MDS2(config-if)# switchport proxy-initiator nwwn 11:22:33:44:55:66:77:88 pwwn ? <hh:hh:hh:hh:hh:hh:hh:hh> Enter pWWN MDS2(config-if)# switchport proxy-initiator nwwn 11:22:33:44:55:66:77:88 pwwn aa:bb:cc:dd:ee:ff:00:11 MDS2(config-if)#

In this example we are using manually configured nWWN and pWWN for the proxy host that is generated on the switch. This makes it easier to define our zoning later on as we do not need to look up the values, but we configured them by ourselves. Be aware that this should be a globally unique value. In the CCIE Data Center lab exam you do not have that many Fibre Channel connected devices, so it should be fine to use these kinds of values for lab purposes.

Now we can verify our proxy-initiator mode configuration.

MDS2(config-if)# sh int iscsi1/1 iscsi1/1 is up Hardware is GigabitEthernet Port WWN is 20:13:00:05:9b:77:0d:c0 Admin port mode is ISCSI snmp link state traps are enabled Port mode is ISCSI Port vsan is 1 Speed is 1 Gbps Interface last changed at Wed Oct 24 23:43:44 2012 iSCSI initiator is identified by name Number of iSCSI session: 0 (discovery session: 0) Number of TCP connection: 0 Configured TCP parameters Local Port is 3260 PMTU discover is enabled, reset timeout is 3600 sec Keepalive-timeout is 60 sec Minimum-retransmit-time is 300 ms Max-retransmissions 4 Sack is enabled QOS code point is 0 Maximum allowed bandwidth is 1000000 kbps Minimum available bandwidth is 70000 kbps Configured round trip time is 1000 usec Send buffer size is 4096 KB Congestion window monitoring is enabled, burst size is 50 KB Configured maximum jitter is 500 us Forwarding mode: store-and-forward TMF Queuing Mode: enabled Proxy Initiator Mode: enabled



nWWN is 11:22:33:44:55:66:77:88 (manually-configured) pWWN is aa:bb:cc:dd:ee:ff:00:11 (manually-configured) 5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec iSCSI statistics Input 0 packets, 0 bytes Command 0 pdus, Data-out 0 pdus, 0 bytes, Unsolicited 0 bytes Output 0 packets, 0 bytes Response 0 pdus (with sense 0), R2T 0 pdus Data-in 0 pdus, 0 bytes

The next question requires you to configure the data-digest feature. The funny thing is that this is a feature that can not be directly configured. However, as just discussed, this feature is only available in a certain forwarding mode.

When enabling this feature the data-digest feature is automatically enabled when the iSCSI initiators send requests for this!

MDS2

MDS2(config-if)# int iscsi1/1 MDS2(config-if)# mode pass-thru

Now we are going to configure 3 initiators that will be logging in to the iSCSI portal. They are member of VSAN 10, but we can’t configure the VSAN database for this VSAN. Remember that the iSCSI portal is currently in the suspended Default VSAN and will not come online!

MDS2

MDS2(config)# iscsi initiator name iqn.initiator-server-1 MDS2(config-iscsi-init)# vsan 10 MDS2(config-iscsi-init)# iscsi initiator name iqn.initiator-server-2 MDS2(config-iscsi-init)# vsan 10 MDS2(config-iscsi-init)# iscsi initiator name iqn.initiator-server-3 MDS2(config-iscsi-init)# vsan 10

We can configure initiators to be in any VSAN. This also means that the same iSCSI portal can be using resources from different VSANs. In each VSAN the switch will create a virtual N-port when necessary.

MDS2

MDS2(config-iscsi-init)# vsan data MDS2(config-vsan-db)# vsan 20 interface iscsi1/1



To overcome the limitation that we can not configure VSAN 10 in the VSAN database configuration. We just enable the iSCSI portal interface in another VSAN. This has no impact on the working of the initiators. As the configuration under the initiator configuration is leading over the VSAN database configuration.

Next we need to give the initiators access to a virtual target.

MDS2

MDS2(config-iscsi-tgt)# sh device-alias database device-alias name J1D1 pwwn c6:83:00:11:0d:18:fa:ce device-alias name J1D2 pwwn c6:83:00:11:0d:18:fb:ce device-alias name J1D3 pwwn c6:83:00:11:0d:18:fc:ce device-alias name J1D4 pwwn c6:83:00:11:0d:18:fd:ce device-alias name J2D1 pwwn 21:00:00:11:0d:18:fe:ce device-alias name J2D2 pwwn 21:00:00:11:0d:18:ff:ce device-alias name J2D3 pwwn 21:00:00:11:0d:18:a0:ce device-alias name J2D4 pwwn 21:00:00:11:0d:18:a1:ce Total number of entries = 8 MDS2(config-iscsi-tgt)# iscsi virtual-target name iqn.jbod-disk-J1D1 MDS2(config-iscsi-tgt)# pwwn c6:83:00:11:0d:18:fa:ce MDS2(config-iscsi-tgt)# initiator ? WORD Enter iSCSI initiator name (Max Size 223) ip Allow iSCSI initiator access to this target by ip address MDS2(config-iscsi-tgt)# initiator iqn.initiator-server-1 permit MDS2(config-iscsi-tgt)# initiator iqn.initiator-server-2 permit MDS2(config-iscsi-tgt)# initiator iqn.initiator-server-3 permit MDS2(config-iscsi-tgt)#

After configuring the virtual-target we can verify our target and initiator configuration.

MDS2(config-iscsi-tgt)# sh iscsi virtual-target target: iqn.jbod-disk-j1d1 Port WWN c6:83:00:11:0d:18:fa:ce Configured node (iSCSI) No. of initiators permitted: 3 initiator iqn.initiator-server-1 is permitted initiator iqn.initiator-server-2 is permitted initiator iqn.initiator-server-3 is permitted All initiator permit is disabled Trespass support is disabled Revert to primary support is disabled MDS2(config-iscsi-tgt)# MDS2(config-iscsi-tgt)#

We can see that the target only allows certain initiators to connect to it and that it is connected to the pWWN of J1D1.

MDS2(config-iscsi-tgt)# MDS2(config-iscsi-tgt)# show iscsi initiator configured iSCSI Node name is iqn.initiator-server-1 Member of vsans: 10 Configured node (iSCSI) iSCSI Node name is iqn.initiator-server-2



Member of vsans: 10 Configured node (iSCSI) iSCSI Node name is iqn.initiator-server-3 Member of vsans: 10 Configured node (iSCSI) MDS2(config-iscsi-tgt)#

The initiators are configured so that they are members of VSAN 10.

Finally we need to enable zoning for these initiators and a target. Where we can only use 2 entries in the zoning.

This is relatively easy as we are using the proxy initiator mode, so we can use the pWWN of the proxy host and the device-alias of the J1D1 disk and we have our zoning completed.

MDS2(config-iscsi-tgt)# zone name ISCSI1 vsan 20 MDS2(config-zone)# zone name ISCSI1 vsan 10 MDS2(config-zone)# member device-alias J1D1 MDS2(config-zone)# member pwwn aa:bb:cc:dd:ee:ff:00:11 MDS2(config-zone)# zoneset name ZS1 vsan 10 MDS2(config-zoneset)# member ISCSI1 MDS2(config-zoneset)# zoneset activate name ZS1 vsan 10 Zoneset activation initiated. check zone status MDS2(config)#



With this configuration we have our iSCSI configuration completed. In the next task we will be configuring the next version of the iSCSI feature on the MDS platform called iSLB!

Task 6: iSLB

This final task of this chapter will be about configuring iSLB. This feature is the next version of iSCSI with more features and has more features specifically related to high availability and load balancing between different MDS switches.

The iSLB feature has a couple unique features that are missing from the iSCSI implementation, which might make it hard to roll-out this feature for hundreds of iSCSI initiators.

The scaling is accomplished by the following features:

Zoning can be automatically done with auto-zoning

By the use of auto-zoning you are able to automatically zone the initiator and virtual-target together without having to manually configure the zoning. The combination of the initiator and virtual-target is done by the use of so-called ‘initiator targets’. This means that under the initiator configuration the virtual-target configuration is done.

Distribution using CFS is possible

By using CFS distribution for this feature you can push the configuration of the iSLB feature to multiple MDS switches in the fabric. This eases the configuration and ensures that you have equal configurations for initiators and targets across the whole fabric.

Load balancing of initiator requests across MDS switches using VRRP and iSCSI redirect

When you are using VRRP, there is a single switch that will receive all iSCSI login requests. The iSLB feature implements iSCSI login redirection so the login request is redirected to the other MDS switch which is also participating in the VRRP configuration.



Keep in mind that you need to keep iSCSI enabled on the switch as well. This is required when you are using more than 200 iSLB initiators, but it’s always a best practice to follow.

iSLB can be using dynamic initiators and targets just like iSCSI. By default the auto learning is using the iSCSI feature, but you can change this to use the iSLB feature instead.

When the initiators or targets are already learned through the iSCSI feature you are not able to convert these to iSLB variants. You need to remove them and re-discover the initiators and targets.

Again in our task we are not allowed to use any dynamic learning, just like for iSCSI.

The initial questions in this task are related to building up the iSLB portal on a different interface than were we configured the iSCSI portal.

When we read the first 3 questions we have enough information to start building our portal. We can only configure MDS2 for configuration changes to initiators and targets. Which means that we should be using distribution of configuration using the CFS protocol.

The third question indicates that the portal should be high available. Where the only option we have is to use the VRRP.

Let’s start the configuration on our switches:

MDS1

MDS1(config)# feature iscsi MDS1(config)# iscsi enable module 1 MDS1(config)# iscsi enable MDS1(config)# int gig1/2 MDS1(config-if)# no shut MDS1(config-if)# int gi1/2.72 MDS1(config-if)# ip add 198.18.72.9 255.255.255.0 MDS1(config-if)# no shut MDS1(config-if)# vrrp 72 MDS1(config-if-vrrp)# address 198.18.72.1 MDS1(config-if-vrrp)# prio 110 MDS1(config-if-vrrp)# no shut MDS1(config-if-vrrp)# int iscsi1/2 MDS1(config-if)# no shut MDS1(config-vsan-db)# vsan 10 interface iscsi1/2 Traffic on iscsi1/2 may be impacted. Do you want to continue? (y/n) [n] y MDS1(config-vsan-db)#

We first enable the feature and configure the VRRP group. We make MDS1 the primary switch. Which switch is the primary, is not given in this case in the question, so we can take any switch we want for this use. Do not forget to put the iSCSI interface into another VSAN, because VSAN 1 is still suspended and interface in that VSAN will not come online unless they are changed to another VSAN which is unsuspended.



The question about the configuration that can only be done on MDS2, is about enabling the distribution of the iSLB feature using the CFS protocol. This should be enabled.

MDS1

MDS1(config)# islb distribute

Now we can verify our configuration.

MDS1(config-if)# sh vrrp Interface VR IpVersion Pri Time Pre State VR IP addr --------------------------------------------------------------------------- GigE1/2.72 72 IPv4 110 1 s master 198.18.72.1 MDS1(config-if)# sh islb ? *** No matching command found in current mode, matching in (exec) mode *** cfs-session Show iSLB cfs session information initiator Show iSLB initiator information merge Show iSLB merge information pending Show iSLB pending configuration pending-diff Show iSLB pending configuration diffs session Show iSLB session information status Show iSLB CFS status virtual-target Show iSLB virtual-targets vrrp Show iSLB vrrp load balance information MDS1(config-if)# sh islb status iSLB Distribute: Enabled iSLB CFS Session: Does not exist Number of load balanced VRRP groups: 0 Number of load-balanced initiators: 0 MDS1(config-if)#

Now MDS2 should be configured for the same features.

MDS2

MDS2(config)# feature iscsi MDS2(config)# iscsi enable module 1 MDS2(config)# iscsi enable MDS2(config)# int gig1/2 MDS2(config-if)# no shut MDS2(config-if)# int gi1/2.72 MDS2(config-if)# ip add 198.18.72.11 255.255.255.0 MDS2(config-if)# no shut MDS2(config-if)# vrrp 72 MDS2(config-if-vrrp)# address 198.18.72.1 MDS2(config-if-vrrp)# no shut MDS2(config-if-vrrp)# int iscsi1/2 MDS2(config-if)# no shut MDS2(config-if)# islb distribute MDS2(config-if)# vsan database MDS2(config-vsan-db)# vsan 10 interface iscsi1/2 Traffic on iscsi1/2 may be impacted. Do you want to continue? (y/n) [n] y MDS2(config-vsan-db)#



The next step in our task is that we need to enable the load balancing feature on the iSLB activated switches. The load balancing feature only works in conjunction with the VRRP. It is based on a ‘weight’ that is given to an initiator. The load balancing is always using the number of initiators to load balance, not to the number of sessions on the switch.

You can specify a weight to an initiator to take care of the load balancing. When a host has a lower bandwidth connection, it can be assigned a lower weight, as the amount of load it generates to the MDS switch is lower than other servers. So the MDS can take more of these smaller bandwidth servers on one interface.

Another use case for using load balancing is when there are certain initiators that you know are going to set-up a lot of iSCSI sessions. Therefore more load is generated on the MDS and the weight number should take care of that.

The default value is 1000.

Load balancing is enabled with only a single command.

MDS1

MDS1(config)# islb ? abort Abort pending iSLB configuration commit Commit pending iSLB configuration distribute Enable CFS for iSLB configuration initiator Configure iSLB initiator save-initiator Make WWNs for iSLB initiator persistent virtual-target Configure iSLB virtual-target vrrp Configure iSLB on this vrrp group zoneset Configure iSCSI zoneset activate MDS1(config)# islb vrrp ? <1-255> IPv4 VR group number ipv6 Configure vrrp IPv6 on this interface MDS1(config)# islb vrrp 72 ? load-balance Enable load-balancing on this vrrp group MDS1(config)# islb vrrp 72 load-balance MDS1(config)# islb commit

The load balancing can be monitored using the following output. All load balancing configuration is distributed using the CFS Protocol, so both switches are aware of all changes going on in the load balancing mechanism. This ensure that when the master fails, which keeps track of the iSCSI login redirects, the backup switch will have all information regarding the load balancing and is able to successfully take over the working of the switches. Another redirect is sent to the iSCSI initiators and sessions are automatically re-initiated.

MDS1(config)# sh islb vrrp summ -- Groups For Load Balance -- -------------------------------------------------------------------------------- VR Id VRRP Address Type Configured Status



-------------------------------------------------------------------------------- 72 IPv4 Enabled -- Interfaces For Load Balance -- -------------------------------------------------------------------------------- Initiator Redirect VR Id VRRP IP Switch WWN Interface Load Enabled -------------------------------------------------------------------------------- M 72 198.18.72.1 20:00:00:0d:ec:54:a9:80 GigE1/2.72 0 Yes 72 198.18.72.1 20:00:00:0d:ec:58:b1:aa GigE1/2.72 0 Yes -- Initiator To Interface Assignment -- -------------------------------------------------------------------------------- Initiator VR Id VRRP IP Switch WWN Interface -------------------------------------------------------------------------------- MDS1(config)# sh islb vrrp interface -- Interfaces For Load Balance -- Interface GigabitEthernet1/2.72 Switch wwn: 20:00:00:0d:ec:54:a9:80 VRRP group id: 72, VRRP IP address: 198.18.72.1 Interface VRRP state: master Interface load: 0 Interface redirection: enabled Group redirection: enabled Number of physical IP address: 1 (1) 198.18.72.9 Port vsan: 10 Forwarding mode: store-and-forward Proxy initiator mode: disabled iSCSI authentication: CHAP or None Interface GigabitEthernet1/2.72 Switch wwn: 20:00:00:0d:ec:58:b1:aa VRRP group id: 72, VRRP IP address: 198.18.72.1 Interface VRRP state: master Interface load: 0 Interface redirection: enabled Group redirection: enabled Number of physical IP address: 1 (1) 198.18.72.11 Port vsan: 10 Forwarding mode: store-and-forward Proxy initiator mode: disabled iSCSI authentication: CHAP or None MDS1(config)#

The master switch (VRRP master) will always redirect the first request towards the backup switch. This is because the master switch always has more load, because it needs to handle all iSCSI login requests. Therefore it automatically gives itself an invisible weight of 1000. This means that when 2 requests with a weight of 1000 come in to the switch it will redirect both to the backup switch. As the master is always less busy than the backup switches. This means that in case the load of the switches is equal, the request is always redirected to the backup.



The next task is only a key thing to remember that with the following questions there are no manual zoning changes allowed. This means that we should be using the auto-zoning feature instead. This also means that we should be using ‘initiator-targets’, as this is the only place where the auto-zoning will work. When you only configure an initiator and a virtual-target, these will not be auto-zoned together, because the switch does not know they belong to each other. Therefore we will be using the initiator target feature.

We will first configure as the next question states, the 5 initiators. Remember that we are only allowed to configure MDS2 for initiators and targets.

MDS2

MDS2(config)# islb initiator name iqn.islb-initiator-host-1 MDS2(config-islb-init)# ? metric Assign load balancing metric for the initiator mutual-chap Assign username for initiator's challenge (mutual chap) no Negate a command or set its defaults static Create static nWWN or pWWN for the initiator target Configure iSLB initiator target username Assign username for iSLB login authentication vsan Assign VSAN membership for the initiator zonename Assign zone name for the initiator targets end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in MDS2(config-islb-init)# static nwwn system-assign MDS2(config-islb-init)# static pwwn system-assign 2

We need to allocate system assigned nWWNs and 2 pWWNs to the initiators. Just like we are able when configuring iSCSI initiators.

MDS2

MDS2(config-islb-init)# zonename ? WORD Enter zone name (Max Size 55) MDS2(config-islb-init)# zonename islb-ipexpert-host-1 Configuring iSLB node iqn.islb-initiator-host-1 fails: Illegal character present in the name (0x40200024) MDS2(config-islb-init)# zonename IPexpertHost1 MDS2(config-islb-init)# islb commit

As the next question dictates we need to use zone names. By default the switch will generate either a long hash or a name with ‘iSLB’ in it. You are able to make it more readable, by configuring the zone name under the initiator. You are not allowed to use any dashes in the zone names. Only underscores are allowed as special character.

After the commit has been initiated, the initiator is shown in the configuration. There you can see that the MDS allocated a nWWN and 2 pWWNs to this initiator.

MDS2(config)# sh run | section "islb initiator"



islb initiator name iqn.islb-initiator-host-1 static nWWN 21:1b:00:0d:ec:54:a9:82 static pWWN 21:1c:00:0d:ec:54:a9:82 static pWWN 21:1d:00:0d:ec:54:a9:82 zonename IPexpertHost1 MDS2(config)# sh islb initiator configured iSCSI Node name is iqn.islb-initiator-host-1 Node WWN is 21:1b:00:0d:ec:54:a9:82 No. of PWWN: 2 Port WWN is 21:1c:00:0d:ec:54:a9:82 Port WWN is 21:1d:00:0d:ec:54:a9:82 Configured node (iSLB) Zone Name for initiator targets: IPexpertHost1 Load Balance Metric: 1000 Number of Initiator Targets: 0 MDS2(config)#

The other hosts can be configured using the same commands.

MDS2

MDS2(config)# islb initiator name iqn.islb-initiator-host-2 MDS2(config-islb-init)# static nwwn system-assign MDS2(config-islb-init)# static pwwn system-assign 2 MDS2(config-islb-init)# zonename IPexpertHost2 MDS2(config-islb-init)# islb initiator name iqn.islb-initiator-host-3 MDS2(config-islb-init)# static nwwn system-assign MDS2(config-islb-init)# static pwwn system-assign 2 MDS2(config-islb-init)# zonename IPexpertHost3 MDS2(config-islb-init)# islb initiator name iqn.islb-initiator-host-4 MDS2(config-islb-init)# static nwwn system-assign MDS2(config-islb-init)# static pwwn system-assign 2 MDS2(config-islb-init)# zonename IPexpertHost4 MDS2(config-islb-init)# islb initiator name iqn.islb-initiator-host-5 MDS2(config-islb-init)# static nwwn system-assign MDS2(config-islb-init)# static pwwn system-assign 2 MDS2(config-islb-init)# zonename IPexpertHost5 MDS2(config-islb-init)# islb commit MDS2(config)#

The next question in our task is that Host 3 is a database server with a lot of iSCSI connections. As we discussed earlier. The load balancing that iSLB does is based on the number of initiators that make a connection with the switch. This particular initiator will have a lot of iSCSI sessions with the switch, meaning that it will generate a lot more load on the switch compared to other initiators.

We do not know how many sessions are set-up compared to other initiators, but we just need to ensure that the load balancing takes care of this heavier initiator. Therefore we will be configuring a little higher load.

MDS2

MDS2(config)# islb initiator name iqn.islb-initiator-host-3 MDS2(config-islb-init)# metric 2000 MDS2(config-islb-init)# islb commit



Now we can verify our initiator configuration and see that everything configured has been properly pushed to all switches in the fabric.

MDS1(config)# show islb initiator config iSCSI Node name is iqn.islb-initiator-host-1 Node WWN is 21:01:00:05:9b:70:5a:42 No. of PWWN: 2 Port WWN is 21:02:00:05:9b:70:5a:42 Port WWN is 21:03:00:05:9b:70:5a:42 Configured node (iSLB) Zone Name for initiator targets: IPexpertHost1 Load Balance Metric: 1000 Number of Initiator Targets: 0 iSCSI Node name is iqn.islb-initiator-host-2 Node WWN is 21:04:00:05:9b:70:5a:42 No. of PWWN: 2 Port WWN is 21:05:00:05:9b:70:5a:42 Port WWN is 21:06:00:05:9b:70:5a:42 Configured node (iSLB) Zone Name for initiator targets: IPexpertHost2 Load Balance Metric: 1000 Number of Initiator Targets: 0 iSCSI Node name is iqn.islb-initiator-host-3 Node WWN is 21:07:00:05:9b:70:5a:42 No. of PWWN: 2 Port WWN is 21:08:00:05:9b:70:5a:42 Port WWN is 21:09:00:05:9b:70:5a:42 Configured node (iSLB) Zone Name for initiator targets: IPexpertHost3 Load Balance Metric: 2000 Number of Initiator Targets: 0 iSCSI Node name is iqn.islb-initiator-host-4 Node WWN is 21:0a:00:05:9b:70:5a:42 No. of PWWN: 2 Port WWN is 21:0b:00:05:9b:70:5a:42 Port WWN is 21:0c:00:05:9b:70:5a:42 Configured node (iSLB) Zone Name for initiator targets: IPexpertHost4 Load Balance Metric: 1000 Number of Initiator Targets: 0 iSCSI Node name is iqn.islb-initiator-host-5 Node WWN is 21:0d:00:05:9b:70:5a:42 No. of PWWN: 2 Port WWN is 21:0e:00:05:9b:70:5a:42 Port WWN is 21:0f:00:05:9b:70:5a:42 Configured node (iSLB) Zone Name for initiator targets: IPexpertHost5 Load Balance Metric: 1000 Number of Initiator Targets: 0 MDS1(config)#



The next 2 questions state that we need to configure the first iSLB virtual-target, however we are not allowed to use the command required for this. Remember that you also can not use any zoning changes to allow the traffic of the iSCSI initiator to the FC target.

In this question we will be using the so-called ‘initiator targets’. These are targets that are directly configured under the iSLB initiator and automatically only this initiator can access the target. This eases the configuration of your initiators and it adds another option for flexibility when you are configuring some disk for just one initiator. It will automatically arrange the zoning for you by using the ‘auto-zoning’ feature.

In this case it’s not really efficient to use the initiator-target feature, because we have a single target that is reached by multiple initiators.

Read ahead as you need to make this target high available with another target and use LUN masking!

MDS2

MDS2(config)# islb initiator name iqn.islb-initiator-host-1 MDS2(config-islb-init)# ? metric Assign load balancing metric for the initiator mutual-chap Assign username for initiator's challenge (mutual chap) no Negate a command or set its defaults static Create static nWWN or pWWN for the initiator target Configure iSLB initiator target username Assign username for iSLB login authentication vsan Assign VSAN membership for the initiator zonename Assign zone name for the initiator targets end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in MDS2(config-islb-init)# target device-alias J2D2 vsan 10 ? <CR> fc-lun Fc-lun of the fc-target iqn-name Name of the target no-zone No automatic zoning revert-primary-port Revert back to primary port when it comes back up sec-device-alias Device-alias of the secondary fc-target sec-pwwn PWWN of the secondary fc-target trespass Enable trespass support

We need to configure this target in VSAN 10. Do not forget to add the initiator to VSAN 10 as well. Either by configuring the iSCSI interface inside this VSAN, which we already did, or by configuring the VSAN command under the initiator.

When you do not do this, the initiator will not be active in this zone.

MDS2

MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 ?



iscsi-lun ISCSI-lun of the target MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa ? <CR> iqn-name Name of the target no-zone No automatic zoning revert-primary-port Revert back to primary port when it comes back up sec-device-alias Device-alias of the secondary fc-target sec-pwwn PWWN of the secondary fc-target trespass Enable trespass support MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa sec-device-alias J2D2 ? MDS2(config-islb-init)# MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa sec-device-alias J1D2 sec-lun 0x0 ? <CR> iqn-name Name of the target no-zone No automatic zoning revert-primary-port Revert back to primary port when it comes back up sec-vsan Assign VSAN membership for the initiator target trespass Enable trespass support MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 ? <CR> - Range separator iqn-name Name of the target no-zone No automatic zoning revert-primary-port Revert back to primary port when it comes back up

Now we configured the required LUN masking. The FC LUN 0x0 should be seen as 0xA.

MDS2

MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x1 iscsi-lun 0xb sec-device-alias J1D2 sec-lun 0x1 sec-vsan 10 iqn-name iqn.jbod-disk2-islb

You can configure the same IQN name, when you only change the LUN masking configuration. This way your target is visible with 2 LUNs. This will also be visible in the show outputs.

MDS2

MDS2(config-islb-init)# islb zoneset activate MDS2(config)# islb commit

The final command required is to make the auto-zoning active. This is not strictly necessary as with CFS distribution enabled, the auto-zoning is automatically activated as soon as the commit command is entered.

The commands for the initiator-targets are very lengthy, which might cause issues with your terminal emulator. When you are running into issues, you can enable a longer terminal width.

MDS2(config)# sh run | section islb islb distribute islb initiator name iqn.islb-initiator-host-1 static nWWN 21:01:00:05:9b:70:5a:42 static pWWN 21:02:00:05:9b:70:5a:42



static pWWN 21:03:00:05:9b:70:5a:42 vsan 10 zonename IPexpertHost1 target pwwn c6:83:00:11:0d:18:fb:ce vsan 10 fc-lun 0x0000 iscsi-lun 0x000a sec -pwwn c6:83:00:11:0d:18:fb:ce sec-vsan 10 sec-lun 0x0000 iqn-name iqn.jbod-disk2 -islb target pwwn c6:83:00:11:0d:18:fb:ce vsan 10 fc-lun 0x0001 iscsi-lun 0x000b sec -pwwn c6:83:00:11:0d:18:fb:ce sec-vsan 10 sec-lun 0x0001 iqn-name iqn.jbod-disk2 -islb islb initiator name iqn.islb-initiator-host-2 static nWWN 21:04:00:05:9b:70:5a:42 static pWWN 21:05:00:05:9b:70:5a:42 static pWWN 21:06:00:05:9b:70:5a:42 zonename IPexpertHost2 islb initiator name iqn.islb-initiator-host-3 static nWWN 21:07:00:05:9b:70:5a:42 static pWWN 21:08:00:05:9b:70:5a:42 static pWWN 21:09:00:05:9b:70:5a:42 zonename IPexpertHost3 metric 2000 islb initiator name iqn.islb-initiator-host-4 static nWWN 21:0a:00:05:9b:70:5a:42 static pWWN 21:0b:00:05:9b:70:5a:42 static pWWN 21:0c:00:05:9b:70:5a:42 zonename IPexpertHost4 islb initiator name iqn.islb-initiator-host-5 static nWWN 21:0d:00:05:9b:70:5a:42 static pWWN 21:0e:00:05:9b:70:5a:42 static pWWN 21:0f:00:05:9b:70:5a:42 zonename IPexpertHost5 islb commit islb vrrp 71 load-balance MDS2(config)#

Although the device-alias mode is enhanced. They are not kept in the configuration, but are translated to the pWWN of the device-alias.

MDS2(config)# show islb initiator config iSCSI Node name is iqn.islb-initiator-host-1 Node WWN is 21:01:00:05:9b:70:5a:42 No. of PWWN: 2 Port WWN is 21:02:00:05:9b:70:5a:42 Port WWN is 21:03:00:05:9b:70:5a:42 Configured node (iSLB) Zone Name for initiator targets: IPexpertHost1 Load Balance Metric: 1000 Number of Initiator Targets: 1 Initiator Target: iqn.jbod-disk2-islb Port WWN c6:83:00:11:0d:18:fb:ce [J1D2] Secondary PWWN c6:83:00:11:0d:18:fb:ce [J1D2] Primary PWWN VSAN 10 Secondary PWWN VSAN 10 No. of LU mapping: 2 iSCSI LUN: 0x000a, FC LUN: 0x0000, Secondary FC LUN: 0x0000 iSCSI LUN: 0x000b, FC LUN: 0x0001, Secondary FC LUN: 0x0001 Zoning support is enabled Trespass support is disabled Revert to primary support is disabled



This verification command shows you in summary how the initiator target is automatically configuring zones and how the 2 LUNs are visible and summarized in this single target.

Now we can configure the remaining initiators for the same target.

MDS2

MDS2(config)# islb initiator name iqn.islb-initiator-host-2 MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb vsan 10 fc-lun 0x1 iscsi-Configuring iSLB initiator target with device-alias J2D2 (VSAN 10) and sec-device-alias J1D2 (VSAN 10) fails: Initiator target name invalid or duplicate (0x403c0044)

Now although this target is only available for the initiator where it is configured. You still can not have overlapping symbolic-names in the configuration! Therefore we will add the host number to the symbolic-name.

MDS2

MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb2 MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x1 iscsi-lun 0xb sec-device-alias J1D2 sec-lun 0x1 sec-vsan 10 iqn-name iqn.jbod-disk2-islb2 MDS2(config-islb-init)# vsan 10 MDS2(config-islb-init)# exit MDS2(config)# islb initiator name iqn.islb-initiator-host-3 MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb3 MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x1 iscsi-lun 0xb sec-device-alias J1D2 sec-lun 0x1 sec-vsan 10 iqn-name iqn.jbod-disk2-islb3 MDS2(config-islb-init)# vsan 10 MDS2(config-islb-init)# exit MDS2(config)# islb initiator name iqn.islb-initiator-host-4 MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb4 MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x1 iscsi-lun 0xb sec-device-alias J1D2 sec-lun 0x1 sec-vsan 10 iqn-name iqn.jbod-disk2-islb4 MDS2(config-islb-init)# vsan 10 MDS2(config-islb-init)# exit MDS2(config)# islb initiator name iqn.islb-initiator-host-5 MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb5 MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x1 iscsi-lun 0xb sec-device-alias J1D2 sec-lun 0x1 sec-vsan 10 iqn-name iqn.jbod-disk2-islb5 MDS2(config-islb-init)# vsan 10 MDS2(config-islb-init)# islb commit

Now we can verify our lengthy configuration!

MDS2(config)# show islb initiator configured iSCSI Node name is iqn.islb-initiator-host-1 Member of vsans: 10 Node WWN is 21:01:00:05:9b:70:5a:42 No. of PWWN: 2 Port WWN is 21:02:00:05:9b:70:5a:42 Port WWN is 21:03:00:05:9b:70:5a:42 Configured node (iSLB) Zone Name for initiator targets: IPexpertHost1 Load Balance Metric: 1000



Number of Initiator Targets: 1 Initiator Target: iqn.jbod-disk2-islb Port WWN c6:83:00:11:0d:18:fb:ce [J1D2] Secondary PWWN c6:83:00:11:0d:18:fb:ce [J1D2] Primary PWWN VSAN 10 Secondary PWWN VSAN 10 No. of LU mapping: 2 iSCSI LUN: 0x000a, FC LUN: 0x0000, Secondary FC LUN: 0x0000 iSCSI LUN: 0x000b, FC LUN: 0x0001, Secondary FC LUN: 0x0001 Zoning support is enabled Trespass support is disabled Revert to primary support is disabled iSCSI Node name is iqn.islb-initiator-host-2 Node WWN is 21:04:00:05:9b:70:5a:42 No. of PWWN: 2 Port WWN is 21:05:00:05:9b:70:5a:42 Port WWN is 21:06:00:05:9b:70:5a:42 Configured node (iSLB) Zone Name for initiator targets: IPexpertHost2 Load Balance Metric: 1000 Number of Initiator Targets: 1 Initiator Target: iqn.jbod-disk2-islb2 Port WWN 21:00:00:11:0d:18:ff:ce [J2D2] Secondary PWWN c6:83:00:11:0d:18:fb:ce [J1D2] Primary PWWN VSAN 10 Secondary PWWN VSAN 10 No. of LU mapping: 2 iSCSI LUN: 0x000a, FC LUN: 0x0000, Secondary FC LUN: 0x0000 iSCSI LUN: 0x000b, FC LUN: 0x0001, Secondary FC LUN: 0x0001 Zoning support is enabled Trespass support is disabled Revert to primary support is disabled iSCSI Node name is iqn.islb-initiator-host-3 Member of vsans: 10 Node WWN is 21:07:00:05:9b:70:5a:42 No. of PWWN: 2 Port WWN is 21:08:00:05:9b:70:5a:42 Port WWN is 21:09:00:05:9b:70:5a:42 Configured node (iSLB) Zone Name for initiator targets: IPexpertHost3 Load Balance Metric: 2000 Number of Initiator Targets: 1 Initiator Target: iqn.jbod-disk2-islb3 Port WWN 21:00:00:11:0d:18:ff:ce [J2D2] Secondary PWWN c6:83:00:11:0d:18:fb:ce [J1D2] Primary PWWN VSAN 10 Secondary PWWN VSAN 10 No. of LU mapping: 2 iSCSI LUN: 0x000a, FC LUN: 0x0000, Secondary FC LUN: 0x0000 iSCSI LUN: 0x000b, FC LUN: 0x0001, Secondary FC LUN: 0x0001 Zoning support is enabled Trespass support is disabled Revert to primary support is disabled



iSCSI Node name is iqn.islb-initiator-host-4 Member of vsans: 10 Node WWN is 21:0a:00:05:9b:70:5a:42 No. of PWWN: 2 Port WWN is 21:0b:00:05:9b:70:5a:42 Port WWN is 21:0c:00:05:9b:70:5a:42 Configured node (iSLB) Zone Name for initiator targets: IPexpertHost4 Load Balance Metric: 1000 Number of Initiator Targets: 1 Initiator Target: iqn.jbod-disk2-islb4 Port WWN 21:00:00:11:0d:18:ff:ce [J2D2] Secondary PWWN c6:83:00:11:0d:18:fb:ce [J1D2] Primary PWWN VSAN 10 Secondary PWWN VSAN 10 No. of LU mapping: 2 iSCSI LUN: 0x000a, FC LUN: 0x0000, Secondary FC LUN: 0x0000 iSCSI LUN: 0x000b, FC LUN: 0x0001, Secondary FC LUN: 0x0001 Zoning support is enabled Trespass support is disabled Revert to primary support is disabled iSCSI Node name is iqn.islb-initiator-host-5 Member of vsans: 10 Node WWN is 21:0d:00:05:9b:70:5a:42 No. of PWWN: 2 Port WWN is 21:0e:00:05:9b:70:5a:42 Port WWN is 21:0f:00:05:9b:70:5a:42 Configured node (iSLB) Zone Name for initiator targets: IPexpertHost5 Load Balance Metric: 1000 Number of Initiator Targets: 1 Initiator Target: iqn.jbod-disk2-islb5 Port WWN 21:00:00:11:0d:18:ff:ce [J2D2] Secondary PWWN c6:83:00:11:0d:18:fb:ce [J1D2] Primary PWWN VSAN 10 Secondary PWWN VSAN 10 No. of LU mapping: 2 iSCSI LUN: 0x000a, FC LUN: 0x0000, Secondary FC LUN: 0x0000 iSCSI LUN: 0x000b, FC LUN: 0x0001, Secondary FC LUN: 0x0001 Zoning support is enabled Trespass support is disabled Revert to primary support is disabled MDS2(config)#

Now we have configured our 2 LUNs to be highly available using auto-zoning in all configured initiators!

The next 2 questions indicate another target that now should only be created under iqn.islb-initiator-host-3. We don’t need to be worried about the high availability as it will be by default!

However we can not use auto-zoning!

MDS2

MDS2(config-vsan-db)# islb initiator name iqn.islb-initiator-host-3



MDS2(config-islb-init)# target device-alias J1D1 ? <CR> fc-lun Fc-lun of the fc-target iqn-name Name of the target no-zone No automatic zoning revert-primary-port Revert back to primary port when it comes back up sec-device-alias Device-alias of the secondary fc-target sec-pwwn PWWN of the secondary fc-target trespass Enable trespass support vsan Assign VSAN membership for the initiator target MDS2(config-islb-init)# MDS2(config-islb-init)# target device-alias J1D1 vsan 20 no-zone iqn-name iqn.islb-jbod1-disk1 Configuring iSLB initiator target with device-alias J1D1 (VSAN 20) fails: Initiator target's primary pWWN is not in the same VSAN as the initiator (0x403c0060)

When you are configuring the initiator target, you must first configure the VSAN where the initiator resides, before it will accept the command. You need to make the initiator member of all VSANs that it needs to access targets in.

MDS2

MDS2(config-islb-init)# vsan 20 MDS2(config-islb-init)# target device-alias J1D1 vsan 20 no-zone iqn-name iqn.islb-jbod1-disk1 MDS2(config-islb-init)# islb commit

Now we are not finished as we still need to create the zoning for this new initiator target.

Here we can not use the IP address or the Symbolic Name of the initiator in the zone. This means we need to use the statically assigned pWWN values or the nWWN. In this case we will use the pWWN that is assigned by the MDS switch.

MDS2

MDS2(config)# zoneset name ZS1 v 20 Enhanced zone session has been created. Please 'commit' the changes when done. MDS2(config-zoneset)# zone name ISLB_1 MDS2(config-zoneset-zone)# show islb initiator configured iqn.islb-initiator-host-3 iSCSI Node name is iqn.islb-initiator-host-3 Member of vsans: 10, 20 Node WWN is 21:07:00:05:9b:70:5a:42 No. of PWWN: 2 Port WWN is 21:08:00:05:9b:70:5a:42 Port WWN is 21:09:00:05:9b:70:5a:42 Configured node (iSLB) Zone Name for initiator targets: IPexpertHost3 Load Balance Metric: 2000 Number of Initiator Targets: 2 Initiator Target: iqn.jbod-disk2-islb3 Port WWN 21:00:00:11:0d:18:ff:ce [J2D2] Secondary PWWN c6:83:00:11:0d:18:fb:ce [J1D2]



Primary PWWN VSAN 10 Secondary PWWN VSAN 10 No. of LU mapping: 2 iSCSI LUN: 0x000a, FC LUN: 0x0000, Secondary FC LUN: 0x0000 iSCSI LUN: 0x000b, FC LUN: 0x0001, Secondary FC LUN: 0x0001 Zoning support is enabled Trespass support is disabled Revert to primary support is disabled MDS2(config-zoneset-zone)# member device J1D1 MDS2(config-zoneset-zone)# member pwwn 21:08:00:05:9b:70:5a:42 MDS2(config-zoneset-zone)# member pwwn 21:09:00:05:9b:70:5a:42 MDS2(config-zoneset-zone)# zoneset activate name ZS1 v 20 MDS2(config)# zone commit v 20

When you are using the pWWN for the zoning, keep in mind to add both of the pWWNs as the initiator will get one out of 2 assigned, so you have to take care that either one can be the active pWWN at that moment.

Now the final question of this task and this chapter is to configure authentication for all initiators. This means that we need to configure the CHAP authentication on the initiator.



MDS2

MDS2(config)# islb initiator name iqn.islb-initiator-host-1 MDS2(config-islb-init)# username host-1 MDS2(config-islb-init)# islb initiator name iqn.islb-initiator-host-2 MDS2(config-islb-init)# username host-2 MDS2(config-islb-init)# islb initiator name iqn.islb-initiator-host-3 MDS2(config-islb-init)# username host-3 MDS2(config-islb-init)# islb initiator name iqn.islb-initiator-host-4 MDS2(config-islb-init)# username host-4 MDS2(config-islb-init)# islb initiator name iqn.islb-initiator-host-5 MDS2(config-islb-init)# username host-5 MDS2(config-islb-init)# exit MDS2(config)# username host-1 password iSLBpassw0rd iscsi MDS2(config)# username host-2 password iSLBpassw0rd iscsi MDS2(config)# username host-3 password iSLBpassw0rd iscsi MDS2(config)# username host-4 password iSLBpassw0rd iscsi MDS2(config)# username host-5 password iSLBpassw0rd iscsi

To finish we create the local authentication database entries, just like was required for the iSCSI configuration.

Here we finished the chapter about Storage Networking Extensions over IP!

The next Chapter will do a deep dive into the subject of Unified Fabric. Where we will focus on Fibre Channel technologies over Ethernet. Which are very different than the FCIP technologies that we used in this chapter.

Take a break and be fresh to start the next chapter!



Chapter 7: Data Center Unified Fabric

Chapter 7: Data Unified Fabric is intended to let you be familiar with the Storage Networking features available on the Cisco Nexus switches and combined with the Cisco MDS switches.

This chapter will be about implementing FCoE features inside of the Nexus switches and the backwards compatibility with Native FC connections. Besides that we will be looking at N-Port Virtualization configurations..

We highly recommend creating your own diagram at the beginning of each lab so you are able to draw on your own diagram, making it much easier when you step into the real lab. Multiple topology drawings are available for this chapter.



Solutions

Task 1: Native Fibre Channel on Nexus

In this chapter we will be configuring several Unified Fabric features related to the Cisco Nexus switches and the Cisco MDS platform. During this chapter we will be configuring features related to transporting Fibre Channel across Ethernet connections. This technology is called Fibre Channel over Ethernet (FCoE).

The first questions in this initial task are to create some initial configuration. We need to make sure that we leave configurations in tact on MDS1 and MDS2, because we will be re-using the topology that we build in previous chapters.

The topology we will be using throughout this chapter is as the following drawing illustrates. We will not be using VDC’s in this topology, but only the native switching. Except for some special feature which we will be talking about in the next couple tasks.

The topology is using the 3 Nexus switches, the 2 MDS switches and the 2 JBOD arrays so we can create a Fibre Channel fabric in combination with the Ethernet network of our lab.

This combination is the Cisco best practice for Datacenter networking, where the Distribution switches (Nexus 5500) are connecting to both the Core switch (Nexus 7000) and Fibre Channel Fabric (MDS switches).



We will be re-using the MDS configurations except that we need to disable the GigabitEthernet interfaces as we do not need the FCIP connections anymore, which might interfere with the next questions.

MDS1

MDS1(config)# interface gig1/1 MDS1(config-if)# shutdown MDS1(config)# interface gig1/2 MDS1(config-if)# shutdown

MDS2

MDS2(config)# interface gig1/1 MDS2(config-if)# shutdown MDS2(config)# interface gig1/2 MDS2(config-if)# shutdown

Now we need to extend our Fibre Channel fabric with native interfaces towards the Nexus 5500 switches.

The Nexus 5500 is the only switch in the Nexus series that supports native Fibre Channel interfaces, besides the UCS 61xx fabric interconnects (which are not fully Nexus switches). The first generation of Nexus 5000 switch (Nexus 5010 and Nexus 5020) supported FC uplinks using dedicated uplink modules with either 10Gbps Ethernet (or FCoE) ports or dedicated 8Gbps Fibre Channel interfaces. These Fibre Channel interfaces are auto-switching for 8Gbps, 4Gbps, 2 Gbps and 1Gbps FC traffic, depending on the module (some only support up to 4Gbps).

With the second generation of Nexus 5000 series. Cisco introduced the concept of ‘universal ports’. These universal ports are by default enabled for Ethernet and FCoE traffic, but can be converted with a single command to native Fibre Channel interfaces. Of course you do need to change the SFP in the port, before it will actually work. Fibre Channel has different SFP’s than Ethernet SFP’s.

The interfaces are not licensed, but before you are able to use the Fibre Channel ports on the Nexus 5500 chassis, you need to have a Storage Protocols Services License. This license is valid for enabling both native Fibre Channel interfaces and FCoE on any of the ports in the chassis.

You need to enable the Fibre Channel ports in the switch and this is a disruptive change! Every additional Fibre Channel that you configure requires a reboot of the whole switch. This is because the ASIC that is controlling the ports needs to be re-initialized and reprogrammed for Fibre Channel traffic on that port.

There are limitations to which ports can be enabled for Fibre Channel traffic and which are not. It means that you need to configure Fibre Channel ports next to each other. It’s not possible to configure one interface as Ethernet and the next interface as Fibre Channel.



The following drawing illustrates an example in which some of the ports are enabled for Fibre Channel and other ports are enabled for Ethernet, which is how you might want to cluster certain ports that are serving a certain system.

The combination of FC and Ethernet is not possible to configure.

What is possible with the Unified Port technology is that the interfaces which can be configured for Fibre Channel are on the right side of the switch. Meaning that all first ports can be configured for Ethernet and the last ports for Fibre Channel.

This can not be mixed or turned around. There is no technical explanation for it, just was a design decision of Cisco to do it this way.

The port layout on the switch will be like the following drawing illustrates.

You are able to configure from a single port to all ports as Fibre Channel ports, there is no limitation to the amount of ports, just that they need to be next to each other.

Now we need to configure the Unified Ports on our switches.

Do not forget to do the reboot of the switch to make them active.

SW2

SW2(config)# slot 1 SW2(config-slot)# port 31-32 type fc SW2(config-slot)# wry [########################################] 100% SW2(config-slot)# reload



WARNING: This command will reboot the system Do you want to continue? (y/n) [n] y

SW3

SW3(config)# slot 1 SW3(config-slot)# port 31-32 type fc SW3(config-slot)# wr [########################################] 100% SW3(config-slot)# reload WARNING: This command will reboot the system Do you want to continue? (y/n) [n] y

After the reboot and you issue a show interface brief, you will see 2 new fc1/31 and fc1/32 in your configuration to be configured. The SFPs in the ports are already of the Fibre Channel type. So they will be immediately recognized and you are able to configure the native Fibre Channel connection to the MDS switches.

Now when we read ahead in the number of questions we see that the interfaces should be combined into a single connection as the FSPF protocol is only allowed to see a single connection in its database. This means that we need to configure a port-channel.

Just like it’s possible to create ethernet port-channels, it’s also possible to configure Fibre Channel port-channels just like on the MDS switches.

We first create our configuration for the MDS switches before the Nexus switches.

The configuration on the MDS switches is just equal to any other port, as the Nexus just acts as a standard Fibre Channel Director/Switch like all other MDS switches. NX-OS is also a rebranding of SAN-OS, so all FC code is still included in the OS and is not different in CLI than the MDS switches.

MDS1

MDS1(config)# interface fc1/13 MDS1(config-if)# switchport mode e MDS1(config-if)# channel-group 10 MDS1(config-if)# no shutdown MDS1(config-if)# interface fc1/14 MDS1(config-if)# switchport mode e MDS1(config-if)# channel-group 10 MDS1(config-if)# no shutdown MDS1(config-if)# interface port-channel10 MDS1(config-if)# switchport mode e MDS1(config-if)# switchport trunk mode on MDS1(config-if)# switchport trunk allowed vsan 10 MDS1(config-if)# switchport trunk allowed vsan add 20 MDS1(config-if)# no shutdown MDS1(config-if)# vsan database MDS1(config-vsan-db)# vsan 10 interface port-channel10

MDS2

MDS2(config)# interface fc1/13 MDS2(config-if)# channel-group 10



MDS2(config-if)# no shutdown MDS2(config-if)# interface fc1/14 MDS2(config-if)# channel-group 10 MDS2(config-if)# no shutdown MDS2(config-if)# interface port-channel10 MDS2(config-if)# switchport mode e MDS2(config-if)# switchport trunk mode on MDS2(config-if)# switchport trunk allowed vsan 10 MDS1(config-if)# switchport trunk allowed vsan add 20 MDS2(config-if)# no shutdown MDS2(config-if)# vsan database MDS2(config-vsan-db)# vsan 10 interface port-channel10

Do not forget to enable the port-channel interface in another VSAN, because we are still using the configuration from the other chapters the Default VSAN is still suspended and not able to enable ports in.

Then we configure the Nexus switches so the port-channels will come online.

SW2

SW2(config)# int fc1/31 SW2(config-if)# channel-group 10 command failed: port not compatible [port mode]

Interfaces that are set to their default auto-mode are not allowed to be configured in a port-channel. This means that we first need to manually set the port-mode before we can enable the port in a port-channel

SW2(config-if)# sw mode e SW2(config-if)# channel-group 10 fc1/31 added to port-channel 10 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring it up SW2(config-if)# no shut SW2(config-if)# int fc1/32 SW2(config-if)# sw mode e SW2(config-if)# channel-group 10 fc1/32 added to port-channel 10 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring it up SW2(config-if)# no shut SW2(config-if)# int port-channel10 Channel group 10 is already a SAN port channel ^ % Invalid command at '^' marker.



The port-channel configuration on the Nexus 5000 series switches is not equal to the MDS configuration for Ethernet and FC port-channels. You must use unique numbering for all port-channels (both Ethernet and FC), but you cannot configure them with the same configuration.

For FC port-channels you need to use the special command ‘san-port-channel’ instead of the regular port-channel command.

There you have the configuration available that you would expect to have with Fibre Channel port-channels.

SW2(config-if)# int san-port-channel 10 SW2(config-if)# sw mode e SW2(config-if)# sw trunk mode on SW2(config-if)# sw trunk allowed vsan 10 SW2(config-if)# sw trunk allowed vsan add 20 SW2(config-if)# no shut SW2(config-if)# vsan database SW2(config-vsan-db)# vsan 10 SW2(config-vsan-db)# vsan 20 SW2(config-vsan-db)# vsan 10 interface san-port-channel 10

SW3

SW3(config)# int fc1/31 SW3(config-if)# sw mode e SW3(config-if)# channel-group 10 fc1/31 added to port-channel 10 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring it up SW3(config-if)# no shut SW3(config-if)# int fc1/32 SW3(config-if)# sw mode e SW3(config-if)# channel-group 10 fc1/32 added to port-channel 10 and disabled please do the same operation on the switch at the other end of the port-channel, then do "no shutdown" at both ends to bring it up SW3(config-if)# no shut SW3(config-if)# int san-port-channel 10 SW3(config-if)# sw mode e SW3(config-if)# sw trunk mode on SW3(config-if)# sw trunk allowed vsan 10 SW3(config-if)# sw trunk allowed vsan add 20 SW3(config-if)# no shut SW3(config-if)# vsan database SW3(config-vsan-db)# vsan 10 SW3(config-vsan-db)# vsan 20 SW3(config-vsan-db)# vsan 10 interface san-port-channel 10

Now we configured both Nexus switches inside their own SAN Port-channel and gave them membership to another VSAN, just to keep the configuration equal to the MDS switches to prevent failures in the traffic when interfaces come online.

First we check if the interfaces are really seen in the Nexus chassis. This is an output of a pre-configured Nexus 5000 switch.

SW2(config-if)# do sh int brief



------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- fc2/1 4094 E on sfpAbsent -- -- 10 fc2/2 4094 E on sfpAbsent -- -- 10 fc2/3 4094 F on sfpAbsent -- -- -- fc2/4 4094 F on sfpAbsent -- -- -- fc2/5 4094 F on sfpAbsent -- -- -- fc2/6 4094 F on sfpAbsent -- -- -- fc2/7 4094 F on sfpAbsent -- -- -- fc2/8 4094 F on sfpAbsent -- -- -- -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth1/1 1 eth trunk up none 1000(D) 3 Eth1/2 1 eth access down SFP not inserted 10G(D) -- Eth1/3 1 eth access down SFP not inserted 10G(D) -- Eth1/4 1 eth access down SFP not inserted 10G(D) -- Eth1/5 1 eth access up none 10G(D) -- Eth1/6 1 eth access up none 10G(D) -- Eth1/7 1 eth trunk up none 10G(D) -- Eth1/8 1 eth access down SFP not inserted 10G(D) -- Eth1/9 1 eth trunk up none 10G(D) -- Eth1/10 1 eth trunk up none 10G(D) -- Eth1/11 1 eth access down SFP not inserted 10G(D) -- Eth1/12 1 eth access down SFP not inserted 10G(D) -- Eth1/13 1 eth access down SFP not inserted 10G(D) -- Eth1/14 133 eth access up none 10G(D) -- Eth1/15 10 eth access down SFP not inserted 10G(D) -- Eth1/16 1 eth access down SFP not inserted 10G(D) -- Eth1/17 1 eth trunk down SFP not inserted 10G(D) 17 Eth1/18 1 eth trunk down SFP not inserted 10G(D) 18 Eth1/19 1 eth trunk up none 10G(D) 4096 Eth1/20 1 eth access up none 10G(D) -- -------------------------------------------------------------------------------- Interface Vsan Admin Status Oper Oper IP Trunk Mode Speed Address Mode (Gbps) -------------------------------------------------------------------------------- san-port-channel 10 4094 on noOperMembers -- -- -------------------------------------------------------------------------------- Port-channel VLAN Type Mode Status Reason Speed Protocol Interface -------------------------------------------------------------------------------- Po3 1 eth trunk up none a-1000(D) lacp Po17 1 eth trunk down No operational members auto(D) none Po18 1 eth trunk down No operational members 10G(D) lacp Po202 1 eth trunk down No operational members auto(D) lacp Po1234 1 eth trunk down No operational members 10G(I) none Po4096 1 eth trunk up none a-10G(D) lacp -------------------------------------------------------------------------------- Port VRF Status IP Address Speed MTU -------------------------------------------------------------------------------- mgmt0 -- up 10.160.45.14 1000 1500



------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- vfc1 1 E on trunking -- TE 10 -- SW2(config-if)#

You can see that the port-channels for both Ethernet and FC are separated from each other and can be configured separately. The FC interfaces, although they are connected to the same ASIC as the Ethernet interfaces are shown different from the normal Ethernet interfaces.

Then we verify our VSAN database membership.

SW2(config-vsan-db)# show vsan member vsan 1 interfaces: vsan 10 interfaces: fc1/31 fc1/32 san-port-channel 10 vsan 20 interfaces: vsan 4079(evfp_isolated_vsan) interfaces: vsan 4094(isolated_vsan) interfaces: SW2(config-vsan-db)#

Here we see that both the port-channel and the individual interfaces have members in the VSAN that we configured it in.

Now we have a successful native FC connection between the MDS and the Nexus switches!

The next question states that we should request the lowest possible Domain ID.

In the previous chapters we configured parameters for giving out FC Domain ID’s that prevent the switches to use very low numbers. Therefore we will automatically get higher numbers assigned by configuring the lower numbers.

SW2

SW2(config)# fcdomain domain ? <0-239> Configure the domain id in decimal, 0 not allowed for static dom <0x0-0xef> Configure the domain id in hexadecimal SW2(config)# fcdomain domain 0 ? preferred Configure the domain id as preferred static Configure the domain id as static SW2(config)# fcdomain domain 0 preferred ? vsan Specify the vsan range SW2(config)# fcdomain domain 0 preferred vsan 10 SW2(config)# fcdomain domain 0 preferred vsan 20



Do not forget to restart the FC Domain when you are changing Domain IDs. Without a restart the domain ID’s will not be effective!

SW2

SW2(config)# fcdomain restart disruptive vsan 10 SW2(config)# fcdomain restart disruptive vsan 20

SW3

SW3(config)# fcdomain domain 0 preferred vsan 10 SW3(config)# fcdomain domain 0 preferred vsan 20 SW3(config)# fcdomain restart disruptive vsan 10 SW3(config)# fcdomain restart disruptive vsan 20

Now the switches will have assigned a higher number FC Domain ID, but have configured the lowest possible value, which is 0.

Now we should be able to see all devices that are located in VSAN 10 and VSAN 20. Meaning we are able to see all the JBOD disks in both VSANs and we should have zoning applied as this is also taken care of when fabrics ‘merge’.

The final task is that we should keep in mind that there are already security mechanisms in place in the VSANs. Both VSAN 10 and VSAN 20 are using port-security. So we should add the Nexus switches SWWNs to those databases before they are fully operational!

We don’t have any hosts connected to the Nexus switches so the changes to the port-security database is not that impacting as we only need to add the switch WWNs to the port-security database to add the switches to the fabric.

Let’s check the port-security database of VSAN 10 on both MDS1 and MDS2 that we configured in previous chapters.

MDS1

MDS1(config)# do sh port-sec database -------------------------------------------------------------------------------- VSAN Logging-in Entity Logging-in Point (Interface) -------------------------------------------------------------------------------- 10 c6:81:00:11:0d:17:fa:ce(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)* 10 c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)* 10 c6:80:00:11:0d:07:fa:ce(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)* 10 c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)* 10 20:01:54:7f:ee:05:08:79(swwn) 20:05:00:0d:ec:6f:4f:40(fc1/5)* 10 20:01:54:7f:ee:05:08:79(swwn) 20:06:00:0d:ec:6f:4f:40(fc1/6)* 10 20:00:00:a3:bf:33:11:33(pwwn) 20:0b:00:0d:ec:6f:4f:40(fc1/11)* 10 20:00:00:a3:fe:00:98:32(pwwn) Any [Total 8 entries]

MDS2

MDS2(config-port-security)# do sh port-sec database --------------------------------------------------------------------------------



VSAN Logging-in Entity Logging-in Point (Interface) -------------------------------------------------------------------------------- 10 21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78(fc1/1)* 10 20:00:00:0d:ec:6f:4f:40(swwn) 20:05:54:7f:ee:05:08:78(fc1/5)* 10 20:00:00:0d:ec:6f:4f:40(swwn) 20:06:54:7f:ee:05:08:78(fc1/6)* 10 20:00:00:a3:de:11:66:2b(pwwn) 20:00:54:7f:ee:05:08:78(swwn) 10 20:00:00:a3:fe:00:98:32(pwwn) Any [Total 5 entries] MDS2(config-port-security)#

Here we see that VSAN 10 is not using a distributed port-security database, so we need to add the Nexus switches to both of the databases on the switches.

We will do this from the MDS point of view, again because it’s not distributed.

Check the SWWN of the 2 switches:

SW2

SW2# show wwn sw Switch WWN is 20:00:00:05:9b:73:1b:c0

SW3

SW3# show wwn sw Switch WWN is 20:00:00:05:9b:a4:1f:aa

Now we add those entries to the connecting MDS switches. In our topology SW2 is connected to MDS1 and SW3 is connected to MDS2.

MDS1

MDS1(config)# port-security data vsan 10 MDS1(config-port-security)# swwn 20:00:00:05:9b:73:1b:c0

MDS2

MDS2(config)# port-security data vsan 10 MDS2(config-port-security)# swwn 20:00:00:05:9b:a4:1f:aa

Now the SWWN of the switches has been added to the port-security database and you are good to go and have a working native Fibre Channel fabric for VSAN 10.

Now we shouldn’t forget about VSAN 20 as well, as this VSAN is also allowed on the trunk to the MDS switches.

The port-security of VSAN 20 is distributed, so the change only needs to be done from one MDS switch. In the previous chapter we were only allowed to do configuration changes on MDS1 so we will honor this again in this question.

MDS1

MDS1(config)# do sh port-security data v 20 -------------------------------------------------------------------------------- VSAN Logging-in Entity Logging-in Point (Interface) --------------------------------------------------------------------------------



20 c6:80:00:11:0d:07:fa:ce(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)* 20 c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)* 20 c6:81:00:11:0d:17:fa:ce(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)* 20 c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)* 20 21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78* [Total 5 entries] MDS1(config)#

Now on to the configuration of our database.

MDS1

MDS1(config-port-security)# port-security database vsan 20 MDS1(config-port-security)# swwn 20:00:00:05:9b:73:1b:c0 MDS1(config-port-security)# swwn 20:00:00:05:9b:a4:1f:aa MDS1(config-port-security)# port-security activate vsan 20 MDS1(config)# port-security commit v 20

After activating the database on the MDS1, we can see an activated database on both MDS switches with all necessary information for activating the security in VSAN 20.

Now we have finished our first task in this chapter by configuring native FC interfaces on the Nexus series switches.



Task 2: Fibre Channel over Ethernet (FCoE)

In this second task we will be configuring a unique feature of the Cisco Nexus platform. The switches have the possibility to transport Fibre Channel frames across Ethernet links.

This is done so you only require one cable instead of two to your server that needs to connect to devices on the Fibre Channel Fabric. This technology is also called ‘Unified Fabric’.

In your server you require a Converged Network Adapter (CNA). This CNA has a chip to process both Fibre Channel and Ethernet frames. The processing occurs simultaneously.

The initial version of FCoE required a fixed configuration of a port to do FCoE. This has changed in a newer version of the protocol, where a helper protocol was introduced called Datacenter Bridging Exchange Protocol (DCBX). This protocol negotiates the features supported by the switch and the connecting server with a CNA.

The switch is always leading in its communication. When the switch communicates a certain configuration value, like the VLAN number. This prevents miscommunications on the server side. So the networking team can direct the server what values to use.

The protocol is an extension to the widely used LLDP protocol and is also called the FCoE Initialization Protocol (FIP).

There are 2 very different types of FCoE. The first is the standard ‘F-port’ virtualization. Where you connect your Fibre Channel environment to the Converged Network Adapter (CNA) of a server. The Nexus then acts as a distribution switch for the first hop which is converged and the next hop, the traffic is split in both an Ethernet uplink and a Fibre Channel uplink.

The second type is the E-port virtualization. This is also called ‘multi-hop’ FCoE as you are no longer converging only the first hop, but also further hops in the network. You can connect Nexus switches using virtual E-ports to extend the Fibre Channel fabric through your Ethernet network. In fact, you do not need a MDS switch in some topologies as they can perform all features necessary, of course depending on the port density of your native Fibre Channel environment.

In this first task about FCoE we will be using the converged F-Ports.

The initial question is related to the fact that we need to configure a server which is connected to interface Ethernet1/24. We want to use a port-channel from the server to the network and we are using the vPC technology for this use.

First we need to ensure that the vPC technology works again across the Nexus switches. We will use the easiest solution, but configuring peer-keepalives across the mgmt0 interface and one of the interlinks of the Nexus switches as the peer-link.

We did not get any requirements in this question so we can use any values we want!



You might see more of these types of questions in the lab, where you are free to use values you want for the required configuration for this question. It lets you think about all the changes that are necessary to complete the question.

We will configure the switches for the vPC feature first.

SW2

SW2(config)# feature vpc SW2(config)# vpc domain 100 SW2(config-vpc-domain)# peer-keepalive destination 172.16.100.3

As we are using the mgmt0 interfaces as a source for the keepalives we do not need to configure the source of the keepalives as it’s using the mgmt0 and the management VRF by default as a source for the keepalives.

SW3

SW3(config)# feature vpc SW3(config)# vpc domain 100 SW3(config-vpc-domain)# peer-keepalive destination 172.16.100.2

Now we need to finish the vPC peer configuration by configuring the peer-link. Keep in mind that the peer-link needs to be a port-channel. In this case we only add one interface to it, as we only require some communication.

SW2

SW2(config)# interface ethernet1/7 SW2(config-if)# channel-group 1001 SW2(config-if)# interface port-channel1001 SW2(config-if)# vpc peer-link

SW3

SW3(config)# interface ethernet1/7 SW3(config-if)# channel-group 1001 SW3(config-if)# interface port-channel1001 SW3(config-if)# vpc peer-link

Let’s verify if the switches see each other and are able to form vPCs.

SW3(config-if)# sh vpc Legend: (*) - local vPC is down, forwarding via vPC peer-link vPC domain id : 101 Peer status : peer adjacency formed ok vPC keep-alive status : peer is alive Configuration consistency status: success Per-vlan consistency status : success Type-2 consistency status : success vPC role : secondary Number of vPCs configured : 0 Peer Gateway : Disabled



Dual-active excluded VLANs : - Graceful Consistency Check : Enabled vPC Peer-link status --------------------------------------------------------------------- id Port Status Active vlans -- ---- ------ -------------------------------------------------- 1 Po1001 up 1,100

Now we finish the first questions by configuring the vPC connecting the host.

SW2

SW2(config-if)# int e1/24 SW2(config-if)# sw mode trunk SW2(config-if)# no shut SW2(config-if)# channel-group 1002 SW2(config-if)# int po1002 SW2(config-if)# sw mode trunk SW2(config-if)# no shut SW2(config-if)# vpc 1002 SW2(config-if)#

SW3

SW3(config-if)# int e1/24 SW3(config-if)# sw mode trunk SW3(config-if)# no shut SW3(config-if)# channel-group 1002 SW3(config-if)# int po1002 SW3(config-if)# sw mode trunk SW3(config-if)# no shut SW3(config-if)# vpc 1002 SW3(config-if)#

Our vPC is online and is able to process traffic just like any other vPC!

Now we are going to configure the FCoE specifics. One very important topic to always keep in mind is that you are required to have a strict distinction between Fibre Channel fabrics!

The Fibre Channel world is arranged 2 fabrics that are in no place connected to each other. This has to do with the auto-learning nature of the Fibre Channel Network that devices can interfere with each other when something goes wrong.

The hosts in the Fibre Channel network are smart enough to detect failures and failover to the other fabric when this is necessary. This means that even we do have a virtual Port-Channel to the connecting server, we still are not going to share Fibre Channel configuration across this link.

The FC traffic is always carried in a dedicated VLAN. You cannot carry multiple VSANs over a single VLAN. There is always a distinct VSAN to VLAN mapping. This mapping of course needs to be equal on all switches in the topology. In this case we will not be using the 2 VSANs across the switches, but we will keep the configuration of the VSAN to the local switch.



SW2

SW2(config)# feature fcoe SW2(config-vlan)# vlan 10 SW2(config-vlan)# fcoe vsan 10 SW2(config-vlan)# vlan 20 SW2(config-vlan)# fcoe vsan 20

SW3

SW3(config)# feature fcoe SW3(config-vlan)# vlan 10 SW3(config-vlan)# fcoe vsan 10 SW3(config-vlan)# vlan 20 SW3(config-vlan)# fcoe vsan 20

Next we configure the vPC interface for the FCoE traffic. We do this by first creating a virtual interface. This virtual Fibre Channel interface (VFC) will be handling all Fibre Channel communication.

To ensure you are only using one leg of the vPC you need to bind the individual interface instead of the port-channel interface to this virtual FC interface.

SW2

SW2(config-vlan)# interface vfc1001 SW2(config-if)# bind interface Ethernet1/24 SW2(config-if)# no shutdown SW2(config-if)# vsan database SW2(config-vsan-db)# vsan 10 interface vfc1001

SW3

SW3(config-vlan)# interface vfc1001 SW3(config-if)# bind interface Ethernet1/24 SW3(config-if)# no shutdown SW3(config-if)# vsan database SW3(config-vsan-db)# vsan 20 interface vfc1001

This configuration enables the connection between the physical Ethernet interfaces. The Virtual FC interface and the VSAN in which it belongs. Now the switch knows that it needs to use which VLAN to tag the traffic when sending it across the physical interface.

We have a clear distinction across the Fibre Channel fabric, but still the Ethernet traffic is using a virtual Port-Channel and can still load balance across the 2 interfaces.

Next we need to ensure that the switches discard frames that are not from their own fabric as we are creating different FC fabrics.

This is done using the FC-MAP feature of the Nexus. By default the Nexus switches use a value of 0E.FC.00 as the Fabric Identifier. When a switch does not belong to a fabric (so the FC-MAP value is different) the switch discards frames.



You are able to configure FC-MAP values in the range of 0E.FC.00 up to 0E.FC.FF.

We will configure a different FC-MAP value on SW2 to ensure that it will drop traffic coming from SW3.

SW2

SW2(config)# fcoe fcmap ? <0xefc00-0xefcff> Enter FCMAP SW2(config)# fcoe fcmap 0x0efc02 Warning: Changing the FC-MAP will flap all VFC interfaces. Continue [y/n]? [no] y SW2(config)#

The CLI help can be misleading as the 0 in front of the FC-MAP value needs to be typed in, but the context help might indicate that it’s not necessary.

Next we are told to configure SW2 to be the primary switch. While the Fibre Channel fabrics can be used independently. It’s still possible to configure the CNA of the Server to connect to one particular fabric over the other switches.

This is controlled using the FCF priority. FCF stands for Fibre Channel Forwarder, meaning a Fibre Channel Switch. This FCF priority is by default set to 128 and the lower the value the more preferred the switch becomes.

Therefore we will lower this priority on SW2 to a very low value to ensure that it’s chosen to become the preferred FCF switch in the fabric.

We can verify the settings by using a single show command.

SW2(config)# show fcoe Global FCF details FCF-MAC is 00:05:9b:73:1b:c0 FC-MAP is 0e:fc:02 FCF Priority is 1 FKA Advertisement period for FCF is 8 seconds SW2(config)#

Here we see that we configured the FC-MAP and the FCF Priority like it it supposed to be.

The next task would be easily solvable by only allowing the FCoE VLANs on the trunk, but we are not allowed to do this. The final question in this task is related to a feature specific to the DCBX protocol.

The DCBX protocol can send messages regarding the VLANs that do not have a FCoE VSAN enabled on them.

You do this by basically shutting down the LAN part of the interface.

SW2

SW2(config)# interface port-channel1001 SW2(config-if)# shutdown lan



SW3

SW3(config)# interface port-channel1001 SW3(config-if)# shutdown lan

By configuring this single command you signal to the CNA that only a few VLANs are allowed on this trunk, which are enabled for FCoE. The other VLANs are not allowed to transfer any packets across the link and the link basically became a Fibre Channel interface, with Ethernet encapsulation!

Next we are going to use the FCoE feature for the multi-hop purpose.

Task 3: Multi hop FCoE

The third task will be regarding the configuration of the FCoE feature to continue the FCoE path after the initial hop which is connected to the CNA of the server.

We will be disabling all ISL links on the MDS switches to ensure only the Nexus switches will be used for ISL traffic. We want to use the Nexus switches with multiple hops in between to form one large fabric where all Fibre Channel traffic is sent across.

This is the initial question and relatively easy to do.

We configured multiple port-channels, but we can shutdown the FC links that connect the 2 MDS switches.

MDS1

MDS1(config-if)# interface fc1/1 MDS1(config-if)# shutdown MDS1(config-if)# interface fc1/2 MDS1(config-if)# shutdown MDS1(config-if)# interface fc1/3 MDS1(config-if)# shutdown MDS1(config-if)# interface fc1/4 MDS1(config-if)# shutdown

MDS2

MDS2(config-if)# interface fc1/1 MDS2(config-if)# shutdown MDS2(config-if)# interface fc1/2 MDS2(config-if)# shutdown MDS2(config-if)# interface fc1/3 MDS2(config-if)# shutdown MDS2(config-if)# interface fc1/4 MDS2(config-if)# shutdown



Now we broke our Fibre Channel fabric. This task is all about fixing this topology. In this task we will be extending VSAN 20 across the Nexus switches.

In this case we will be using the same FCoE technology as we used in the previous task except that in this case we will not be configuring F-ports, but E-ports to extend the fabric across the Nexus switches.

There is no real other stuff happening other than the previous task where we configured a vPC as an F-Port.

We need to ensure that VSAN 20 keeps functioning in the way that is shown in Drawing 2. This means that we cannot use the interlink between SW2 and SW3 for traffic and we need to use SW1 as well.

The native FC connections connecting SW2, SW3 and the MDS switches already works and VSAN 20 is working across this connection.

This means that we already have a good connection with our current Fibre Channel Fabric.

MDS1(config-port-security)# do sh flogi database vsan 20 --------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME --------------------------------------------------------------------------- fc1/3 20 0x0a00e8 c6:80:00:11:0d:07:fa:ce c5:9a:00:06:2b:01:00:07 fc1/3 20 0x0a00ef c6:80:00:11:0d:01:01:07 c5:9a:00:06:2b:01:00:07 fc1/4 20 0x0a01e8 c6:81:00:11:0d:17:fa:ce c5:9a:00:06:2b:02:00:07



fc1/4 20 0x0a01ef c6:81:00:11:0d:02:01:07 c5:9a:00:06:2b:02:00:07 Total number of flogi = 4.

Next we are configuring the FCoE virtual E-ports that will connect the Nexus switches to each other.



On the Nexus 7000 the architecture of FCoE is a little different than on the Nexus 5000 series. First of all the Nexus 7000 does not support any native FC interfaces, but only supports to be a FCoE forwarder. The Nexus 7000 also doesn’t support to be a Fibre Channel Bridge. Which means that it’s not capable of encapsulating FC frames into Ethernet packets to become FCoE frames. This of course has to do that there are no native FC packets coming into the Nexus 7000 switch.

One other important change on the Nexus 7000 platform is that FCoE features are not configured in the normal configuration. The control-plane of the Fibre Channel traffic is separated from the Ethernet traffic. This is done because of the possibility to create Virtual Device Contexts (VDCs) on the Nexus 7000 switches.

Normally you require a license to use VDC’s, but when you have the FCoE license you can enable the Storage VDC as well.

The Storage VDC or FCoE VDC works different from other VDCs. Normal VDCs require you to allocate interfaces to it. The Storage VDC works based on the EtherType of packets. FCoE packets have a different EtherType on which the interface knows where to direct traffic to.

The drawing below illustrates how the Storage VDC works and separates traffic coming in over a single link.



You can only create a single Storage VDC where all FCoE traffic is terminated. The use of normal Ethernet VDC’s is still supported. In this lab we will be using the global configuration and one Storage VDC as this is the minimal configuration that is required to support FCoE on the Nexus 7000.

To start our FCoE E-port configuration we start on the Nexus 5000s switches as they are pretty much equal to the previous task where we configured F-port FCoE.

Drawing 2 illustrates that there are 2 connections between the Nexus 7000 and the Nexus 5000s. As the task did not explicitly state this, we shouldn’t have to do this, as we could also create 2 E-ports between the Nexus switches, but this configuration is more commonly seen.

SW2

SW2(config)# interface ethernet1/11-12 SW2(config-if)# channel-group 1002 mode active SW2(config-if)# no shutdown SW2(config-if)# interface port-channel1002 SW2(config-if)# switchport mode trunk SW2(config-if)# switchport trunk allowed vlan 20 SW2(config-if)# no shutdown SW2(config-if)# exit SW2(config)# interface vfc1 SW2(config-if)# bind interface port-channel1002 SW2(config-if)# switchport mode e SW2(config-if)# switchport trunk allowed vsan 20 SW2(config-if)# no shutdown SW2(config-if)# vsan database SW2(config-vsan-db)# vsan 20 interface vfc1

SW3

SW3(config)# interface ethernet1/11-12 SW3(config-if)# channel-group 1002 mode active SW3(config-if)# no shutdown SW3(config-if)# interface port-channel1002 SW3(config-if)# switchport mode trunk SW3(config-if)# switchport trunk allowed vlan 20 SW3(config-if)# no shutdown SW3(config-if)# exit SW3(config)# interface vfc1 SW3(config-if)# bind interface port-channel1002 SW3(config-if)# switchport mode e SW3(config-if)# switchport trunk allowed vsan 20 SW3(config-if)# no shutdown SW3(config-if)# vsan database SW3(config-vsan-db)# vsan 20 interface vfc1

After this configuration the port-channel will come only, but the VFC interfaces will remain non-operational as the other side has not been configured yet.

Now we move on by configuring the Nexus 7000 switch. The configuration is a little different than on the Nexus 5000.



SW1-1

SW1-1(config)# install feature-set fcoe

We first need to enable a complete ‘feature-set’. This set enables a number of features, one of which is VDCs and all Fibre Channel related processes.

SW1-1

SW1-1(config)# vdc StorageVDC type storage SW1-1(config-vdc)# system default switchport SW1-1(config-vdc)# feature lldp

Next we configure the VDC that we want to use for Storage traffic. We need to enable LLDP in this VDC, because this protocol is used in the FIP (DCBX) protocol to identify which features are possible with the end host.

SW1-1

SW1-1(config-vdc)# allocate fcoe-vlan-range 20

Then we enable which VLANs that are enabled for FCoE that need to be carried through the VDC. So instead of issuing the allocate interface command in this place, you are using the allocate VLAN command to ensure these VLANs are connected to the VDC and FCoE traffic is sent to this special VDC.

SW1-1

SW1-1(config-vdc)# allocate shared interface ethernet2/1-4 SW1-1(config-vdc)# allocate shared interface port-channel1-2 SW1-1(config-vdc)# exit

The final step of the VDC configuration is to add the interfaces that are used for sharing the Ethernet and FC traffic. Additionally with the VLAN configuration you need to configure the shared interfaces.

Your configuration options are getting limited when you issue this command and you can only configure a plain trunk connection then on the Ethernet port and no other features.

SW1-1

SW1-1(config)# license fcoe module 2



The next step is to enable one of the available licenses in the chassis on the module. Without an FCoE license enabled on the module the feature will not work. Keep in mind that the FCoE feature only works on F-type linecards! In the CCIE Data Center lab we only use the first generation of linecards so we have a M1 and a F1 linecard available in our chassis. In this lab we are using the F-type line card. This is also the reason why we are using different interfaces than we are used to on the Nexus 5000s as uplinks to the F1 linecard.

Then we configure our Ethernet interfaces with a trunk connection and the port-channel configuration required to support the connectivity to the Nexus 5000s.

SW1-1

SW1-1(config)# vlan 20 SW1-1(config-vlan)# exit SW1-1(config)# interface ethernet2/1-2 SW1-1(config-if)# channel-group 1 mode active SW1-1(config-if)# no shutdown SW1-1(config-if)# interface port-channel1 SW1-1(config-if)# switchport mode trunk SW1-1(config-if)# switchport trunk allowed vlan 20 SW1-1(config-if)# no shutdown SW1-1(config-if)# exit SW1-1(config)# interface ethernet2/3-4 SW1-1(config-if)# channel-group 2 mode active SW1-1(config-if)# no shutdown SW1-1(config-if)# interface port-channel2 SW1-1(config-if)# switchport mode trunk SW1-1(config-if)# switchport trunk allowed vlan 20 SW1-1(config-if)# no shutdown SW1-1(config-if)# exit

Now we step into the VDC and create the configurations there related to the FCoE protocol.

SW1-1

SW1-1(config)# switchto vdc StorageVDC SW1-1-StorageVDC# conf t

Within the Storage VDC we must enter the configuration prompt again.

SW1-1

SW1-1-StorageVDC(config)# vsan database SW1-1-StorageVDC(config-vsan-db)# vsan 20 SW1-1-StorageVDC(config-vsan-db)# exit

We create the VSAN just like we would on a normal MDS switch.

SW1-1

SW1-1-StorageVDC(config)# interface ethernet2/1-4 SW1-1-StorageVDC(config-if)# no shutdown SW1-1-StorageVDC(config-if)# interface port-channel1-2 SW1-1-StorageVDC(config-if)# no shutdown SW1-1-StorageVDC(config-if)# vlan 20 SW1-1-StorageVDC(config-vlan)# fcoe vsan 20 SW1-1-StorageVDC(config-vlan)# exit



We open up all the interfaces that are shared with this VDC and then connect the VSAN 20 to the VLAN that is also shared with this VDC. Just like we would on the Nexus 5000s.

SW1-1

SW1-1-StorageVDC(config)# interface vfc1 SW1-1-StorageVDC(config-if)# bind interface port-channel1 SW1-1-StorageVDC(config-if)# switchport mode e SW1-1-StorageVDC(config-if)# switchport trunk allowed vsan 20 SW1-1-StorageVDC(config-if)# no shutdown SW1-1-StorageVDC(config-if)# vsan database SW1-1-StorageVDC(config-vsan-db)# vsan 20 interface vfc1 SW1-1-StorageVDC(config-vsan-db)# exit

We create the Virtual E-port on a VFC interface bound to the port-channel interface connecting to SW2.

After this configuration and we apply the VSAN database configuration we have enabled our fabric to being extended in to SW1-1 just like we want.

SW1-1

SW1-1-StorageVDC(config)# interface vfc2 SW1-1-StorageVDC(config-if)# bind interface port-channel2 SW1-1-StorageVDC(config-if)# switchport mode e SW1-1-StorageVDC(config-if)# switchport trunk allowed vsan 20 SW1-1-StorageVDC(config-if)# no shutdown SW1-1-StorageVDC(config-if)# vsan database SW1-1-StorageVDC(config-vsan-db)# vsan 20 interface vfc2

When finishing the second interface connecting to SW3 we finished the lengty configuration.

The FCoE configuration is successfully applied and we are able to see the fabric being extended across multiple Nexus switches.

On MDS1 we should see the JBOD connected to MDS2 and vice versa!

The next question states that we should protect against loopbacks. What is possible with VFC connections is that interfaces are created that are looped back to the same switch.

To prevent this, you can enable a mechanism that checks for the VF ID in the FCoE packets and when this matches to the switch its own VFID it will reject the traffic.

SW1-1

SW1-1(config)# switchto vdc StorageVDC SW1-1-StorageVDC# conf t SW1-1-StorageVDC(config)# fcoe veloopback



The next couple questions are all related to each other. We need to configure link authentication, just like we did in the Fibre Channel Networking chapter. This technology is called FC-SP, where a hashed authentication key is used to send a challenge to the other switch to authenticate the interface.

This technology is also available on the Nexus switches when you are using FCoE interfaces.

An important topic is the combination of interface modes. You can configure an interface with different modes as it might be required that you configure an interface with optional authentication. The table below demonstrates the different modes and what will happen when these 2 interfaces are connected to each other.

On Auto-Active Auto-Passive Off

On Yes Yes Yes Link down

Auto-Active Yes Yes Yes No authentication

Auto-Passive Yes Yes No authentication No authentication

Off Link down No authentication No authentication No authentication

The On and Off modes are simple modes that do what they tell. They enforce authentication or reject it. The auto modes are there for optional authentication. There should be at least 1 active link when both sides are on auto as the passive link will wait for an authentication request before starting any negotiation.

Our question states that SW1-1 should never start the negotiations for authentication. The other questions state various comments about which passwords are used.

Read these carefully as they all contain vital information to finish the task.

First look up the SWWNs of all the 3 switches so you have the information to create the authentication entries.

SW1-1

SW1-1(config)# switchto vdc StorageVDC SW1-1-StorageVDC# show wwn switch Switch WWN is 20:00:00:05:ab:a3:16:76

SW2

SW2# show wwn sw Switch WWN is 20:00:00:05:9b:73:1b:c0

SW3

SW3# show wwn sw



Switch WWN is 20:00:00:05:9b:a4:1f:aa

Now we start by configuring the passwords used to authenticate the switches towards each other.

SW1-1

SW1-1(config)# switchto vdc StorageVDC SW1-1-StorageVDC# conf t SW1-1-StorageVDC(config)# feature fcsp SW1-1-StorageVDC(config)# fcsp dhchap ? devicename Configure password of the remote devices dhgroup Configure DHCHAP Diffie-Hellman Group List (In preference) hash Configure DHCHAP Hash Algorithm List (In order of preference) password Configure local DHCHAP password SW1-1-StorageVDC(config)# fcsp dhchap password Nexus7000password ? <CR> <hh:hh:hh:hh:hh:hh:hh:hh> WWN of the device with which to use this local password SW1-1-StorageVDC(config)# fcsp dhchap password Nexus7000password

SW2

SW2(config)# feature fcsp SW2(config)# fcsp dhchap password SecureNexus5000-1

SW3

SW3(config)# feature fcsp SW3(config)# fcsp dhchap password IPexpertIsAwesome

Then we continue by configuring the switches to authenticate the devices using their specified passwords.

SW1-1

SW1-1(config)# switchto vdc StorageVDC SW1-1-StorageVDC# conf t SW1-1-StorageVDC(config)# fcsp dhchap device 20:00:00:05:9b:73:1b:c0 password SecureNexus5000-1

We authenticate SW2 with the password as specified in the question.

SW1-1

SW1-1-StorageVDC(config)# fcsp dhchap device 20:00:00:05:9b:a4:1f:aa password IPexpertIsAwesome



Same accounts for SW3 which is authenticated with the specified password.

To finish we configure the interfaces to support the FC-SP configuration without actively starting to negotiate it.

SW1-1

SW1-1-StorageVDC(config)# interface vfc1 SW1-1-StorageVDC(config-if)# fcsp auto-passive SW1-1-StorageVDC(config)# interface vfc2 SW1-1-StorageVDC(config-if)# fcsp auto-passive

Next we configure SW2 and SW3 for the equal configuration.

Configuration of both switches is even equal as it’s using the same password for both switches.

SW2

SW2(config)# fcsp dhchap device 20:00:00:05:ab:a3:16:76 password Nexus7000password SW2(config)# interface vfc1 SW2(config-if)# fcsp auto-active

SW3

SW3(config)# fcsp dhchap device 20:00:00:05:ab:a3:16:76 password Nexus7000password SW3(config)# interface vfc1 SW3(config-if)# fcsp auto-active

Now our interfaces are successfully authenticated using a secure hash, but wait we should enable a SHA-1 hash! By default the MD5 hash is enabled, so we don’t comply with the full questioning.

SW1-1

SW1-1(config)# switchto vdc StorageVDC SW1-1-StorageVDC# conf t SW1-1-StorageVDC(config)# fcsp dhchap hash sha1

SW2

SW2(config)# fcsp dhchap hash sha1

SW3

SW3(config)# fcsp dhchap hash sha1



Now when all links are flapped again, they will be authenticated using a SHA-1 hash of the correct key and we answered the questions correctly.

To finish this task we need to enable a mechanism so only the active switches are allowed in the Fibre Channel Fabric of VSAN 20.

This means that we are going to use Fabric Binding. This was introduced in one of the latest versions of NX-OS, but is now fully supported. Pay attention that Fabric Binding is only supported in the SAN Enterprise license on the Nexus series and the Mainframe license on the MDS switches!

Configuration is relatively easy as we configured it before in the Fibre Channel chapter.

SW1-1

SW1-1(config)# switchto vdc StorageVDC SW1-1-StorageVDC# show wwn switch Switch WWN is 20:00:00:05:ab:a3:16:76 SW1-1-StorageVDC# conf t SW1-1-StorageVDC(config)# feature fabric-binding SW1-1-StorageVDC(config)# fabric-binding data vsan 20 SW1-1-StorageVDC(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00 SW1-1-StorageVDC(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8

First add the SWWNs of the 2 MDS switches.

SW1-1-StorageVDC(config-fabric-binding)# swwn 20:00:00:05:ab:a3:16:76 SW1-1-StorageVDC(config-fabric-binding)# swwn 20:00:00:05:9b:73:1b:c0 SW1-1-StorageVDC(config-fabric-binding)# swwn 20:00:00:05:9b:a4:1f:aa

Then we add the SWWNs of the 3 Nexus switches to the database.

SW1-1-StorageVDC(config-fabric-binding)# exit SW1-1-StorageVDC(config)# fabric-binding activate v 20

Keep in mind that you need to configure this on all the switches as the fabric-binding feature can not be distributed using the Cisco Fabric Services (CFS), which are also available on the Nexus switches.

SW2

SW2(config)# feature fabric-binding SW2(config)# fabric-binding data vsan 20 SW2(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00 SW2(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8 SW2(config-fabric-binding)# swwn 20:00:00:05:ab:a3:16:76 SW2(config-fabric-binding)# swwn 20:00:00:05:9b:73:1b:c0 SW2(config-fabric-binding)# swwn 20:00:00:05:9b:a4:1f:aa SW2(config-fabric-binding)# exit SW2(config)# fabric-binding activate v 20

SW3

SW3(config)# feature fabric-binding SW3(config)# fabric-binding data vsan 20 SW3(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00 SW3(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8



SW3(config-fabric-binding)# swwn 20:00:00:05:ab:a3:16:76 SW3(config-fabric-binding)# swwn 20:00:00:05:9b:73:1b:c0 SW3(config-fabric-binding)# swwn 20:00:00:05:9b:a4:1f:aa SW3(config-fabric-binding)# exit SW3(config)# fabric-binding activate v 20

Don’t forget to include the MDS switches as well, as they are still part of the fabric too!

MDS1

MDS1(config)# feature fabric-binding MDS1(config)# fabric-binding data vsan 20 MDS1(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00 MDS1(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8 MDS1(config-fabric-binding)# swwn 20:00:00:05:ab:a3:16:76 MDS1(config-fabric-binding)# swwn 20:00:00:05:9b:73:1b:c0 MDS1(config-fabric-binding)# swwn 20:00:00:05:9b:a4:1f:aa MDS1(config-fabric-binding)# exit MDS1(config)# fabric-binding activate v 20

MDS2

MDS2(config)# feature fabric-binding MDS2(config)# fabric-binding data vsan 20 MDS2(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00 MDS2(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8 MDS2(config-fabric-binding)# swwn 20:00:00:05:ab:a3:16:76 MDS2(config-fabric-binding)# swwn 20:00:00:05:9b:73:1b:c0 MDS2(config-fabric-binding)# swwn 20:00:00:05:9b:a4:1f:aa MDS2(config-fabric-binding)# exit MDS2(config)# fabric-binding activate v 20



Now we completed our task and have a successful multi-hop FCoE network running for VSAN 20. The next task will be about optimizing this network for FCoE which has some strict QoS requirements for the network!

Task 4: FCoE Quality of Service (QoS)

This task is about optimizing the network to run FCoE services. The FCoE protocol uses a few very strict rules which is causing the Ethernet even being called Data Center Bridging (DCB) or Data Center Ethernet (DCE). Basically it comes down to a few changed topics and most important within the QoS configuration.

This is related to the nature of the Fibre Channel protocol. This protocol was developed with a lossless network in mind. The B2B credit structure caused switches to ensure that packets will arrive at their destination without being dropped.

Now Ethernet is a very dirty medium and there is not a single guarantee when you send a packet on it, that it will arrive on the other end. Therefore there was a solution necessary to support Fibre Channel lossless architectures on top of the Ethernet links.

The solution for this is called the ‘Pause’ frame and the No-Drop queue.

The protocol which is used in this case is the IEEE 802.3x Link Level Flow Control. This allows hosts or switches to signal each other to pause the traffic so the host or switch can empty it’s almost full buffer. This prevents tail drops (or WRED drops) as the buffer is never exceeded in this case. This is essential for the behavior of Fibre Channel traffic when traveling across a Ethernet link.

This technology is also called Priority Flow Control (PFC). It works together with 802.1p bits in a tagged (802.1q) frame. The protocol already communicated which CoS value is to be used for FCoE traffic, which is also the queue in which the pause frame is active and the no-drop behavior is honored.

By default this value is CoS 3. This can be changed, but this is not recommended.

Priority Flow Control can be enabled on a per link basis, but this is not necessary in most scenarios as the DCBX protocol already negotiates this behavior with the connected host or switch. This negotiation is accepted by the other end and the CoS value on which to apply PFC is configured.



The first question we see is that we should make our topology to support FCoE on all Nexus switches in which we should follow the best practices.

There are a number of options possible to configure on the Nexus 7000 switches. By default when you enable the FCoE feature, it will not change anything about the QoS behavior of the device. This needs to be manually changed. The Nexus 7000 has a couple of built-in QoS behaviors that can be altered.

By default a template is active which is not honoring the No Drop behavior of CoS 3. Then there are multiple templates available where you are able to configure an even deeper level of Priority No Drop QoS, which might be necessary to place certain critical FCoE systems in, so they are getting prioritized over other FCoE traffic. Or it can be used for Control traffic that may not be dropped.

The table below demonstrates the CoS to Queue mapping on the Nexus 700 with the available default templates.

Template Default QoS Priority QoS No Drop Qos Priority No Drop QoS

default-nq-8e-policy 0,1,2,3,4 5,6,7 - -

default-nq-7e-policy 0,1,2,4 5,6,7 3 -

default-nq-6e-policy 0,1,2 5,6,7 3 4

default-nq-4e-policy 0 5,6,7 1,2,3 4

The first change we need to do is to configure the Nexus 7000 switch to actually support FCoE queues in its configuration. By default the QoS policy on the Nexus 7000 does not incorporate a FCoE queue. So we should enable this.

We will pick the default-nq-7e-policy for this use, as this is the most standard No Drop configuration and we do not require any additional queues to be mapped as well.

SW1-1

SW1-1(config)# system qos SW1-1(config-sys-qos)# service-policy type network-qos default-nq-7e-policy SW1-1(config-sys-qos)# exit



The next step is to enable the No Drop queue in the QoS configuration. The policy that is used is one which is by default available on the Nexus 7000s. You need to enable a No Drop queue for CoS 3 on the entire switch when enabling FCoE. Without this configuration the FCoE feature will not work correctly and especially in terms of congestion, the FCoE traffic will not be lossless!

One of the nice things of the Nexus 5000 series is that it automatically is enabled to support FCoE. There are 2 QoS groups by default enabled for queueing and one is selected to have the no-drop queue enabled.

SW2(config-if)# show queuing interface Ethernet1/1 queuing information: TX Queuing qos-group sched-type oper-bandwidth 0 WRR 50 1 WRR 50 RX Queuing qos-group 0 q-size: 243200, HW MTU: 1600 (1500 configured) drop-type: drop, xon: 0, xoff: 1520 Statistics: Pkts received over the port : 882892828 Ucast pkts sent to the cross-bar : 859238738 Mcast pkts sent to the cross-bar : 23654090 Ucast pkts received from the cross-bar : 213450997 Pkts sent to the port : 229752814 Pkts discarded on ingress : 0 Per-priority-pause status : Rx (Inactive), Tx (Inactive) qos-group 1 q-size: 76800, HW MTU: 2240 (2158 configured) drop-type: no-drop, xon: 128, xoff: 240 Statistics: Pkts received over the port : 0 Ucast pkts sent to the cross-bar : 0 Mcast pkts sent to the cross-bar : 0 Ucast pkts received from the cross-bar : 0 Pkts sent to the port : 0 Pkts discarded on ingress : 0 Per-priority-pause status : Rx (Inactive), Tx (Inactive) Total Multicast crossbar statistics: Mcast pkts received from the cross-bar : 16301832

The queuing configuration is supported by the class-map and policy configuration.

The default class-maps that are available are as the following output shows.

There it’s shown that the FCoE traffic is by default matched on CoS 3, which can be changed.

SW2(config-if)# show class-map Type qos class-maps =================== class-map type qos match-any class-fcoe



match cos 3 class-map type qos match-any class-default match any class-map type qos match-any class-all-flood match all flood class-map type qos match-any class-ip-multicast match ip multicast Type queuing class-maps ======================= class-map type queuing class-fcoe match qos-group 1 class-map type queuing class-default match qos-group 0 class-map type queuing class-all-flood match qos-group 2 class-map type queuing class-ip-multicast match qos-group 2 Type network-qos class-maps ============================== class-map type network-qos class-fcoe match qos-group 1 class-map type network-qos class-default match qos-group 0 class-map type network-qos class-all-flood match qos-group 2 class-map type network-qos class-ip-multicast match qos-group 2

The default policy-maps are configured to match on the class-maps and to apply appropriate queueing and MTU sizes.

SW2(config-if)# show policy-map Type qos policy-maps ==================== policy-map type qos default-in-policy class type qos class-fcoe set qos-group 1 class type qos class-default set qos-group 0 Type queuing policy-maps ========================



policy-map type queuing default-in-policy class type queuing class-fcoe bandwidth percent 50 class type queuing class-default bandwidth percent 50 policy-map type queuing default-out-policy class type queuing class-fcoe bandwidth percent 50 class type queuing class-default bandwidth percent 50 Type network-qos policy-maps =============================== policy-map type network-qos default-nq-policy class type network-qos class-fcoe pause no-drop mtu 2158 class type network-qos class-default mtu 1500

Interfaces have these policies applied by default. Which makes it very easy for a QoS best practice configuration on the Nexus 5000.

SW2(config-if)# show policy-map interface Global statistics status : enabled port-channel3 Service-policy (qos) input: default-in-policy policy statistics status: disabled Class-map (qos): class-fcoe (match-any) Match: cos 3 set qos-group 1 Class-map (qos): class-default (match-any) Match: any set qos-group 0 Service-policy (queuing) input: default-in-policy policy statistics status: disabled Class-map (queuing): class-fcoe (match-any) Match: qos-group 1 bandwidth percent 50 Class-map (queuing): class-default (match-any) Match: qos-group 0 bandwidth percent 50 Service-policy (queuing) output: default-out-policy policy statistics status: disabled Class-map (queuing): class-fcoe (match-any) Match: qos-group 1 bandwidth percent 50



Class-map (queuing): class-default (match-any) Match: qos-group 0 bandwidth percent 50

Now we know that QoS is enabled on the switches and that they all take care of FCoE traffic with the No-drop queue we even included the last question of this task as well, because with the adaption of the default QoS policies we also included the MTU statements.

Within the Nexus switches it’s not necessary to change the MTU size on the interfaces directly. You can change MTU sizes per queue on the Nexus. This means that the FCoE queue can have a different MTU than all the others.

This is automatically arranged by the templates that we apply as that the FCoE class will get the MTU size to support maximum frame size of the Fibre Channel packet, which is 2112.

The next question about blocking receivers. The Nexus knows a special mechanism to use for QoS and queuing. It uses a technology called Virtual Output Queuing. This technology uses the ingress buffers of all interfaces to create a very large pool of Egress buffers.

It means that traffic is held at the ingress port, until the egress port has time to transmit the packet. This is arranged by the ‘token’ principle that the Nexus switches use. Before traffic is sent across the switch fabric between ASICs and/or linecards. The traffic needs to have a token before it’s allowed to go over the fabric. Now the packet is stalled at the ingress port, until it receives a token to be transmitted over to the egress interface.

This Virtual Output Queueing Unicast limit is required to enable the feature that we are looking for in this question. Which is enabled with a simple single command.

SW2

SW2(config)# hardware unicast voq-limit



The last question of this task introduces another important topic about FCoE traffic. The native Fibre Channel protocol knows the concept of B2B credits (Buffer to Buffer credits) to ensure a lossless behavior of the link. When you are using a long link on a Fibre Channel switch, it means you require more B2B credits than before, because there are more packets in transit on an interface at any given time, so it takes more time before acknowledgements are received for these packets and before you are able to send additional packets onto the link.

Within FCoE there is not a concept of B2B credits, but there is the concept of Pause frames to hold up the sending of packets until further notice.

Now this Pause frame is sent by the receiver towards the sender. When you have a long link, it means that at the time the receiver has sent this message, the sender already sent more packets onto the link, because there are more packets in transit. This might cause a buffer overflow and might cause the switch to have to drop packets.

To overcome this, we need to ensure the buffer size is set to the correct size according to the link size. The Nexus 5000 and 7000 have strict limitations on the length of an FCoE link as there is not much buffer space available on those platforms.

On the Nexus 5500 where we are configuring these values the maximum supported link-size is up to 3000 meters or 3 kilometers. We need to configure the buffer size, the buffer fill rate at which it should start sending Pause frames, because this needs to be done before the buffer is full and finally the buffer rate at which it can send a frame so traffic may be sent again.

We need to perform this configuration on both SW2 and SW3. Within the documentation there is an example configuration for 3000 meter links. When we divide this by 3 and multiply by 2, we have the buffer sizes for 2000 meter links. Which are the values we are going to use.



3000 meter values:

• Buffer size: 152000 bytes

• Pause Threshold: 103360 bytes

• Resume Threshold: 83520 bytes

When we divide these by 3 and multiply by 2 we get the values for 2000 meter links.

3000 meter values:

• Buffer size: 101333 bytes

• Pause Threshold: 68907 bytes

• Resume Threshold: 55680 bytes

By default the Nexus 5000 switches are enabled to support FCoE links of up to 300 meters, which should be enough within any datacenter.

Now on to the configuration of SW2 and SW3.

SW2

SW2# show policy-map type network-qos Type network-qos policy-maps =============================== policy-map type network-qos default-nq-policy class type network-qos class-fcoe pause no-drop mtu 2158 class type network-qos class-default mtu 1500 SW2# conf t Enter configuration commands, one per line. End with CNTL/Z.

We can not change the default policy-map, so we need to create a new policy-map of which we base our configuration from the current default.

Then we apply our special buffer sizes.

SW2

SW2(config)# policy-map type network-qos 2000m-nq-policy SW2(config-pmap-nq)# class type network-qos class-fcoe SW2(config-pmap-nq-c)# pause no-drop SW2(config-pmap-nq-c)# mtu 2158 SW2(config-pmap-nq-c)# pause no-drop buffer-size 101333 ? pause-threshold Buffer limit for pausing in bytes



SW2(config-pmap-nq-c)# pause no-drop buffer-size 101333 pause-threshold 68907 resume-threshold 83520 SW2(config-pmap-nq-c)# system qos SW2(config-sys-qos)# service-policy type network-qos 2000m-nq-policy SW2(config-sys-qos)# exit

Now we verify our changes in configuration!

SW2(config)# show policy-map type network-qos Type network-qos policy-maps =============================== policy-map type network-qos 2000m-nq-policy class type network-qos class-fcoe pause no-drop buffer-size 101333 pause-threshold 58880 resume-threshold 38400 mtu 2158 class type network-qos class-default mtu 1500 policy-map type network-qos default-nq-policy class type network-qos class-fcoe pause no-drop mtu 2158 class type network-qos class-default mtu 1500 SW2(config)# SW2(config)# show policy-map interface Global statistics status : enabled port-channel3 Service-policy (qos) input: default-in-policy policy statistics status: disabled Class-map (qos): class-fcoe (match-any) Match: cos 3 set qos-group 1 Class-map (qos): class-default (match-any) Match: any set qos-group 0 Service-policy (queuing) input: default-in-policy policy statistics status: disabled Class-map (queuing): class-fcoe (match-any) Match: qos-group 1 bandwidth percent 50 Class-map (queuing): class-default (match-any) Match: qos-group 0 bandwidth percent 50 Service-policy (queuing) output: default-out-policy policy statistics status: disabled Class-map (queuing): class-fcoe (match-any) Match: qos-group 1 bandwidth percent 50



Class-map (queuing): class-default (match-any) Match: qos-group 0 bandwidth percent 50 SW2(config)# show queuing int Ethernet1/1 queuing information: TX Queuing qos-group sched-type oper-bandwidth 0 WRR 50 1 WRR 50 RX Queuing qos-group 0 q-size: 218560, HW MTU: 1600 (1500 configured) drop-type: drop, xon: 0, xoff: 1366 Statistics: Pkts received over the port : 883236801 Ucast pkts sent to the cross-bar : 859564057 Mcast pkts sent to the cross-bar : 23672744 Ucast pkts received from the cross-bar : 213458097 Pkts sent to the port : 229759946 Pkts discarded on ingress : 0 Per-priority-pause status : Rx (Inactive), Tx (Inactive) qos-group 1 q-size: 101440, HW MTU: 2240 (2158 configured) drop-type: no-drop, xon: 690, xoff: 83 Statistics: Pkts received over the port : 0 Ucast pkts sent to the cross-bar : 0 Mcast pkts sent to the cross-bar : 0 Ucast pkts received from the cross-bar : 0 Pkts sent to the port : 0 Pkts discarded on ingress : 0 Per-priority-pause status : Rx (Inactive), Tx (Inactive) Total Multicast crossbar statistics: Mcast pkts received from the cross-bar : 16301864



We see that the policy-map is correctly generated and now applied to all interfaces, because we see different queue sizes and other xon/xoff values!

We successfully completed task 4!

Task 5: N-Port Virtualization (NPV) and N-Port ID Virtualization (NPIV)

This next task is about a technology introduced to simplify the Fibre Channel networks in the access. The switches all need to carry a lot of information and need to maintain a strict routing protocol and zoning database.

This introduces a lot of control-plane complexity in all the Fibre Channel switches. To ease this the idea was to create access switches which are simple and only allow devices to come online and send traffic out over 1 or more uplinks to the Fibre Channel core network.

These switches can also operate independently, but are usually not powerful enough to be part of a large Fibre Channel network.



By default when you configure an F-port where an N-port is connected to. The MDS switch only allows up to 1 device to log-in to it. Except for Fabric Loop (FL) ports, where multiple devices in the loop can log-in. Except that this last technology is very different.

We want normal F-ports to log-in multiple times to a single link on a smart MDS switch in the core network.

This is where we start using “N-Port Virtualization” or NPV. This technology allows devices to log-in to a fabric and then translates this request up to a core switch on a dedicated uplink connection.

The core switch also requires some special configuration. The connections down to the access switch (or edge switch) are just default F-ports, but a feature called N-port ID Virtualization (NPIV) needs to be enabled to accept the translated log-ins from the access switch in to the core switch.

The NPIV feature on the core switch enables the fact that it allows for handing out multiple FCIDs to a single F-port.

This task and the next one will be using a little different topology to support the NPV features. NPV requires a reboot of the device after which it can no longer be used as a normal MDS switch. The chances of seeing this technology used on the CCIE Data Center lab is small, because it would cost a whole MDS switch just for this single feature.

The drawing below is a subset of Drawing 3, which the topology we are using for this task.

The JBODs are the multiple hosts that want to connect to MDS1 where MDS2 should handle their logins.

The terminology in NPV environments is that the switch where devices are connected is called the “edge” switch and where the log-ins are handles is the “core switch”.



The argument in this task that the fabric should support more than 239 switches per VSAN comes from the fact that you can not have more than 239 Domain ID’s handed out in your Fibre Channel Fabric. This is due to the fact that this is a single byte number, with some reservations.

The NPV edge switch does not consume a FC Domain ID as it’s not really participating in the Fabric. Which introduces the possibility of having over 239 switches in the Fabric (VSAN).

We should first enable the ISL links again between the switches.

MDS1

MDS1(config-if)# interface fc1/1 MDS1(config-if)# no shutdown MDS1(config-if)# interface fc1/2 MDS1(config-if)# no shutdown MDS1(config-if)# interface fc1/3 MDS1(config-if)# no shutdown MDS1(config-if)# interface fc1/4 MDS1(config-if)# no shutdown

MDS2

MDS2(config-if)# interface fc1/1 MDS2(config-if)# no shutdown MDS2(config-if)# interface fc1/2 MDS2(config-if)# no shutdown MDS2(config-if)# interface fc1/3 MDS2(config-if)# no shutdown MDS2(config-if)# interface fc1/4 MDS2(config-if)# no shutdown

After that we enable the NPIV feature on the core switch, so it allows handing out multiple FCIDs on a single interface.

MDS2

MDS2(config-if)# feature npiv

Next we should make MDS1 the edge switch. When you enable NPV it automatically does a write erase of the whole switch and performs a reboot. This is a very impacting command!

MDS1

MDS1(config)# feature npv Verify that boot variables are set and the changes are saved. Changing to npv mode erases the current configuration and reboots the switch in npv mode. Do you want to continue? (y/n):y >> MDS-Bootloader-01.00.19 (Feb 1 2010 - 15:13:26), Build: 01.00.19 PowerPC CPU: 8548E, Version: 2.1, (SVR:0x80390021) Core: E500, Version: 2.2, (PVR:0x80210022) Clocks: CPU:1333 MHz, CCB: 533 MHz,



DDR: 266 MHz, LBC: 33 MHz L1: D-cache 32 kB enabled I-cache 32 kB enabled INFO: Booting off primary flash. I2C: ready DRAM: DDR: ddrioovcr 0xd0000000 Total SDRAM memory is 1024 MB 40000000 INFO: SDRAM tests PASSED. DRAM: ECC initialization in progress...Done. done. INFO: Board rev = 6 type = 4 index 9037 L2 cache 512KB: enabled IDE: Bus 0: OK Device 0: Model: SILICONSYSTEMS UDMA 1GB-4673 Firm: 3.38 Ser#: CC33521774500002550 Type: Hard Disk Capacity: 977.4 MB = 0.9 GB (2001888 x 512) Booting bootflash:/kickstart-latest ... ................... Automatic boot of image at addr 0x00000000 ... Starting kernel... Entered kgdb_console_init:1975 INIT: version 2.85 booting Checking all filesystems....r. done. Setting kernel variables done. Setting the System Clock using the Hardware Clock as reference...System Clock set. Local time: Thu Nov 8 14:07:31 UTC 2012 Loading system software Uncompressing system image: bootflash:/system-latest CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC n_port virtualizer mode. --------------------------------------------------------------- INIT: Entering runlevel: 3 Starting NFS servers: nfsd mountd. 2012 Nov 8 14:08:08 Nov 8 14:08:08 %KERN-2-SYSTEM_MSG: Starting kernel... - kernel 2012 Nov 8 14:08:09 Nov 8 14:08:08 %KERN-1-SYSTEM_MSG: Entered kgdb_console_init:1975 - kernel 2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-PS_FAIL: Power supply 1 failed or shut down (Serial number QCS1427K1CM) 2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number QCS1427K1NK) 2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok 2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-FAN_OK: Fan module ok 2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-FAN_OK: Fan module ok 2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-FAN_OK: Fan module ok 2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-FAN_OK: Fan module ok [########################################] 100% Copy complete, now saving to disk (please wait)... User Access Verification MDS1 login: admin Password: Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2012, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are




owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php MDS1#

Next we configure the interfaces to support the topology. We will be using all the interfaces that we have without bundling them. Traffic is able to load balance across the available uplinks towards the core switch.

MDS2

MDS2(config-if)# interface fc1/1 MDS2(config-if)# switchport mode f MDS2(config-if)# interface fc1/2 MDS2(config-if)# switchport mode f MDS2(config-if)# interface fc1/3 MDS2(config-if)# switchport mode f MDS2(config-if)# interface fc1/4 MDS2(config-if)# switchport mode f

Default F ports are used for connecting core to edge switches in the NPV topology.

We then ensure that the interfaces are placed into VSAN 10 as that is what the task states.

MDS2

MDS2(config-if)# vsan database MDS2(config-vsan-db)# vsan 10 interface fc1/1 MDS2(config-vsan-db)# vsan 10 interface fc1/2 MDS2(config-vsan-db)# vsan 10 interface fc1/3 MDS2(config-vsan-db)# vsan 10 interface fc1/4

Now that we configured our core switch to support our topology we focus on the NPV edge switch MDS1 again.

We configure the uplinks to the core switch to be NPV uplinks.

MDS1

MDS1(config)# int fc1/1-4 MDS1(config-if)# sw mode np MDS1(config-if)# no shut MDS1(config-if)#

Once this is enabled the interfaces come only on the core switch and the NPV switch performs an FLOGI into the core.

MDS2(config-vsan-db)# do sh int fc1/1 fc1/1 is up Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:05:00:05:9b:77:0d:c0 Admin port mode is F, trunk mode is on





snmp link state traps are enabled Port mode is F, FCID is 0x1e0020 Port vsan is 10 Speed is 4 Gbps Rate mode is shared Transmit B2B Credit is 32 Receive B2B Credit is 16 Receive data field Size is 2112 Beacon is turned off 5 minutes input rate 16 bits/sec, 2 bytes/sec, 0 frames/sec 5 minutes output rate 8 bits/sec, 1 bytes/sec, 0 frames/sec 7 frames input, 668 bytes 0 discards, 0 errors 0 CRC, 0 unknown class 0 too long, 0 too short 8 frames output, 720 bytes 0 discards, 0 errors 1 input OLS, 1 LRR, 2 NOS, 0 loop inits 1 output OLS, 0 LRR, 1 NOS, 0 loop inits 16 receive B2B credit remaining 32 transmit B2B credit remaining 32 low priority transmit B2B credit remaining Interface last changed at Fri Nov 9 00:03:29 2012 MDS1(config-vsan-db)# do sh flogi data -------------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME -------------------------------------------------------------------------------- fc1/1 10 0x1e0020 20:05:54:7f:ee:18:07:a0 20:01:54:7f:ee:18:07:a1 fc1/2 10 0x1e0000 20:06:54:7f:ee:18:07:a0 20:01:54:7f:ee:18:07:a1 fc1/3 10 0x1e0020 20:03:54:7f:ee:18:07:a0 20:01:54:7f:ee:18:07:a1 fc1/4 10 0x1e0000 20:04:54:7f:ee:18:07:a0 20:01:54:7f:ee:18:07:a1 Total number of flogi = 4. MDS1(config-vsan-db)# do sh fcns data VSAN 10: -------------------------------------------------------------------------- FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------- 0x1e0000 N 20:06:54:7f:ee:18:07:a0 (Cisco) npv 0x1e0020 N 20:05:54:7f:ee:18:07:a0 (Cisco) npv 0x1e0040 N 20:04:54:7f:ee:18:07:a0 (Cisco) npv 0x1e0060 N 20:03:54:7f:ee:18:07:a0 (Cisco) npv Total number of entries = 4 MDS1(config-vsan-db)#

Now the switch is ready to receive traffic and log-ins! Let’s verify on the NPV switch.

MDS1(config-if)# show npv status npiv is disabled disruptive load balancing is disabled External Interfaces: ==================== Interface: fc1/5, State: Failed(Mismatch in VSAN for this upstream port) Interface: fc1/6, State: Failed(Mismatch in VSAN for this upstream port)



Number of External Interfaces: 2 Server Interfaces: ================== Number of Server Interfaces: 0

We see a mismatch in VSAN configuration. The NPV switch is still VSAN aware and you are even able to configure multiple VSANs on the NPV switch when this would be necessary.

Therefore we also need to do our VSAN configuration on the NPV switch.

MDS1

MDS1(config-if)# vsan database MDS1(config-vsan-db)# vsan 10 MDS1(config-vsan-db)# vsan 10 interface fc1/1 MDS1(config-vsan-db)# vsan 10 interface fc1/2 MDS1(config-vsan-db)# vsan 10 interface fc1/3 MDS1(config-vsan-db)# vsan 10 interface fc1/4

After configuring the VSAN database the status looks fine and the NPV switch is now really ready to accept logins and traffic from end hosts.

MDS1(config-vsan-db)# show npv status npiv is disabled disruptive load balancing is disabled External Interfaces: ==================== Interface: fc1/1, VSAN: 10, FCID: 0x1e0020, State: Up Interface: fc1/2, VSAN: 10, FCID: 0x1e0000, State: Up Interface: fc1/3, VSAN: 10, FCID: 0x1e0040, State: Up Interface: fc1/4, VSAN: 10, FCID: 0x1e0060, State: Up Number of External Interfaces: 4 Server Interfaces: ================== Number of Server Interfaces: 0 MDS1(config-vsan-db)#

Now it’s time to configure our JBODs to be logging in to the NPV switch.

MDS1

MDS1(config-if)# interface fc1/5 MDS1(config-if)# switchport mode f MDS1(config-if)# no shutdown MDS1(config-if)# interface fc1/6 MDS1(config-if)# switchport mode f MDS1(config-if)# no shutdown MDS1(config-if)# vsan database MDS1(config-vsan-db)# vsan 10 interface fc1/5 MDS1(config-vsan-db)# vsan 10 interface fc1/6



When you would forget to add the interfaces to the VSAN database, the ports will not come online and will give you this message.

MDS1(config)# show int fc1/5 fc1/5 is down (NPV upstream port not available)

Now when we verify our Core switch we see FLOGI’s being performed against the switch, just like we would expect from any other end host system.

MDS1(config-vsan-db)# show flogi data -------------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME -------------------------------------------------------------------------------- fc1/1 10 0x1e0020 20:05:54:7f:ee:18:07:a0 20:0a:54:7f:ee:18:07:a1 fc1/1 10 0x1e0100 21:01:00:e0:8b:2a:f0:54 20:01:00:e0:8b:2a:f0:54 fc1/2 10 0x1e0000 20:06:54:7f:ee:18:07:a0 20:0a:54:7f:ee:18:07:a1 Total number of flogi = 3. MDS1(config-vsan-db)# sh fcns data VSAN 10: -------------------------------------------------------------------------- FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------- 0x1e0000 N 20:06:54:7f:ee:18:07:a0 (Cisco) npv 0x1e0020 N 20:05:54:7f:ee:18:07:a0 (Cisco) npv 0x1e0100 N 21:01:00:e0:8b:2a:f0:54 (Qlogic) scsi-fcp:init Total number of entries = 3 MDS1(config-vsan-db)#

Now the zoning and all other features related to this end host are applied to the core switch and not to the edge switch. The edge switch really is nothing different than some sort of port extender of the network.

On the NPV switch it’s also visible that the hosts are logging in to the switch.

MDS1(config-if)# sh npv status npiv is disabled disruptive load balancing is disabled External Interfaces: ==================== Interface: fc1/1, VSAN: 10, FCID: 0x1e0020, State: Up Interface: fc1/2, VSAN: 10, FCID: 0x1e0000, State: Up Interface: fc1/3, VSAN: 10, FCID: 0x1e0040, State: Up Interface: fc1/4, VSAN: 10, FCID: 0x1e0060, State: Up Number of External Interfaces: 4 Server Interfaces: ================== Interface: fc1/5, VSAN: 10, State: Up Interface: fc1/6, VSAN: 10, State: Up



Number of Server Interfaces: 2 MDS1(config-if)#

The next question is related to steering traffic of the NPV switch across different uplinks.

CCIE Data Center Lab Preparation DSG


By default traffic will be balanced across the uplinks, but only once when they come online. There will be no second time the traffic is load balanced.

It’s possible to steer traffic manually across different uplinks of the NPV switch. This is done using Traffic-Maps. There you are able to specify which server interface needs to talk with which uplink interface.

You can first check the current load balancing configuration with the following outputs.

MDS1(config-if)# show npv traffic-map no traffic-map configuration found MDS1(config-if)# show npv flogi-table -------------------------------------------------------------------------------- SERVER EXTERNAL INTERFACE VSAN FCID PORT NAME NODE NAME INTERFAC E -------------------------------------------------------------------------------- fc1/5 10 0x1e0100 21:01:00:e0:8b:2a:f0:54 20:01:00:e0:8b:2a:f0:54 fc1/1 fc1/6 10 0x1e0100 21:01:00:e0:8b:2a:f0:54 20:01:00:e0:8b:2a:f0:54 fc1/1 Total number of flogi = 2. MDS1(config-if)# show npv external-interface-usage NPV Traffic Usage Information: ---------------------------------------- Server-If External-If ---------------------------------------- fc1/5 fc1/1 fc1/6 fc1/1 ---------------------------------------- MDS1(config-if)#

Now we are going to alter the behavior with traffic-map configuration.

MDS1

MDS1(config)# npv traffic-map server-interface fc1/5 external-interface fc1/1 MDS1(config)# npv traffic-map server-interface fc1/6 external-interface fc1/3

Now the traffic is balanced according to the questions.

Finally we should ensure that traffic is balanced over all uplinks automatically.

This means we need to enable disruptive load balancing which might cause some traffic loss, but does ensure an optimal load balancing across all interfaces.

MDS1

MDS1(config)# npv ? auto-load-balance Configure auto load balancing among preferred external links traffic-map Configure NPV traffic engineering MDS1(config)# npv auto-load-balance ?



disruptive Enable disruptive auto load balancing among external links MDS1(config)# npv auto-load-balance disruptive Enabling this feature may flap the server interfaces whenever load is not in a balanced state. This process may result in traffic disruption. Do you want to proceed? (y/n): y MDS1(config)#

Now we finished task 5!

The final task is about the same feature, but then on the Nexus 5000 platform instead of the Cisco MDS platform.

Task 6: FCoE NPV

The final task of this chapter will be about configuring the N-Port Virtualization feature on the Nexus 5000 switch.

This mechanism works on the exact same way as the NPV feature works for Native FC ports. Of course the Nexus 5000 switch also supports the native NPV features as well as the FCoE features.

The technology works by configuring a VFC port just like on Multi-Hop FCoE, but instead of configuring it as an E-port. It’s configured as an NP-port.

Be careful when you enable this, because when you enable FCoE and NPV, you will perform a reboot of the switch and enable both of the features. Whilst in this task we only require to have FCoE-NPV enabled, which does not require a reboot of the switch!

SW2

SW2(config)# feature fcoe-npv

Do not forget to enable NPIV on SW3 otherwise it will not accept multiple logins through a single F-port.

SW3

SW3(config)# feature npiv

We will configure the fourth uplink to SW3 as the uplink and VSAN 20. In relation to the previous tasks we will be using VLAN 20 for the VSAN to VLAN mapping.

SW2

SW2(config)# interface e1/8 SW2(config-if)# switchport mode trunk SW2(config-if)# switchport trunk allowed vlan 20 SW2(config-if)# no shutdown SW2(config-if)# vlan 20 SW2(config-vlan)# fcoe vsan 20



SW2(config-vlan)# exit SW2(config)# interface vfc8 SW2(config-if)# bind interface e1/8 SW2(config-if)# switchport mode np SW2(config-if)# no shutdown

SW3

SW3(config)# interface e1/8 SW3(config-if)# switchport mode trunk SW3(config-if)# switchport trunk allowed vlan 20 SW3(config-if)# no shutdown SW3(config-if)# vlan 20 SW3(config-vlan)# fcoe vsan 20 SW3(config-vlan)# exit SW3(config)# interface vfc8 SW3(config-if)# bind interface e1/8 SW3(config-if)# switchport mode f SW3(config-if)# no shutdown

And with this final configuration we finished Chapter 7!

The next chapter will be about configuring security features on the Nexus switches!



Chapter 8: Security Features

Chapter 8: Security Features is intended to let you be familiar with the Security features which are available on the Nexus platform. You will be configuring both AAA services and other management security as well as LAN security features like DHCP snooping and other protective features.

We highly recommend creating your own diagram at the beginning of each lab so you are able to draw on your own diagram, making it much easier when you step into the real lab. Multiple topology drawings are available for this chapter.



Solutions

Task 1: Port Security

In this chapter we will be configuring several security features on the Cisco Nexus series platform. We only focus on the Nexus switches in this chapter, because we already discussed the security features on the MDS switches.

There is a slight overlap in features that are more related to management of the devices than to the security side of the device, but according to the blueprint they fall under security features, which is the reason why we are going to discuss them here.

The Nexus switches are really switches, so do not expect a lot of security features on these devices like there are on IOS routers. Things like Zone Based Firewalls or anything related to stateful security of data plane traffic is not available.

The security on the Nexus switches is purely related to the interfaces, control-plane and a little bit to LAN based security features like DHCP snooping.

The drawing below is the generic topology drawing of this chapter regarding the hardware that will be used in the questions of this chapter. Drawing 1 illustrates the physical topology.

One important topic is Cisco TrustSec. This technology has been introduced on the Nexus series switches to introduce security within the datacenter.

This technology is only available on a selected number of devices and ensures you can create a multi-tenant environment. Traffic is identified with tags. All devices that are using this tag are authenticated between each other and traffic that is sent between the devices is secured using encryption, anti-replay protection and message integrity checks.



The second major task of this chapter is the Access Control Lists (ACL’s) where you can protect VLANs, Interfaces and Layer 3 VLAN interfaces. With ACLs you are able to protect an entity according to the Source and/or Destination IP address, IP protocol and Source and/or Destination TCP/UDP port number. VLANs can be protected by filtering on MAC addresses (source and/or destination).

This initial task will be about configuring a feature which has been available on Catalyst switches for a long time and is now introduced on the Datacenter switches as well.

The port security feature allows you to configure a maximum number of hosts that are allowed to connect to the interface. Hosts mean in this case MAC addresses. These MAC addresses can be auto learnt or statically configured. In a data center environment you can easily have thousands of MAC addresses so this is not often seen in these environments.

One flexibility in the feature that has been introduced in the Nexus switches is that it’s possible to configure the port-security maximum amounts per VLAN. So you are able to configure port-security on trunk links as well and specify maximums per VLAN.

There are a lot of options with this feature so therefore there are a high number of questions in this task.

There are limitations to this feature. Some are system limits and others are configurable.

8192 is the maximum amount of MAC addresses that can be secured by the port security feature on an entire system

1024 is the maximum amount of MAC addresses that can be learnt on any interface

You can configure a maximum per VLAN, but this is not configured by default

The feature not only protects against too much hosts logging in to the same interface, it also protects against the moving of MAC addresses.

This means that when a certain host is learnt or configured on an interface and it is connected to another interface. This is seen as a violation and the host is denied access or the interface is shutdown. This is called a ‘MAC Move Violation’.

The feature does not work for vPC interfaces as the database can not be synchronized between vPC peer switches. This introduces violations when hosts move between member interfaces of the vPC, therefore this command is not available on vPC interfaces. It is available on standard port-channels, where it doesn’t matter to which member interface the host is accessed.

The first questions in this task are related to setting up the Nexus switches for configuration. We need to configure the management network and some hostnames, because the switches boot up with a blank configuration. This is something we did a number of times, so it shouldn’t be a surprise.

SW1



Enter the password for "admin": IPexpert123 Confirm the password for "admin": IPexpert123 ---- Basic System Configuration Dialog ---- This setup utility will guide you through the basic configuration of the system. Setup configures only enough connectivity for management of the system. Please register Cisco Nexus Family devices promptly with your supplier. Failure to register may affect response times for initial service calls. Nexus devices must be registered to receive entitled support services. Press Enter at anytime to skip a dialog. Use ctrl-c at anytime to skip the remaining dialogs. Would you like to enter the basic configuration dialog (yes/no): no Nexus Switch 10.2.8.74 login: admin Password: IPexpert123 Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license. Some parts of this software may be covered under the GNU Public License or the GNU Lesser General Public License. A copy of each such license is available at http://www.gnu.org/licenses/gpl.html and http://www.gnu.org/licenses/lgpl.html switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# switchname SW1 SW1(config)# interface mgmt0 SW1(config-if)# ip address 172.16.100.1/24 SW1(config-if)# no shutdown SW1(config-if)# ip default-gateway 172.16.100.254 SW1(config)# ntp server 172.16.100.254

After configuring the management interface and the hostname, we are good to go. The other 2 switches are configured in a similar way.

SW2

Enter the password for "admin": IPexpert123 Confirm the password for "admin": IPexpert123 ---- Basic System Configuration Dialog ---- This setup utility will guide you through the basic configuration of the system. Setup configures only enough connectivity for management of the system. Please register Cisco Nexus Family devices promptly with your supplier. Failure to register may affect response times for initial service calls. Nexus devices must be registered to receive entitled support services.






Press Enter at anytime to skip a dialog. Use ctrl-c at anytime to skip the remaining dialogs. Would you like to enter the basic configuration dialog (yes/no): no Nexus Switch 10.2.8.75 login: admin Password: IPexpert123 Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license. Some parts of this software may be covered under the GNU Public License or the GNU Lesser General Public License. A copy of each such license is available at http://www.gnu.org/licenses/gpl.html and http://www.gnu.org/licenses/lgpl.html switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# switchname SW2 SW2(config)# interface mgmt0 SW2(config-if)# ip address 172.16.100.2/24 SW2(config-if)# no shutdown SW2(config-if)# ip default-gateway 172.16.100.254 SW2(config)# ntp server 172.16.100.254

SW3

Enter the password for "admin": IPexpert123 Confirm the password for "admin": IPexpert123 ---- Basic System Configuration Dialog ---- This setup utility will guide you through the basic configuration of the system. Setup configures only enough connectivity for management of the system. Please register Cisco Nexus Family devices promptly with your supplier. Failure to register may affect response times for initial service calls. Nexus devices must be registered to receive entitled support services. Press Enter at anytime to skip a dialog. Use ctrl-c at anytime to skip the remaining dialogs. Would you like to enter the basic configuration dialog (yes/no): no Nexus Switch 10.2.8.76 login: admin Password: IPexpert123 Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license. Some parts of this software may be covered under the GNU Public License or the GNU Lesser General Public License. A copy of







each such license is available at http://www.gnu.org/licenses/gpl.html and http://www.gnu.org/licenses/lgpl.html switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# switchname SW3 SW3(config)# interface mgmt0 SW3(config-if)# ip address 172.16.100.3/24 SW3(config-if)# no shutdown SW3(config-if)# ip default-gateway 172.16.100.254 SW3(config)# ntp server 172.16.100.254

Now we start with configuring the logical topology of this lab. This logical topology is fairly simple as we do not require much topology to test the security features.

In this lab we will simulate a lot of hosts behind interfaces, but they are not really connected. You will see similar questions in the CCIE Data Center lab exam.

The topology is created with standard LACP port-channels between all switches and a number of client interfaces where simulated hosts are then ‘connected’.

Below is Drawing 2 which is used as the logical topology in this chapter.

Within this chapter we will be configuring several VLANs. When VLAN numbers are mentioned and do not exist yet you should create them on the fly while working through the task.

We will start by configuring the port-channels between the switches. Ensure you are using LACP as this is a requirement in the task.

SW1

SW1(config-if)# int e3/1-2 SW1(config-if)# channel-group 101 mode active SW1(config-if)# no shut SW1(config-if)# SW1(config-if)# int port-chan101 SW1(config-if)# sw mode trunk





SW1(config-if)# no shut SW1(config-if)# SW1(config-if)# int e3/5-6 SW1(config-if)# channel-group 102 mode active SW1(config-if)# no shut SW1(config-if)# SW1(config-if)# int port-chan102 SW1(config-if)# sw mode trunk SW1(config-if)# no shut

After configuring the port-channel on the first switch. It will come only, but the ports will be in isolated state, because there is no LACP communication received back from the neighboring switch.

SW1(config-if)# sh port-chan summ Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 101 Po101(SD) Eth LACP Eth3/1(I) Eth3/2(I) 102 Po102(SD) Eth LACP Eth3/5(I) Eth3/6(I) SW1(config-if)#

This will change as soon as we start configuring the other 2 switches for port-channeling the interfaces together.

SW2

SW2(config-if)# int e1/1-2 SW2(config-if)# channel-group 101 mode active SW2(config-if)# no shut SW2(config-if)# SW2(config-if)# int port-chan101 SW2(config-if)# sw mode trunk SW2(config-if)# no shut SW2(config-if)# SW2(config-if)# int e1/5-6 SW2(config-if)# channel-group 103 mode active SW2(config-if)# no shut SW2(config-if)# SW2(config-if)# int port-chan103 SW2(config-if)# sw mode trunk SW2(config-if)# no shut

SW3

SW3(config-if)# int e1/1-2 SW3(config-if)# channel-group 102 mode active SW3(config-if)# no shut SW3(config-if)# SW3(config-if)# int port-chan102 SW3(config-if)# sw mode trunk SW3(config-if)# no shut SW3(config-if)# SW3(config-if)# int e1/5-6 SW3(config-if)# channel-group 103 mode active



SW3(config-if)# no shut SW3(config-if)# SW3(config-if)# int port-chan103 SW3(config-if)# sw mode trunk SW3(config-if)# no shut

Now when the port-channeling is checked we see that the interfaces are in participating state which means that the port-channels are up and running.

SW1

SW1(config-if)# sh port-chan summ Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 101 Po101(SU) Eth LACP Eth3/1(P) Eth3/2(P) 102 Po102(SU) Eth LACP Eth3/5(P) Eth3/6(P) SW1(config-if)#

SW2


SW3




Now we can move on to the first task of configuring the port-security feature.

This first question states that only 10 hosts may be connected to a certain ethernet interface on SW2.

The action that should be taken when the maximum has been reached is that the interface should be shutdown when another host is connected. Now this final point is an easy one, as by default the action that is taken in case of a violation of the port-security rules is to shut down the interface.

There are 3 actions that can be taken when a port-security rules are violated.

Shutdown

This action means that the interface will go in to errdisable state and the interface is completely shutdown at that point. After it is re-enabled it keeps its port-security configuration without changing anything. Including the previously learnt MAC addresses.

This is the default mode.

Restrict

Traffic from secure MAC addresses is allowed on the interface, but traffic from any unsecure MAC addresses is dropped and a count is kept for the dropped packets

Protect

Traffic from secure MAC addresses is still allowed, but the interface is protected as MAC address learning is disabled right after the first unsecure MAC address is seen. This means that new MAC addresses are no longer learnt. Traffic from previously learned safe MAC addresses can still pass through the interface

Let’s start with the configuration of the interface to protect it for 10 MAC addresses.

In this case we will statically configure the violation again, to ensure it’s done. Although it’s a default. You will never get points taken off your score for over configuration that doesn’t harm anything related to the tasks.

SW2

SW2(config)# feature port-security SW2(config)# int e1/11 SW2(config-if)# switchport SW2(config-if)# switchport port-security SW2(config-if)# switchpor port-security maximum ? <1-1025> Maximum addresses 1 to 1025 SW2(config-if)# switchpor port-security maximum 10 ? <CR> vlan Vlan on which the mac address should be secured SW2(config-if)# switchpor port-security maximum 10



SW2(config-if)# switc port-security ? <CR> aging Port-security aging commands mac-address MAC address maximum Max secure addresses violation Security violation mode SW2(config-if)# switc port-security violation ? protect Security violation protect mode restrict Security violation restrict mode shutdown Security violation shtudown mode SW2(config-if)# switc port-security violation shutdown

As is shown in the show run output. The violation is not configured in the configuration, because it’s a default.

Do not forget to enable the port-security feature by entering ‘switchport port-security’. Without this command, the other commands related to it, will show up in the configuration, but the feature is not enabled on this interface and will not work.

SW2(config-if)# do sh run int e1/11 !Command: show running-config interface Ethernet1/11 !Time: Sat Oct 27 05:08:14 2012 version 5.1(3)N1(1) interface Ethernet1/11 switchport port-security switchport port-security maximum 10

With the following output you are able to check if it’s correct what you configured and what the current status of the connection is.

SW2(config-if)# do sh port-security Total Secured Mac Addresses in System (excluding one mac per port) : 0 Max Addresses limit in System (excluding one mac per port) : 8192 ---------------------------------------------------------------------------- Secure Port MaxSecureAddr CurrentAddr SecurityViolation Security Action (Count) (Count) (Count) ---------------------------------------------------------------------------- Ethernet1/11 10 0 0 Shutdown ============================================================================ SW2(config-if)#



The next question states that the entries that are learnt through this interface should time-out after a certain time. By default when MAC addresses are learnt through the interface they are kept in the database forever, as aging is disabled.

You can change this behavior by configuring aging. This can be done in two ways.

Absolute

The absolute aging removes a MAC address after a certain amount of time. After the MAC address has been learnt, the switch starts the timer and after the configured amount of time, it deletes the entry from the database.

Inactivity

After the last packet received from the MAC address. The switch starts a timer, when after it expires the MAC address is deleted from the database. When a new packet is received from the host, the timer is reset again.

The question states that the MAC address entry should be removed after inactivity for 6 minutes. This means we are going to use this option for aging under the interface.

SW2

SW2(config)# int e1/11 SW2(config-if)# switchport port-sec aging ? time Port-security aging time type Type of timers SW2(config-if)# switchport port-sec aging type ? absolute Absolute Timer inactivity Inactivity Timer SW2(config-if)# switchport port-sec aging type inactivity SW2(config-if)# switchport port-sec aging time ? <1-1440> Aging time in minutes. Enter a value between 1 and 1440 SW2(config-if)# switchport port-sec aging time 6

After configuring the aging, you will see it is applied to the interface.

SW2(config-if)# sh port-sec int e1/11 Configured Port Security : Enabled Opertional Port Security : Enabled Port Status : Secure Down Violation Mode : Shutdown Aging Time : 6 mins Aging Type : Inactivity Maximum MAC Addresses : 10 Total MAC Addresses : 0 Configured MAC Addresses : 0 Sticky MAC Addresses : 0 Security violation count : 0 SW2(config-if)#



The next question asks to configure port-security on another interface with statically configured MAC addresses. As soon as you configure the static MAC addresses you should not forget to also put on the maximum number of MAC addresses. When you do not configure the maximum amount, the interface will keep learning MAC addresses indefinitely.

SW3

SW3(config)# feature port-security SW3(config)# int e1/11 SW3(config-if)# switchport SW3(config-if)# switchport port-security SW3(config-if)# switchport port-security maximum 5 SW3(config-if)# switchport port-security mac-address ? E.E.E 48 bit mac address format HHHH.HHHH.HHHH (Option 1) EE-EE-EE-EE-EE-EE 48 bit mac address format HHHH.HHHH.HHHH (Option 2) EE:EE:EE:EE:EE:EE 48 bit mac address format HHHH.HHHH.HHHH (Option 3) EEEE.EEEE.EEEE 48 bit mac address format HHHH.HHHH.HHHH (Option 4) sticky Sticky MAC address SW3(config-if)# switchport port-security mac-address 0010.4431.a1b3 SW3(config-if)# switchport port-security mac-address 1022.a0f5.b3de SW3(config-if)# switchport port-security mac-address 0011.99ff.22aa SW3(config-if)# switchport port-security mac-address 5581.a09a.b00c SW3(config-if)# switchport port-security mac-address ba01.dad3.c0ff SW3(config-if)#

Do not forget to enable the port-security feature on the interface itself, otherwise it will not work.

After configuring the 5 MAC addresses statically we verify that configuration is applied.

SW3(config-if)# show port-security Total Secured Mac Addresses in System (excluding one mac per port) : 4 Max Addresses limit in System (excluding one mac per port) : 8188 ---------------------------------------------------------------------------- Secure Port MaxSecureAddr CurrentAddr SecurityViolation Security Action (Count) (Count) (Count) ---------------------------------------------------------------------------- Ethernet1/11 5 5 0 Shutdown ============================================================================ SW3(config-if)# show port-security int e1/11 Configured Port Security : Enabled Opertional Port Security : Enabled Port Status : Secure Down Violation Mode : Shutdown Aging Time : 0 mins Aging Type : Absolute Maximum MAC Addresses : 5 Total MAC Addresses : 5 Configured MAC Addresses : 5 Sticky MAC Addresses : 0 Security violation count : 0



You can see that the configured MAC addresses are equal to the maximum number of allowed MAC addresses on this interface. Meaning that only these addresses are allowed on the interface and no other.

Then we are asked to configure an action after violating packets are received. As previously explained. This means that we will be using the ‘restrict’ action, as this is the only action that will keep the interface up, but keeps a packet count of all violating packets. Which is what the question asks for.

SW3

SW3(config)# int e1/11 SW3(config-if)# switchport port-security violation restrict SW3(config-if)#

Next up is the configuration of the port-channel between SW2 and SW3. We should only allow up to 100 MAC addresses on this interface before denying access to new ones and protecting the interface by stopping the MAC address learning feature.

This means that we will be using the protect action after a violation of the port-security rules.

When you read ahead on the questions, you see that the interface should automatically ‘remember’ all MAC addresses that are received. This feature basically means that we want all MAC addresses received on the port-channel to be statically configured, except we don’t want to configure them statically.

This feature is called ‘sticky’. When a MAC address is learnt and this feature is enabled, it is automatically saved into the port-security database and never aged out.

We start with the configuration on the port-channel.

SW2

SW2(config-if)# interface po102 SW2(config-if)# switchport port-security SW2(config-if)# switchport port-sec max 100 SW2(config-if)# switchport port-sec violation protect

Don’t forget to perform the configuration on both switches.

SW3

SW3(config-if)# interface po102 SW3(config-if)# switchport port-security SW3(config-if)# switchport port-sec max 100 SW3(config-if)# switchport port-sec violation protect



Now we configured the basic security, but we also want the feature enabled to save the MAC addresses automatically into the port-security database.

SW2

SW2(config-if)# switchport port-security mac-address sticky

SW3

SW3(config-if)# switchport port-security mac-address sticky

Now after the configuration we see that the maximum is correctly configured and that the interface is already learning port-security entries.

SW3(config-if)# do sh port-security Total Secured Mac Addresses in System (excluding one mac per port) : 4 Max Addresses limit in System (excluding one mac per port) : 8188 ---------------------------------------------------------------------------- Secure Port MaxSecureAddr CurrentAddr SecurityViolation Security Action (Count) (Count) (Count) ---------------------------------------------------------------------------- port-channel102 100 1 0 Protect Ethernet1/11 5 5 0 Restrict ============================================================================ SW3(config-if)# sh port-security int po102 Configured Port Security : Enabled Opertional Port Security : Enabled Port Status : Secure UP Violation Mode : Protect Aging Time : 0 mins Aging Type : Absolute Maximum MAC Addresses : 100 Total MAC Addresses : 2 Configured MAC Addresses : 0 Sticky MAC Addresses : 2 Security violation count : 0 SW3(config-if)#

It’s possible to check which MAC addresses are learnt in the database. The entries are placed into the running configuration. This means that they are not automatically saved after reboots. This only happens after you save the configuration manually on the box.

SW3(config-if)# show port-security address Total Secured Mac Addresses in System (excluding one mac per port) : 5 Max Addresses limit in System (excluding one mac per port) : 8187 ---------------------------------------------------------------------- Secure Mac Address Table ---------------------------------------------------------------------- Vlan Mac Address Type Remaining Remotely Remotely Ports age learnt aged (mins) out ---- ----------- ------ ------ ------- ----- ---- 1 BA01.DAD3.C0FF STATIC 0 No No Ethernet1/11



1 CCEF.48CE.EBEE STICKY 0 No No port-channel102 1 0010.4431.A1B3 STATIC 0 No No Ethernet1/11 1 0011.99FF.22AA STATIC 0 No No Ethernet1/11 1 00C0.DD14.8D3C STICKY 0 No No port-channel102 1 1022.A0F5.B3DE STATIC 0 No No Ethernet1/11 1 5581.A09A.B00C STATIC 0 No No Ethernet1/11 ====================================================================== SW3(config-if)#

Finally we should ensure that the maximum of 100 MAC addresses is divided among a few VLANs. The VLANS should get an even share, except for VLAN 10, which should get 66% of the maximum (or 2/3).

Let’s create the VLANs and configure this maximum on both switches.

VLAN 10 will get 66 of the 100 MAC addresses. Which leaves 33 for even distribution among the remaining 3 VLANS. This means a total of 11 MAC addresses per VLAN.

SW2

SW2(config-if)# vlan 10,11,12,13 SW2(config-vlan)# exit SW2(config)# int port-channel102 SW2(config-if)# switchport port-sec maximum 66 vlan 10 SW2(config-if)# switchport port-sec maximum 11 vlan 11 SW2(config-if)# switchport port-sec maximum 11 vlan 12 SW2(config-if)# switchport port-sec maximum 11 vlan 13 ERROR: Maximum value on the port is less than the vlan value SW2(config-if)# switchport port-sec maximum 10 vlan 13 SW2(config-if)#

Now because we can’t divide the number completely in whole numbers. We get an error message that we configure too much MAC addresses per VLAN.

We solve this by giving the last VLAN one address less, as we can’t up the total amount of MAC addresses, but we are still dividing the amount evenly as possible.

SW2(config-if)# show run port-security !Command: show running-config port-security !Time: Sat Oct 27 07:10:42 2012 version 5.1(3)N1(1) feature port-security interface port-channel102 switchport port-security switchport port-security maximum 100 switchport port-security violation protect switchport port-security mac-address sticky switchport port-security maximum 66 vlan 10 switchport port-security maximum 11 vlan 11 switchport port-security maximum 11 vlan 12 switchport port-security maximum 10 vlan 13 interface Ethernet1/11



switchport port-security switchport port-security aging type inactivity switchport port-security aging time 6 switchport port-security maximum 10 SW2(config-if)#

After the configuration we verify the configuration and check all our previous configuration.

The last 3 questions of this task are about the security of an interface on SW3. To simulate a router we configure an interface on SW2 as a routed interface.

When we configure the SVI interface on SW3 for VLAN 100 and configure Ethernet1/7 as a trunk interface. We should already be able to ping each other.

Do this before continuing with the port-security task so you are sure the Layer 3 path is working correctly.

SW2

SW2(config)# int e1/7 SW2(config-if)# no switchport SW2(config-if)# ip add 198.18.100.1/24 SW2(config-if)# no shut SW2(config-if)# do sh run int e1/7 !Command: show running-config interface Ethernet1/7 !Time: Sat Oct 27 07:16:29 2012 version 5.1(3)N1(1) interface Ethernet1/17 no switchport ip address 198.18.100.1/24 SW2(config-if)#

SW3

SW3(config-if)# vlan 100 SW3(config-vlan)# exit SW3(config)# feature interface-vlan SW3(config)# int vlan 100 SW3(config-if)# ip add 198.18.100.2/24 SW3(config-if)# no shut SW3(config-if)# int e1/7 SW3(config-if)# switchport mode access SW3(config-if)# switchport access vlan 100 SW3(config-if)# no shut SW3(config-if)#



After the configuration of the Layer 3 routed interface on SW2 and the access port on SW3 where we are able to configure the port-security at a later stage. We should now be able to ping the other end.

It might happen that this doesn’t work from the perspective of SW3.

SW3(config)# do ping 198.18.100.1 PING 198.18.100.1 (198.18.100.1): 56 data bytes Request 0 timed out Request 1 timed out Request 2 timed out Request 3 timed out Request 4 timed out --- 198.18.100.1 ping statistics --- 5 packets transmitted, 0 packets received, 100.00% packet loss

This might be related to the missing of a trunk allowed list on the port-channel between the switches. This can cause it that the ping does not work.

SW3

SW3(config)# int po102 SW3(config-if)# sw trunk allowed vlan 10-13

After placing a trunk allowed list on the port-channel you are sure that the ping will work in all cases.

SW3(config)# ping 198.18.100.1 PING 198.18.100.1 (198.18.100.1): 56 data bytes 64 bytes from 198.18.100.1: icmp_seq=0 ttl=254 time=3.167 ms 64 bytes from 198.18.100.1: icmp_seq=1 ttl=254 time=4.923 ms 64 bytes from 198.18.100.1: icmp_seq=2 ttl=254 time=4.962 ms 64 bytes from 198.18.100.1: icmp_seq=3 ttl=254 time=4.976 ms 64 bytes from 198.18.100.1: icmp_seq=4 ttl=254 time=6.528 ms --- 198.18.100.1 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 3.167/4.911/6.528 ms SW3(config)#

Now we can continue with configuring the port-security question, the final question of this task.

We should configure port-security for a specific MAC address on the interface of SW3. The problem is that SW2 has a different MAC address on its physical interface. So when we configure this. The interface will go in errdisable state.

Let’s configure the port-security feature as the question is expecting it.

SW3

SW3(config)# int e1/7 SW3(config-if)# switchport port-secu SW3(config-if)# switchport port-sec max 1 SW3(config-if)# switchport port-sec mac-address 1234.5678.abcd



SW3(config-if)#

Now we know for sure that the interface will be disabled, because the interface on SW2 is not using this MAC address as its source.

SW3(config)# show port-sec int e1/17 Configured Port Security : Enabled Opertional Port Security : Enabled Port Status : Secure Down Violation Mode : Shutdown Aging Time : 0 mins Aging Type : Absolute Maximum MAC Addresses : 1 Total MAC Addresses : 1 Configured MAC Addresses : 1 Sticky MAC Addresses : 0 Security violation count : 0 SW3(config)# show port-sec Total Secured Mac Addresses in System (excluding one mac per port) : 4 Max Addresses limit in System (excluding one mac per port) : 8188 ---------------------------------------------------------------------------- Secure Port MaxSecureAddr CurrentAddr SecurityViolation Security Action (Count) (Count) (Count) ---------------------------------------------------------------------------- port-channel102 100 0 0 Protect Ethernet1/11 5 5 0 Restrict Ethernet1/17 1 1 0 Shutdown ============================================================================ SW3(config)# show int e1/17 Ethernet1/17 is down

To ensure that the port-security feature works like expected, we need to change the MAC address of the SW2 interface. Then the interface comes back online and the port-security feature learns the correct MAC address.

SW2

SW2(config-if)# int e1/7 SW2(config-if)# shutdown SW2(config-if)# mac-address 1234.5678.abcd SW2(config-if)# no shutdown

Let’s verify the port-security feature to verify we completed this task.

SW3(config)# show port-sec address Total Secured Mac Addresses in System (excluding one mac per port) : 4 Max Addresses limit in System (excluding one mac per port) : 8188 ---------------------------------------------------------------------- Secure Mac Address Table ---------------------------------------------------------------------- Vlan Mac Address Type Remaining Remotely Remotely Ports age learnt aged (mins) out ---- ----------- ------ ------ ------- ----- ---- 1 BA01.DAD3.C0FF STATIC 0 No No Ethernet1/11 1 0010.4431.A1B3 STATIC 0 No No Ethernet1/11



100 1234.5678.ABCD STATIC 0 No No Ethernet1/17 1 0011.99FF.22AA STATIC 0 No No Ethernet1/11 1 1022.A0F5.B3DE STATIC 0 No No Ethernet1/11 1 5581.A09A.B00C STATIC 0 No No Ethernet1/11 ====================================================================== SW3(config)#

Now we finished Task 1 and we will continue with other various security features.

Task 2: DHCP Snooping, DAI, IP Source Guard

The features discussed in this chapter are all used in conjunction with each other. They all use the same database to perform there function.

It all comes down that we want to secure the switch against Layer 2 and Layer 3 spoofing attacks. Because most hosts in a network are using DHCP to get their IP address it’s very easy to create a Layer 2 to Layer 3 mapping on the switch to verify a correctness of the traffic coming in.

When you are enabling DHCP snooping on a switch, it means that by default all interfaces are seen as untrusted and the only packets that are allowed in are DHCP REQUEST packets from hosts. You can then enable certain interfaces to become trusted, which means that you are allowing DHCP servers to be connected through those interfaces.

Be aware when using this in a production network that the maximum number of bindings that can be kept on a Nexus switch is 2000.

One other option what the DHCP Snooping feature is able to do, except for securing the switch. Is that it can insert an Option 82 field in the DHCP packet. This field is used to identify the user which is accessing the DHCP server. Based on the connected VLAN and Port number you might get another address assigned from the DHCP server. Although this is usually only seen in Service Provider environments, it can be interesting when you need this feature.

The DHCP Snooping feature needs to be enabled globally and second it needs to be enabled on certain VLANs.

Let’s first configure the DHCP snooping feature.

SW1

SW1(config)# feature dhcp SW1(config)# ip dhcp snooping ? <CR> information DHCP Snooping information verify DHCP snooping verify vlan DHCP Snooping vlan SW1(config)# ip dhcp snooping SW1(config)# ip dhcp snooping vlan 50 SW1(config)# vlan 50 SW1(config-vlan)# exit



SW1(config)# interface Ethernet3/10 SW1(config-if)# switchport mode access SW1(config-if)# switchport access vlan 50 SW1(config-if)# ip dhcp snooping trust

The DHCP snooping feature is enabled on VLAN 50 and the interface Ethernet3/10 is used for receiving DHCP Offer messages by the DHCP server. Don’t forget to configure the port within the VLAN as the question states that the DHCP server is directly connected to the switch. So this needs to be an access interface in VLAN 50.

We are only required to configure SW1 as the question does not state anything about configuring other switches for this feature as well.

The next question states that we should include the Option 82 information regarding the VLAN and Port number into the DHCP packet.

This is done by enabling this feature. Typically you would expect this feature for DHCP Relay configurations, but in this case it’s supported in DHCP snooping as well.

SW1

SW1(config)# ip dhcp snooping information option

We can verify that the features are enabled in the correct way as we want them to.

SW1(config)# show ip dhcp snooping Switch DHCP snooping is enabled DHCP snooping is configured on the following VLANs: 50 DHCP snooping is operational on the following VLANs: 50 Insertion of Option 82 is enabled Verification of MAC address is enabled DHCP snooping trust is configured on the following interfaces: Interface Trusted ------------ ------- Ethernet1/10 Yes SW1(config)#

The next question asks to configure a verification. This verification can be performed by the switch to ensure that the DHCP packet is not spoofed. This means that the switch checks if the Source MAC address is equal to the MAC address specified within the DHCP packet.

SW1

SW1(config)# ip dhcp snooping verify mac-address SW1(config)#



This concludes the configuration for the DHCP snooping feature. At this point the switch starts building up a database of entries of clients that log-in through this switch.

Every time the DHCP server hands out a lease to a client, this information is stored into the DHCP snooping database. This database now offers a perfect Layer 2 MAC Address to Layer 3 IP address mapping on the switch.

This information is then leveraged by other features like the one we are configuring now.

Dynamic ARP Inspection (DAI) is a feature to really ensure you are safe against MAC address spoofing attacks.

Dynamic ARP inspection takes care of a number of features.

Source MAC spoofing

The Source MAC address of an ARP request is checked against the DHCP snooping database to ensure the request came from the correct interface

Optimal routing of packets

Because the DHCP snooping database knows where to find the ultimate destination of the ARP request. It only sends it out on trusted interfaces or even where the requested address is located

Checks for Valid ARP packets

The DAI process checks if ARP packets are send like they should be and verified that there is no faulty information in the packet.

Now it’s also possible to create exceptions to this case. You can create an Access List and apply that to the DAI process to ensure that ARP request sent for a specific range of IP addresses is always allowed.

And finally logging is a very important topic. How long do you want to keep track of log messages for this feature.

We start by configuring the initial question related to DAI. We want to ensure that VLAN 50 is secured for ARP Spoofing, meaning we need to enable DAI on that VLAN.

There is one exception and that is that it should not perform these checks on the port-channel interfaces going to the other switches.

Let’s configure these two questions.

SW1

SW1(config)# ip arp inspection vlan 50 SW1(config)# int port-chan101 SW1(config-if)# ip arp inspection ? trust Configure trust state SW1(config-if)# ip arp inspection trust



SW1(config-if)# int port-chan102 SW1(config-if)# ip arp inspection trust

Now let’s verify that it is enabled now.

SW1(config-if)# show ip arp inspec vlan 50 Source Mac Validation : Disabled Destination Mac Validation : Disabled IP Address Validation : Disabled Vlan : 50 ----------- Configuration : Enabled Operation State : Active DHCP logging options : Deny SW1(config-if)# show ip arp inspec interfaces Interface Trust State ------------- ----------- Port-channel101 Trusted Port-channel102 Trusted

Now we need to make an exception for this feature for the 198.18.50.0/28 subnet. As just discussed it’s possible to let DAI ignore certain ARP packets that are related to a special ARP Access List.

Which is what we will configure in this case.

SW1

SW1(config)# arp access-list VLAN50 SW1(config-arp-acl)# permit ip 198.18.28.0 0.0.0.15 mac any SW1(config-arp-acl)# exit SW1(config)# ip arp inspect filter VLAN50 vlan 50

Next we need to ensure that the last 50 messages are logged, both for denies and accepts.

SW1

SW1(config)# SW1(config)# ip arp inspect ? log-buffer Log Buffer Configuration validate Validate addresses vlan Enable/Disable ARP Inspection on vlans SW1(config)# ip arp inspect vlan 50 logging ? dhcp-bindings Logging of packet that match DHCP bindings SW1(config)# ip arp inspect vlan 50 logging dhcp-bindings ? all Log all packets that match DHCP bindings none Do not log packets permit Log DHCP Binding Permitted packets SW1(config)# ip arp inspect vlan 50 logging dhcp-bindings all SW1(config)# ip arp inspect log-buffer ? entries Number of entries for log buffer SW1(config)# ip arp inspect log-buffer entries ?



<1-1024> Number of entries for log buffer SW1(config)# ip arp inspect log-buffer entries 50

We first ensure that both permits and denies are logged, by configuring ‘all’ logging.

Then we configure the maximum amount of entries. By default this is set to 32 messages.

SW1(config)# show ip arp inspect vlan 50 Source Mac Validation : Disabled Destination Mac Validation : Disabled IP Address Validation : Disabled Vlan : 50 ----------- Configuration : Enabled Operation State : Active DHCP logging options : All SW1(config)# show ip arp inspect log Syslog Buffer Size : 50 Syslog Rate : 5 entries per 1 seconds SW1(config)#

The final question related to Dynamic ARP inspection is a feature to extend the default verification methods that it performs. You can enable some extended checks, like previously mentioned. One of them is checking for a Valid IP address.

This means that the DAI feature will check for invalid or faulty IP addresses in ARP packets and will drop these when it sees an invalid IP.

This feature is enabled with a single command.

SW1

SW1(config)# ip arp inspect validate ip

Now we finished the DAI configuration and we can check if all our settings are correctly applied .

SW1(config)# show ip arp inspection vlan 50 Source Mac Validation : Disabled Destination Mac Validation : Disabled IP Address Validation : Enabled Vlan : 50 ----------- Configuration : Enabled Operation State : Active DHCP logging options : All SW1(config)# show ip arp inspect interfaces Interface Trust State ------------- ----------- Ethernet1/1 Trusted



SW1(config)# show ip arp inspect statistics Vlan : 50 ----------- ARP Req Forwarded = 0 ARP Res Forwarded = 0 ARP Req Dropped = 0 ARP Res Dropped = 0 DHCP Drops = 0 DHCP Permits = 0 IP Fails-ARP Req = 0 IP Fails-ARP Res = 0 SW1(config)# show ip arp inspect log Syslog Buffer Size : 50 Syslog Rate : 5 entries per 1 seconds SW1(config)#

The next feature tested in this task is IP Source Guard.

This feature ensures a very strict IP spoofing security on a per interface level. The only way a packet is allowed through it either when there is a DHCP Snooping database entry for the IP address or there is a statically configured entry.

The feature is enabled per interface and cannot be enabled globally per VLAN. Keep in mind that the feature is only enabled on VLANs where DHCP snooping is already enabled!

Let’s start by configuring the first question.

SW1

SW1(config)# int e3/11 SW1(config-if)# ip verify ? source IP Source Guard related commands SW1(config-if)# ip verify source ? dhcp-snooping-vlan Vlans on which snooping is enabled SW1(config-if)# ip verify source dhcp-snooping-vlan SW1(config-if)# int e3/13 SW1(config-if)# ip verify source dhcp-snooping-vlan SW1(config-if)# int e3/14 SW1(config-if)# ip verify source dhcp-snooping-vlan

We then need to make an exception to interface Ethernet3/12 where a host with a statically configured IP address is connected to.

SW1

SW1(config)# ip source ? binding Static binding SW1(config)# ip source binding ? A.B.C.D IP address SW1(config)# ip source binding 198.18.50.254 ? E.E.E MAC address (Option 1) EE-EE-EE-EE-EE-EE MAC address (Option 2)



EE:EE:EE:EE:EE:EE MAC address (Option 3) EEEE.EEEE.EEEE MAC address (Option 4) SW1(config)# ip source binding 198.18.50.254 4019.a201.b04e ? vlan VLAN SW1(config)# ip source binding 198.18.50.254 4019.a201.b04e vlan 50 ? interface Interface SW1(config)# ip source binding 198.18.50.254 4019.a201.b04e vlan 50 interface ? ethernet Ethernet IEEE 802.3z port-channel Port Channel interface SW1(config)# ip source binding 198.18.50.254 4019.a201.b04e vlan 50 interface Ethernet3/12 SW1(config)# int e3/12 SW1(config-if)# ip verify source dhcp-snooping-vlan

Do not forget to enable the IP Source Guard feature on interface Ethernet3/12 as well, because otherwise IP spoofing attacks are still allowed trough as the interface is not protected!

The final 2 questions are about another feature related to IP Source Guard, but only applicable to Layer 3 interfaces.

On Layer 3 interfaces it’s easier to check if a spoofing is done as you can check if the source IP address is really coming from that particular IP subnet.

When this check is done, so the traffic should come from the subnet on which it is received on. It’s called ‘strict’ Unicast Reverse Path Forwarding Check (uRPF).

Another mode of this same technology is that the switch/router is able to reach the source IP address based on its routing table. This is called ‘loose’ mode.

We first need to configure the Layer 3 interface that we want to apply it to.

SW1

SW1(config-if)# feature interface-vlan SW1(config)# int vlan 50 SW1(config-if)# ip add 198.18.50.1/24 SW1(config-if)# no shut

The mode which we need for our question is the Loose mode as we agree on reaching the network in any way possible even including a default route.

We will configure the uRPF check on the VLAN 50 SVI interface.

SW1(config-if)# ip verify ? unicast Unicast Reverse Path Forwarding SW1(config-if)# ip verify unicast ? source Validation of source address SW1(config-if)# ip verify unicast source ? reachable-via Specify reachability check to apply to the source address



SW1(config-if)# ip verify unicast source reachable-via ? any Source is reachable via any interface rx Source is reachable via interface on which packet was received

When ‘rx’ is given as the configuration command then the strict mode would be activated. This means that traffic entering on a specific VLAN with IP subnet, it should be using a Source IP address of this VLAN and Subnet.

SW1(config-if)# ip verify unicast source reachable-via any ? <CR> allow-default Loose Default Route Unicast Reverse Path Forwarding SW1(config-if)# ip verify unicast source reachable-via any allow-default SW1(config-if)#

Which completes Task 2. Now we can continue with Task 3 which is about configuring Access Control Lists for both IP and MAC addresses.

Task 3: Access Control Lists

This task will explore the possibilities with Access Control Lists on the NX-OS. We will be configuring security for a number of layer 3 and layer 2 ports and interfaces.

ACL’s have been used since a very long time and has been available in IOS since many years. In the past you configured ACL’s with numbers. There were very specific reservations for these ACL numbers. There was also a strict distinction between 2 types of ACL’s, Standard and Extended.

Within NX-OS there is no distinction between these types of ACL’s and all ACL’s are using the Extended features. Which means that you can use all variables possible.

There are 2 types of ACL’s that are tested in the CCIE Data Center lab. In the previous task we already configured an ARP ACL. In this task we will be configuring IP Access Lists and MAC Access Lists. The biggest difference between the 2 is that an IP ACL can be used at almost any place in the switch to match certain IP traffic. MAC ACL’s are used for VLAN Access Control Lists (VACL). They offer the option to limit certain Layer 2 frames based on MAC addresses to pass through the switch or not. IP Access Lists can only match on IP addresses.

Be very careful when reading the questions in these tasks. The tasks can be very difficult in wording, so read very well in which direction the traffic is flowing. You also need to be prepared for questions about common applications, of which you should know the port numbers and the traffic characteristics.



The following types of traffic are able to be identified through MAC Access Lists:

• Source and/or Destination MAC address

• EtherType (Upper level protocol)

• VLAN ID

• 802.1p bits (Class of Service)

• IP (version 4) Access Lists are able to match a lot more different types of traffic characteristics.

• Source IP address

• Destination IP address

• Upper level protocol (TCP, UDP, EIGRP, GRE, OSPF, PIM, IP-IP, ESP, etc.)

• ICMP types / codes

• IGMP types

• DSCP values

• Source TCP/UDP port

• Destination TCP/UDP port

• Packet length

• Fragmented packets

• TCP bits (SYN, ACK, FIN, RST, etc.)

With all this variation of possible matching in the Access List, it gives you a lot of flexibility. Be aware for lengthy Access Lists when a lot of different rules are applied.

It’s also possible to create Access Lists that are connected to a time range. This means that the Access List is only applied during a specified time range. Which could be that Internet traffic is only allowed during business hours, or any other rule.

One important topic to think about in real life deployments is Logging. When every match and deny is logged, this could cause a lot of traffic to the Supervisor / Control-Plane as this might give thousands of log messages every second.



In our lab scenario we will not have any traffic running on the network, so we are safe to configure any feature we like.

We start by configuring our first IP Access Control List.

By default, when you are configuring an Access Control List. There is already one rule inserted into this list, which is a rule that denies all traffic when no other rule has been specified.

The rule is always present at the bottom of the ACL and is never shown. So keep in mind that all traffic that is not matched by any other rule is denied to pass through.

It’s a common practice, especially in CCIE scenarios to insert a deny for all traffic at the bottom of an ACL to ensure that you do not forget about it.

In the first question of the task it’s stated that we should secure VLAN 50 for traffic that should not be there. This means we will be using an IP Access Control List.

Let’s start by configuring our ACL for VLAN 50, be sure to pick names which are easily remembered and easy to type.

The first requirement is to create an entry for a host to be allowed to the VLAN. If you read carefully you will see that the questions are all related to traffic going TO the VLAN that we want to secure.

One very important decision to make when creating ACLs, is in which direction the ACL needs to be applied. The ACL can be applied in inbound and outbound direction.

Inbound

The inbound direction will match the traffic that comes IN to the interface, from the layer 2 domain / subnet. It comes IN to the switch via this interface and there the traffic is matched

Outbound

The outbound direction will match traffic that moves OUT of the interface onto the Layer 2 domain / IP subnet. This matches traffic going TO hosts on the subnet connected to the interface

For this task in this first Access List configuration we will be applying the ACL in outbound direction.

SW1

SW1(config)# ip access-list VLAN50_OUT

We give the ACL a recognizable name.

SW1(config-acl)# permit ip host 198.18.255.100 198.18.50.0 ? A.B.C.D Destination wildcard bits



The first task is to allow a single host (source address) to send traffic to the VLAN. We could apply this rule to ‘any’ destination address, but we need to be as specific as possible. So we will configure the ACL with a wildcard mask which complies with the entire VLAN.

SW1

SW1(config-acl)# permit ip host 198.18.255.100 198.18.50.0 0.0.0.255 SW1(config-acl)#

Now the ACL is not applied to the interface yet, but we will wait for the application of the ACL until we finished building all the rules that we are required to in this task.

Next question is that some Secure Web traffic, meaning SSL or HTTPS connections, is coming from a certain range. We need to ensure that VLAN 50 receives this traffic.

We need to allow a /18 mask. This means in wildcard masks that 6 bits of the third octet need to be a 1 which totals to 63.

SW1

SW1(config-acl)# permit tcp 198.18.128.0 0.0.63.255 eq 443 198.18.50.0 0.0.0.255 gt 1024 SW1(config-acl)#



The traffic is coming from servers, meaning that the source port will be Secure Web, as the VLAN 50 will contain the clients that used this port as the Destination port.

The clients are using non-reserved ports, which means that they are using ports that are not reserved for special cases (like HTTPS). The port range is all ports above 1024. Luckily NX-OS foresees in this and there is a operator to match ports that are greater than a specific number or range.

The next question will include a number of services that need to be allowed, but we cannot configure them inside of the ACL. We need another option to configure applications that need to be matched. This is found from a feature which is borrowed from the Cisco ASA firewalls. It’s now possible to combine certain equal parameters and use them in a single Access Control Entry (ACE). This is just for ease of configuration as the switch then automatically generates various entries based on the grouping.

These combinations are called object-groups. They can be made for both IP addresses and for Applications (port numbers). In this case we will need both, because we can only have two entries in the ACL to solve the question.

We cannot use one entry for the first subnet and the second entry for the second subnet, because we have both TCP and UDP applications in this question. It’s not possible to create a TCP and a UDP match in a single entry, so therefore we will require 2 entries.

Let’s start by configuring the object groups.

SW1

SW1(config)# object-group ? ip IP Object groups ipv6 IPv6 Object groups SW1(config)# object-group ip ? address Address object group port IP port object group (can be used in IPv4 and IPv6 access-lists) SW1(config)# object-group ip address ? WORD Object-group name (Max Size 64) SW1(config)# object-group ip address TASK3_5_IP SW1(config-ipaddr-ogroup)# ? <1-4294967295> Sequence number A.B.C.D A.B.C.D Network address of object-group member A.B.C.D/LEN A.B.C.D/nn Network prefix of the object-group member host Host address of the object-group member no Negate a command or set its defaults end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-ipaddr-ogroup)# 198.19.0.0/16 SW1(config-ipaddr-ogroup)# 198.18.192.0/24 SW1(config-ipaddr-ogroup)# exit



We first created the object-group related to the IP subnets that are matched. Instead of wildcard masks it’s also possible in NX-OS to use the CIDR notation, making an ACL much easier to read.

SW1

SW1(config)# object-group ip port TASK3_5_PORT SW1(config-port-ogroup)# ? <1-4294967295> Sequence number eq Match only packets on a given port number gt Match only packets with a greater port number lt Match only packets with a lower port number neq Match only packets not on a given port number no Negate a command or set its defaults range Match only packets in the range of port numbers end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-port-ogroup)# eq ? <0-65535> Port number SW1(config-port-ogroup)# eq 80 SW1(config-port-ogroup)# eq 53 SW1(config-port-ogroup)# eq 110 SW1(config-port-ogroup)# eq 25 SW1(config-port-ogroup)# exit

Next we configure our applications. Ensure you are aware of the port numbers for these common services. It’s easy rememberable, because you will need these port numbers in real-life as well while troubleshooting your network.

In this case don’t forget to add DNS to the group where TCP will be matched against, because DNS uses both a TCP and a UDP port number.

SW1

SW1(config)# ip access-list VLAN50_OUT SW1(config-acl)# permit tcp ? A.B.C.D Source network address A.B.C.D/LEN Source network prefix addrgroup Source address group any Any source address host A single source host SW1(config-acl)# permit tcp addrgroup ? WORD Address group name SW1(config-acl)# permit tcp addrgroup TASK3_5_IP ? A.B.C.D Destination network address A.B.C.D/LEN Destination network prefix addrgroup Destination address group any Any destination address eq Match only packets on a given port number gt Match only packets with a greater port number host A single destination host lt Match only packets with a lower port number



neq Match only packets not on a given port number portgroup Src port group range Match only packets in the range of port numbers SW1(config-acl)# permit tcp addrgroup TASK3_5_IP portgroup TASK3_5_PORT 198.18.50.0/24 ? <CR> ack Match on the ACK bit capture Enable packet capture on this filter for session dscp Match packets with given dscp value eq Match only packets on a given port number established Match established connections fin Match on the FIN bit gt Match only packets with a greater port number log Log matches against this entry lt Match only packets with a lower port number neq Match only packets not on a given port number packet-length Match packets based on layer 3 packet length portgroup Dst port group precedence Match packets with given precedence value psh Match on the PSH bit range Match only packets in the range of port numbers rst Match on the RST bit syn Match on the SYN bit time-range Specify a time range urg Match on the URG bit SW1(config-acl)# permit tcp addrgroup TASK3_5_IP portgroup TASK3_5_PORT 198.18.50.0/24 SW1(config-acl)# permit udp addrgroup TASK3_5_IP eq 53 198.18.50.0/24 SW1(config-acl)#

Then we finish the configuration by applying the object-groups to an Access List Entry. The first matches TCP applications based on 2 object-groups and the second only requires to match DNS for UDP traffic. So we are not using another object-group for that.

As stated before, also in the Access List itself you can use CIDR notation (/24) instead of wildcard masks, making your ACL a lot easier to read.

The next task is to connect the ACL to the VLAN. We can do this in 2 ways. We have an SVI for the VLAN on the Switch and because all this traffic is Layer 3 routes. We can connect it to the SVI of the VLAN, which would be the normal case!

The question prohibits us from doing that, which means we will need the second option and that is to apply this IP Access List to the VLAN like a VACL (VLAN ACL).

Normally those VACL’s are used to match and deny Layer 2 traffic, but it’s also possible to match based on IP ACLs within the VACL.

So we will be configuring a VACL to match our newly created IP ACL under. In this case it doesn’t really matter like with IP ACLs in which direction it is applied, as the VACL is always applied to ALL traffic of the VLAN, so both inbound and outbound traffic is matched against it.

SW1

SW1(config)# vlan access-map VLAN50



SW1(config-access-map)# ? action Specify the action clause match Specify the match clause no Negate a command or set its defaults statistics Enable per-entry statistics for the ACL end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-access-map)# match ? ip Display IP information ipv6 Display IPv6 information mac MAC configuration commands SW1(config-access-map)# match ip ip ipv6 SW1(config-access-map)# match ip ? address Match an access list SW1(config-access-map)# match ip address ? WORD List name (Max Size 64) SW1(config-access-map)# match ip address VLAN50_OUT SW1(config-access-map)# action forward SW1(config-access-map)# exit SW1(config)# vlan filter ? WORD List name (Max Size 64) SW1(config)# vlan filter VLAN50 ? vlan-list Specify list of VLANs to apply access control SW1(config)# vlan filter VLAN50 vlan-list 50 SW1(config)#

Traffic is matched in both directions. The action that is chosen here is to forward traffic. All traffic that is not allowed to pass through is not matched. So will hit the default deny anything rule.

Now we are done with our first IP Access List.

Next up is another question where we are not sure about all information.

We need to protect a certain host of which we do not know the IP address. Therefore we can only allow the entire VLAN traffic, but we need to protect the host. We do know on which port it is connected.

IP Access Lists can be applied at multiple places. One of the more common points is the VLAN interface (SVI). We just used a VACL to match an IP ACL. The last option is to configure the IP ACL under a physical Interface, so only traffic from that interface is matched against it.

SW2

SW2(config)# ip access-list E1/15_IN



SW2(config-acl)# deny tcp 198.18.50.0/24 host 198.19.0.25 eq 143

We create an ACL where we deny traffic towards a single host. The NX-OS knows a shortcut for this in terms of the host keyword in ACLs. Again keep attention to the direction of the traffic. In this case it’s a client accessing a server, which should be denied. Meaning that it’s the destination port number which is the application port.

SW2(config-acl)# permit ip any any

Finally we should only be securing against the particular host. All other traffic should still be passed through this interface. So we need to explicitly allow this.

Next step is to apply the ACL to the interface.

SW2

SW2(config)# int e1/15 SW2(config-if)# ip ? access-group Specify access control for packets port Port policy SW2(config-if)# ip port ? access-group Specify access control for packets SW2(config-if)# ip port access-group E1/15_IN ? in Inbound packets out Outbound packets SW2(config-if)# ip port access-group E1/15_IN in SW2(config-if)#

As the traffic is coming from the VLAN going towards the server, we need to apply this ACL in inbound direction.

Be very careful with using this command, as there are 2 different ones. The ‘access-group’ variant is used for Layer 3 traffic hitting the interface when it would be in a ‘no switchport’ configuration. Only when the ‘port’ command is used in this case, the traffic is matched against it while the port is in a normal switching mode.

Our next question is related to the security of the management interface.

You can apply a ACL for protecting the control-plane on 2 positions. The first is on the VTY. This means that you are putting an ACL on the Telnet and SSH lines to protect others from not connecting to the box via Telnet or SSH. This doesn’t protect all other protocols.

When you protect the management interface (mgmt0), you will be protecting all traffic, assuming that all management traffic is using this interface.

In this question we are looking for this application of the protection for rogue devices.

Access-list configuration is pretty standard and we should be familiar already. Because we are only allowed to create a single entry we will need to use a special command.



Within the ACL you can create a range command, where all the ports in the specified range are allowed in the entry.

As Telnet and SSH are using connecting port numbers, we can easily use this range command for our question.

SW1

SW1(config-if)# ip access-list MGMT SW1(config-acl)# deny tcp 192.0.2.0/24 range ? <0-65535> Port number SW1(config-acl)# deny tcp 192.0.2.0/24 range 22-23 ? ^ % Invalid number at '^' marker. SW1(config-acl)# deny tcp 192.0.2.0/24 range 22 ? <0-65535> Port number SW1(config-acl)# deny tcp 192.0.2.0/24 range 22 23 ? A.B.C.D Destination network address A.B.C.D/LEN Destination network prefix addrgroup Destination address group any Any destination address host A single destination host SW1(config-acl)# deny tcp 192.0.2.0/24 any range 22 23 SW1(config-acl)# permit ip any any SW1(config-acl)#

Remember that you need to protect against Telnet and SSH sessions that are being set-up towards the switch, which means the position of the port numbers.

Now we can apply the ACL to the interface. Now it doesn’t need to be a port ACL, but a regular ACL as the management interface is a Layer 3 interface.

SW1

SW1(config-acl)# int mgmt0 SW1(config-if)# ip ? access-group Specify access control for packets address Configure IP address on interface arp Configure ARP parameters directed-broadcast IP directed-broadcast port-unreachable Enable sending ICMP port-unreachable redirects Send ICMP Redirect messages unreachables Enable sending ICMP unreachables (other than port-unreachable) SW1(config-if)# ip access-group MGMT in SW1(config-if)#

Then finally don’t forget to apply this rule to all the switches as they all need to be protected.

SW2

SW2(config-if)# ip access-list MGMT SW2(config-acl)# deny tcp 192.0.2.0/24 any range 22 23 SW2(config-acl)# permit ip any any



SW2(config-acl)# SW2(config-acl)# int mgmt0 SW2(config-if)# ip access-group MGMT in

SW3

SW3(config-if)# ip access-list MGMT SW3(config-acl)# deny tcp 192.0.2.0/24 any range 22 23 SW3(config-acl)# permit ip any any SW3(config-acl)# SW3(config-acl)# int mgmt0 SW3(config-if)# ip access-group MGMT in

The next question is related to a special feature of the Nexus platform. It is related to SPAN sessions, but this session is related to ACLs in conjunction with SPAN / Monitor sessions.

This feature is called ACL Capture.

SW1

SW1(config)# hardware access-list ? capture Configure ACL capture lou LOU resource Hardware resource update Configure atomic/non-atomic update and default-result SW1(config)# hardware access-list capture ACL Capture enabled - disabling ACL logging for all VDCs

This feature first needs to be enabled globally, before it will work. With enabling of ACL Capture. You will disable all ACL logging.

SW1(config)# monitor session 1 type acl-capture

The type command is very important for this configuration. A normal SPAN session cannot be used.

SW1(config-acl-capture)# destination interface ethernet3/23 SW1(config-acl-capture)# no shutdown ^ % Invalid command at '^' marker.

For some reason, the normal command does not work in this case. You will need to use the abbreviated version.

SW1(config-acl-capture)# no shut SW1(config-acl-capture)# exit

We verify the configuration of the Monitor Session.

SW1(config)# show mon ses 1 session 1 --------------- type : acl-capture state : down (No operational src/dst) destination ports : Eth3/23



Next step is to create the Access-List and to connect the SPAN session to the ACL.

SW1

SW1(config)# ip access-list SPAN SW1(config-acl)# permit tcp any any ? <CR> ack Match on the ACK bit capture Enable packet capture on this filter for session dscp Match packets with given dscp value eq Match only packets on a given port number established Match established connections fin Match on the FIN bit fragments Check non-initial fragments gt Match only packets with a greater port number log Log matches against this entry lt Match only packets with a lower port number neq Match only packets not on a given port number packet-length Match packets based on layer 3 packet length portgroup Dst port group precedence Match packets with given precedence value psh Match on the PSH bit range Match only packets in the range of port numbers rst Match on the RST bit syn Match on the SYN bit time-range Specify a time range urg Match on the URG bit SW1(config-acl)# permit tcp any any capture ? session Session ID <1-48> for this session SW1(config-acl)# permit tcp any any capture sess 1 SW1(config-acl)# permit ip any any

The session is connected to the entry that you want to be copied to the SPAN session. Don’t forget to finish the configuration with a permit for all other traffic. As this ACL is just applied to an interface. It would break the other traffic moving through the interface.

Finally the ACL should be applied to the interface. Again pay attention to if this is a Layer 2 or a Layer 3 interface.

SW1

SW1(config-acl)# interface e3/22 SW1(config-if)# ip port access-group SPAN in SW1(config-if)#



The next question is related to Layer 2 security. Which means that we should create a MAC Access List for VLAN 50.

Pay close attention to how the question is formulated. The access-list should only match traffic from the server farm. Meaning that the mentioned MAC addresses are used as source addresses for the access list.

The MAC access-list is created in a very similar way as that IP ACLs are created.

SW1

SW1(config)# mac access-list VLAN50 SW1(config-mac-acl)# ? <1-4294967295> Sequence number capture Enable packet capture on this filter for session deny Specify packets to reject no Negate a command or set its defaults permit Specify packets to forward remark Access list entry comment statistics Enable per-entry statistics for the ACL end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-mac-acl)# permit ? E.E.E Source MAC address (Option 1) EE-EE-EE-EE-EE-EE Source MAC address (Option 2) EE:EE:EE:EE:EE:EE Source MAC address (Option 3) EEEE.EEEE.EEEE Source MAC address (Option 4) any Any source address SW1(config-mac-acl)# permit 0bad.c0ff.ee00 ? E.E.E Source wildcard bits (Option 1) EE-EE-EE-EE-EE-EE Source wildcard bits (Option 2) EE:EE:EE:EE:EE:EE Source wildcard bits (Option 3) EEEE.EEEE.EEEE Source wildcard bits (Option 4)

Wildcard masks work exactly the same as with IP Access Lists. In this case we chose a relatively easy mask.

SW1

SW1(config-mac-acl)# permit 0bad.c0ff.ee00 0000.0000.00ff ? E.E.E Destination MAC address (Option 1) EE-EE-EE-EE-EE-EE Destination MAC address (Option 2) EE:EE:EE:EE:EE:EE Destination MAC address (Option 3) EEEE.EEEE.EEEE Destination MAC address (Option 4) any Any destination address SW1(config-mac-acl)# permit 0bad.c0ff.ee00 0000.0000.00ff any ? <CR> <0x0-0xffff> MAC protocol number aarp Appletalk AARP appletalk Appletalk capture Enable packet capture on this filter for session cos CoS value



decnet-iv DECnet Phase IV diagnostic DEC Diagnostic Protocol etype-6000 Ethertype 0x6000 etype-8042 Ethertype 0x8042 ip IP (Internet Protocol V4) lat DEC LAT lavc-sca DEC LAVC,SCA mop-console DEC MOP Remote console mop-dump DEC MOP dump time-range Specify a time range vines-echo VINES Echo vlan VLAN number

You can specify which traffic is allowed, but this is not necessary in our case.

SW1

SW1(config-mac-acl)# permit 0bad.c0ff.ee00 0000.0000.00ff any SW1(config-mac-acl)# exit

Then the access-list needs to be applied to a VACL. We already created this VACL for VLAN 50. So ensure you do not override the previous settings.

VLAN Access Maps work in a similar way as Route-Maps do. So we can create additional entries in the map. The first will be for the IP ACL and the second for the MAC ACL.

SW1

SW1(config)# vlan access-map VLAN50 ? <CR> <1-4294967295> Sequence number SW1(config)# vlan access-map VLAN50 20 SW1(config-access-map)# match mac ? address Match an access list SW1(config-access-map)# match mac address ? WORD List name (Max Size 64) SW1(config-access-map)# match mac address VLAN50 SW1(config-access-map)# action forward SW1(config-access-map)# exit

After the second entry in the VACL is created, we can verify that it did not overrule any of the previous configuration of the VACL.

SW1(config)# show run | sec access-map vlan access-map VLAN50 10 match ip address VLAN50_OUT action forward vlan access-map VLAN50 20 match mac address VLAN50 action forward SW1(config)#



The next question is that we should be gathering statistics about all entries. This is enabled with a single command in each of the VACL sequences.

SW1

SW1(config)# vlan access-map VLAN50 10 SW1(config-access-map)# statistics SW1(config-access-map)# vlan access-map VLAN50 20 SW1(config-access-map)# statistics SW1(config-access-map)#

Now we can see that entries are logged per entry.

SW1(config-access-map)# show vlan access-map Vlan access-map VLAN50 10 match ip: VLAN50_OUT action: forward statistics per-entry Vlan access-map VLAN50 20 match mac: VLAN50 action: forward statistics per-entry

The final question of this task is related to COntrol Plane Policing (CoPP). Within the Nexus platform the CoPP is already configured, unlike in IOS. Rate-limiters / Policers are configured for all control-plane protocols and are in working.

They are using templates according to the features that are used on the switch. By default these are good templates and chances are that you will never have to change them. Therefore we do not focus on tweaking these here.

However, if you are using the box strongly for Layer 2 or Layer 3 purposes you will need to change the template, so the routing protocols are given more space over the Layer 2 control protocols.

Default CoPP Policy (copp-system-policy-default)

Scaled Layer 2 CoPP Policy (copp-system-policy-scaled-l2)

Scaled Layer 3 CoPP Policy (copp-system-policy-scaled-l3)

Customized CoPP Policy (copp-system-policy-customized)

The question states that we should be using optimizations for Layer 3 routing and we have a template for that! This is exactly the change that we need to perform on both SW2 and SW3.

SW2

SW2(config-access-map)# control-plane SW2(config-cp)# service-policy input copp-system-policy-scaled-l3

SW3

SW3(config-access-map)# control-plane SW3(config-cp)# service-policy input copp-system-policy-scaled-l3



To verify that the command has been executed correctly, we can use the following show output.

SW2(config-cp)# show copp status Last Config Operation: service-policy input copp-system-policy-scaled-l3 Last Config Operation Timestamp: 01:21:43 UTC Nov 9 2012 Last Config Operation Status: Success Policy-map attached to the control-plane: copp-system-policy-scaled-l3 SW2(config-cp)#

Now SW2 and SW3 are using a correct CoPP policy scaled for the primary use of Layer 3 Routing on the box.

With this final question we finished the task and we will continue with Task 4 about AAA services.



Task 4: AAA services

This fourth task is focusing on Authentication, Authorization and Accounting (AAA) services which are available on the NX-OS platform.

AAA services are used widely for authentication and proper rights management on Cisco devices. Within NX-OS the options are widened.

AAA services are used for management options within the NX-OS device. You can administer WHO can log-in (Authentication), WHAT the person can do (Authorization) and WHEN the person did something (Accounting). This ensures a very granular possibility.

Now the Authorization within NX-OS is changed thoroughly. This is because NX-OS is using a very different authorization scheme (who can access what) than classical IOS.

In Classical IOS users were given a Privilege Level. This privilege level was then connected to certain commands that the user was able to execute once it logged in to the router or switch running IOS. Within NX-OS this has changed to ‘roles’. A user can be given one or more roles and connected to a role you can allow certain command sets, or allow the configuration of certain features. When features are enabled in a role, the user is allowed to execute all commands related to this feature including configuration and show commands.

The role based authentication (RBAC) is not discussed in this security chapter as that is more related to management of the device. This is why this feature is discussed in the management chapter.

The Authentication commands are the most important as they determine when a user is allowed to log-in to a device or not. In classical IOS you had to configure either AAA services or use the normal logging in using a simple password and enable password.

In NX-OS there is no such thing as an Enable password, but just a username and password connected to a certain role. When you enable AAA services in classical IOS you really have to be careful when enabling these features. As you might even get logged out of the box. Especially in the CCIE lab exams this is very dangerous as you might need to go to the proctor and ask for a reload of your device. Much better to prevent that from happening.

Within NX-OS this is simpler. There is no distinction between enabling the AAA services, versus a classical mode. You are always in the same mode, making life much easier.

You can configure whether the device authenticates using the local database, which is default, or using an AAA server.

NX-OS knows the 2 most widely used AAA protocols.

RADIUS

Based on open standards and used in all platforms from all vendors.



Based on UDP traffic, but including guarantees if requests were received or not.

TACACS+

Cisco proprietary protocol. However it does finds its integration in other vendor products

Based on TCP traffic

There are not real differences in RADIUS or TACACS+, other than the type of traffic. As of features they both offer equal features. Sometimes applications are using only RADIUS or TACACS+, but overall they are equal in terms of features and functionality.

In the first few questions of this task we are configuring the AAA servers. We cannot use any global configuration for that, which means we should be using ‘groups’ for server configurations. The advantage of groups over global configuration is that groups can be used in different places in the configuration. The groups also offer failover to other AAA servers when the primary one failed.

First we should configure the RADIUS and TACACS+ servers while doing configuration in groups.

SW1

SW1(config)# aaa group ? server Configure aaa server group SW1(config)# aaa group server ? radius RADIUS server group name SW1(config)# aaa group server radius ? WORD RADIUS server group name (Max Size 127) SW1(config)# aaa group server radius RADIUS_SERVERS SW1(config-radius)# ? deadtime Duration for which non-reachable server is skipped no Negate a command or set its defaults server RADIUS server name or IP address source-interface Source interface to be used to reach radius server use-vrf Vrf to be used to contact servers in this group end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-radius)# server 172.16.100.201 specified RADIUS server not found, first configure it using radius-server host ... and then retry

Now we can’t configure a RADIUS server within a group before the server has been globally configured. So we need to configure it in global configuration first.

SW1

SW1(config-radius)# exit



SW1(config)# radius-server host 172.16.100.201 key IPexpertAAA SW1(config)# radius-server host 172.16.100.201 key IPexpertAAA ? <CR> accounting Use for accounting auth-port RADIUS server's port for authentication authentication Use for authentication pac Secure Radius Enable retransmit RADIUS server retransmit count timeout RADIUS server timeout period in seconds SW1(config)# radius-server host 172.16.100.201 key IPexpertAAA SW1(config)# aaa group server radius RADIUS_SERVERS SW1(config-radius)# server 172.16.100.201 SW1(config-radius)# use-vrf management SW1(config-radius)# source-interface mgmt0 SW1(config-radius)# exit SW1(config)#

After configuring the global server we can configure the server inside the group. Where we can also specify the source interface and source VRF that we want to use for RADIUS requests.

Next we will configure the TACACS+ server configuration.

SW1

SW1(config)# taca? ^ % Invalid command at '^' marker. ? ^ % Invalid command at '^' marker. SW1(config)# taca ^ % Invalid command at '^' marker. SW1(config)# aaa group server ? radius RADIUS server group name SW1(config)# aaa group server tacacs ^ % Invalid command at '^' marker. SW1(config)#

By default only RADIUS configuration is enabled. For TACACS+ you need to enable the feature first.

SW1

SW1(config)# SW1(config)# feature tacacs+ SW1(config)# SW1(config)# tacacs-server host 172.16.100.202 ? <CR> key TACACS+ shared secret port TACACS+ server port test Paremeters to send test packets timeout TACACS+ server timeout period in seconds SW1(config)# tacacs-server host 172.16.100.202 key IPexpertAAA



SW1(config)# aaa group server ? radius RADIUS server group name tacacs+ TACACS+ server group name SW1(config)# aaa group server tacacs+ TACACS_SERVERS SW1(config-tacacs+)# server 172.16.100.202 SW1(config-tacacs+)# use-vrf management SW1(config-tacacs+)# source-interface mgmt0

Now we have configured our servers both for TACACS+ and RADIUS.

As there is no specification on which switch the configuration should be applied, we need to configure it on all switches.

SW2

SW2(config)# radius-server host 172.16.100.201 key IPexpertAAA SW2(config)# aaa group server radius RADIUS_SERVERS SW2(config-radius)# server 172.16.100.201 SW2(config-radius)# use-vrf management SW2(config-radius)# source-interface mgmt0 SW2(config-radius)# exit SW2(config)# feature tacacs SW2(config)# tacacs-server host 172.16.100.202 key IPexpertAAA SW2(config)# aaa group server tacacs+ TACACS_SERVERS SW2(config-tacacs+)# server 172.16.100.202 SW2(config-tacacs+)# use-vrf management SW2(config-tacacs+)# source-interface mgmt0

SW3

SW3(config)# radius-server host 172.16.100.201 key IPexpertAAA SW3(config)# aaa group server radius RADIUS_SERVERS SW3(config-radius)# server 172.16.100.201 SW3(config-radius)# use-vrf management SW3(config-radius)# source-interface mgmt0 SW3(config-radius)# exit SW3(config)# feature tacacs SW3(config)# tacacs-server host 172.16.100.202 key IPexpertAAA SW3(config)# aaa group server tacacs+ TACACS_SERVERS SW3(config-tacacs+)# server 172.16.100.202 SW3(config-tacacs+)# use-vrf management SW3(config-tacacs+)# source-interface mgmt0

The next question is related to monitoring the RADIUS servers. Within NX-OS it’s possible to pro-actively monitor the reach ability of the AAA servers, which prevents timeouts while logging in.

First we need to configure the time-out values on the groups before it’s declared automatically dead.

SW1

SW1(config)# aaa group server radius RADIUS_SERVERS SW1(config-radius)# deadtime ? <0-1440> Length of time, in minutes SW1(config-radius)# deadtime 22 SW1(config-radius)# exit



SW1(config)# aaa group server tacacs+ TACACS_SERVERS SW1(config-tacacs+)# deadtime 22 SW1(config-tacacs+)# exit

SW2

SW2(config)# aaa group server radius RADIUS_SERVERS SW2(config-radius)# deadtime 22 SW2(config-radius)# exit SW2(config)# aaa group server tacacs+ TACACS_SERVERS SW2(config-tacacs+)# deadtime 22 SW2(config-tacacs+)# exit

SW3

SW3(config)# aaa group server radius RADIUS_SERVERS SW3(config-radius)# deadtime 22 SW3(config-radius)# exit SW3(config)# aaa group server tacacs+ TACACS_SERVERS SW3(config-tacacs+)# deadtime 22 SW3(config-tacacs+)# exit

Next we configure the monitoring. This is done by doing a simple AAA log-in to the server. Which is why we need the username and password.

SW1

SW1(config)# radius-server ? deadtime Duration for which non-reachable server is skipped directed-request Enable direct authentication requests to server host RADIUS server's DNS name or its IP address key Global RADIUS server shared secret retransmit Global RADIUS server retransmit count test Parameters to send test packets timeout Global RADIUS server timeout period in seconds SW1(config)# radius-server test ? idle-time Time interval for monitoring the server password User password in test packets username User name in test packets SW1(config)# radius-server test username ipexpert password IPexpert123 idle-time 2 SW1(config)# tacacs-server test username ipexpert password IPexpert123 idle-time 2 SW1(config)#

SW2

SW2(config)# radius-server test username ipexpert password IPexpert123 idle-time 2 SW2(config)# tacacs-server test username ipexpert password IPexpert123 idle-time 2

SW3

SW3(config)# radius-server test username ipexpert password IPexpert123 idle-time 2 SW3(config)# tacacs-server test username ipexpert password IPexpert123 idle-time 2



By default there is a lengthy time-out value for AAA requests. Which means that users need to wait long before they can log-in using another AAA server or perform a fall-back to the local database.

You are also able to configure the amount of retransmits that should be done.

Therefore we will configure a short time-out value for log-in requests.

SW1

SW1(config)# radius-server timeout 2 SW1(config)# tacacs-server timeout 2 SW1(config)#

SW2


SW3


This finalizes the configuration for AAA servers on the devices.

Let’s verify if our configuration was applied correctly.

SW1(config)# show radius-server retransmission count:1 timeout value:2 deadtime value:0 source interface:any available Global Idle Time:2 Global Test Username:ipexpert Global Test Password:IPexpert123 total number of servers:2 following RADIUS servers are configured: 172.16.100.201: available for authentication on port:1812 available for accounting on port:1813 RADIUS shared secret:******** SW1(config)# show tacacs-server timeout value:2 deadtime value:0 source interface:any available Global Idle Time:2 Global Test Username:ipexpert Global Test Password:IPexpert123



total number of servers:1 following TACACS+ servers are configured: 172.16.100.202: available on port:49 TACACS+ shared secret:******** SW1(config)#

In the next questions we will be applying the AAA configuration to the log-in process of the switches.

Keep in mind that by default when you configure authentication, it only applies to the telnet and SSH sessions. Console sessions are left out, as they are configured with a separate command.

This is done, first of all to ensure you do not configure any faulty parameters causing you to be locked out from the switch. Besides that it’s common that you configure the console connection only with a local authentication, because usually you only perform console log-in, when there is something wrong with the switch. Meaning that the chance is high that you are also not able to reach the AAA servers anymore.

First we only want SW2 to authenticate to the RADIUS server and a fall-back to the local database.

The local database should only be queried once the RADIUS server is dead and does not respond. In case the RADIUS server replies with a reject message. It’s not falling back to the local database. This is because it means that the user is not allowed to log-in to the device. So a fall-back would violate the authentication rules.

Now when the fall-back would be done. The switch would automatically use the same username and password as would’ve been typed in. First we configure RADIUS authentication on SW2.

SW2

SW2(config)# aaa authentication ? dhchap Configure methods for dhchap login Configure methods for login SW2(config)# aaa authentication login default ? fallback Configure fallback behavior group Specify server groups local Use local username authentication none No authentication SW2(config)# aaa authentication login ? ascii-authentication Enable ascii authentication chap CHAP authentication for login console Configure console methods default Configure default methods error-enable Enable display of error message on login failures mschap MSCHAP authentication for login mschapv2 MSCHAP V2 authentication for login SW2(config)# aaa authentication login default ?



fallback Configure fallback behavior group Specify server groups local Use local username authentication none No authentication SW2(config)# aaa authentication login default group ? WORD Server group name (Max Size 127) SW2(config)# aaa authentication login default group RADIUS_SERVERS ? <CR> WORD Server group name (Max Size 127) none No authentication SW2(config)# aaa authentication login default group RADIUS_SERVERS SW2(config)# aaa authentication login default fallback ? error Fallback in case all AAA servers configured for remote authentication are unreachable (Authentication error) SW2(config)# aaa authentication login default fallback error ? local Fallback to local authentication SW2(config)# aaa authentication login default fallback error local SW2(config)#

Now when we do not configure any second method the switch will automatically fall-back to the local username/password database and will not re-ask for a password, because it’s the default. In our case we configured the fallback to be sure enough.

Next up is the console authentication which was just mentioned to be done differently from the SSH and Telnet log-in. As there is no specification of the switch. It should be configured on all.

SW1

SW1(config)# aaa authentication login console local

SW2


SW3


Next question relates to the configuration of SW3 for authentication. This is where we need to use a Cisco Proprietary protocol, meaning that we will use TACACS+ for this host.

SW3

SW3(config)# aaa authentication login default group TACACS_SERVERS



Next up is that users without a role, should not be able to log-in to the device. Users will get roles pushed through RADIUS and TACACS+ attribute configurations.

By default when users do not get this attribute pushed. They will be configured with the ‘network-operator’ or ‘vdc-operator’ role.

This can be disabled with a single command.

SW1

SW1(config)# no aaa user default-role

SW2


SW3


The next question is related to the notification for AAA servers if the servers are available.

By default users do not get to know how they have been logged in, of course for security purposes. We can enable this feature.

SW1

SW1(config)# aaa authentication login ? ascii-authentication Enable ascii authentication chap CHAP authentication for login console Configure console methods default Configure default methods error-enable Enable display of error message on login failures mschap MSCHAP authentication for login mschapv2 MSCHAP V2 authentication for login SW1(config)# aaa authentication login error-enable

SW2

SW2(config)# aaa authentication login error-enable

SW3

SW3(config)# aaa authentication login error-enable

This can be verified when logging into the switch using SSH. There it’s shown that the AAA servers were not reachable for log-in and that the local database is used instead.

TerminalServer#ssh 172.16.100.1 Nexus 5000 Switch Password: Remote AAA servers unreachable; local authentication done



Bad terminal type: "xterm-256color". Will assume vt100. Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php SW1#

The next question is related to how NX-OS stores usernames and passwords in it’s database.

By default it stores them with an MD5 hashed password in the database, but this can be even stronger. By using the AES encryption algorithm.

This is known as ‘type 6’ passwords. Where MD5 hashes are known as ‘type 5’.

Before this feature can be enabled, you will need to configure a Master Key which is used for encrypting the passwords.

There is no key specified, so you can use something you like.

SW1

SW1# key config-key ascii New Master Key: IPexpert123 Retype Master Key: IPexpert123

Next we can enable the feature.

SW1

SW1# conf t SW1(config)# feature password encryption aes

We should be encrypting the currently configured usernames and passwords so they are also using the new encryption mechanism.

SW1(config)# exit SW1# encryption re-encrypt obfuscated

Finally, don’t forget to enable it on all the switches, because there was no specification given on which switch it should be enabled.

SW2

SW2# key config-key ascii New Master Key: IPexpert123switch2 Retype Master Key: IPexpert123switch2






SW2# conf t SW2(config)# feature password encryption aes SW2(config)# exit SW2# encryption re-encrypt obfuscated

As the authentication is a local decision we can use any key we want on the switch. It’s even recommended because this increases the security between the switches.

SW3

SW3# key config-key ascii New Master Key: IPexpert123switch3 Retype Master Key: IPexpert123switch3 SW3# conf t SW3(config)# feature password encryption aes SW3(config)# exit SW3# encryption re-encrypt obfuscated

Finally we should enable accounting towards the RADIUS server to ensure that all commands that any user typed in will be logged.

SW2

SW2(config)# aaa accounting default group RADIUS_SERVERS

As previously mentioned. NX-OS uses another mechanism than Classical IOS for giving rights to users. This means that all AAA servers are currently configured to support the IOS privilege levels. These are levels between 0 and 15 for determining what users are able to do on the switches.

The NX-OS device performs a default mapping for privilege levels received through TACACS+ to default existing roles on the devices.

Privilege Mapped role

0 Some Show and Exec commands can be executed

1 to 13 Same as 0 and separately configured features using role configuration related to privilege levels

14 vdc-admin role

15 network-admin role



You can also configure privilege levels within NX-OS. This needs to be enabled to configure the privileges connected to the usernames. It’s also possible to configure the privilege levels as roles within the configuration.

To just enable backwards compatibility what we want, we just need to enable the feature.

SW3

SW3(config)# feature privilege

Next up is the forcing the users to configure strong passwords (or not).

A strong password consists out of the following characteristics.

Minimal of 8 characters long

Does not contain many consecutive characters (such as abcd)

Does not contain many repeating characters (such as aaabbb)

Does not contain dictionary words

Does not contain proper names

Contains both uppercase and lowercase characters

Contains at least 1 or more numbers

By default this strength check for passwords is enabled. When you disable and re-enable the feature. It does not check for the current passwords to be configured with a strong password or not. Keep this in mind in a production network.

In this case we should disable the check on SW3.

SW3

SW3(config)# no password ? strength-check Strength check of password SW3(config)# no password strength-check SW3(config)#

The final question of this task is to configure a username which expires. After the expiration date and time. The user will not be able to log-in anymore.

SW3

SW3(config)# username rick ? <CR> expire Expiry date for this user account(in YYYY-MM-DD format) keypair Generate SSH User Keys password Password for the user priv-lvl Privilege level which the user is to be assigned to role Role which the user is to be assigned to ssh-cert-dn Update cert dn



sshkey Update ssh key for the user for ssh authentication SW3(config)# username rick expire ? WORD Expiry in YYYY-MM-DD format (Max Size 10) SW3(config)# username rick expire 2013-12-31 ? <CR> password Password for the user priv-lvl Privilege level which the user is to be assigned to role Role which the user is to be assigned to SW3(config)# username rick expire 2013-12-31 password IPexpertIsAwesome role network-admin SW3(config)#



Now we finished Task 4 and we will continue with a feature not specifically mentioned on the blueprint, but very likely to be tested as other things might be depending on it.

Besides that it is using the RADIUS configuration from this task to simulate authentications.

Task 5: 802.1X

This fifth task will be about configuring the IEEE 802.1X feature. This feature takes care of ensuring the identity of users connecting to the network.

This is done by offering users a username/password dialog screen. Where they need to enter there network credentials before the proper VLAN is configured on the switchport.

Without authentication users can not access the network!

This technology is actually related to LAN security and usually found in workgroup switches. Within a datacenter environment this is usually not seen.

The code is implemented in NX-OS, but only available in the Nexus 7000. The Nexus 5000 has no features for configuring this.

The items is not on the CCIE Data Center lab blueprint, but the chances are that it is treated in the lab exam, because for the next task about Cisco TrustSec it is required.

Because the 802.1X feature is not fully integrated in the OS it doesn’t have all the features that it would usually have in a LAN environment.

The feature will not be a large portion of the exam, which is why we will not focus a lot on the feature here.

First we are required to configure some basic functionality of 802.1X.

We need to be sure that on some of the interfaces of SW1 users are authenticated before they get access to the network. They should be authenticated against the RADIUS server which we configured in the previous task.

SW1

SW1(config)# feature dot1x SW1(config)# aaa authentication dot1x default group RADIUS_SERVERS SW1(config)#

First we configure the RADIUS authentication that should be used, before any of the interface configuration is done.

SW1

SW1(config)# int e3/25-31 SW1(config-if)# dot1x port-control auto SW1(config-if)#



The interfaces are enabled with a single command. Relatively easy feature to enable, especially because not all code is implemented on the Nexus 7000 platform. There is no such thing as Guest VLANs or other advanced features.

The feature is purely used for authentication of the user against the network.

By default the 802.1X standard allows for a single host to be connected through the interface. When there are multiple hosts connected through the same interface they are not allowed access unless they authenticate.

This is disabled by default, so when we want this, we need to enable support for multiple hosts.

SW1

SW1(config)# int e3/26-27 SW1(config-if)# dot1x host-mode multi-host

Next we should configure a re-authentication timer. By default the re-authentication is disabled and when a host connects and authenticates it can access the network forever. As soon as it disconnects and connects again it needs to re-authenticate again.

When we configure re-authentication, the host will get a request to authenticate to the switch again, to be sure that still the same user is connected to this interface.

This re-authentication can be configured globally or per interface. In our case we will configure it at interface level, because we are not allowed to configure it globally.

SW1

SW1(config)# int e3/25-31 SW1(config-if)# dot1x re-authentication SW1(config-if)# dot1x timeout re-authperiod 3600



Re-authentication timers are given in seconds. In our case for 1 hour we need to specify 3600 seconds as re-authentication timer.

The next task is about giving access to a printer on our network which has no client for 802.1X authentication. We still want the device to get access to the network in the correct VLAN, but as the printer has no client software, it cannot do this by itself so we need to help in this process.

What we can do is to use the MAC address of the client, in this case the printer, as the authentication parameter towards RADIUS.

The RADIUS server will receive a request for authentication filled out with the clients MAC address. We need to make sure an entry exists for this MAC address and then the port is still authenticated.

This feature is by default disabled and can be enabled on a per interface level.

SW1

SW1(config)# int e3/31 SW1(config-if)# dot1x mac-auth-bypass

The next question states that a client is allowed to re-issue authentication up to 4 times. This means that it is allowed to re-authenticate after rejection of the RADIUS Server for another 3 times before the interface is fully shutdown.

We can also configure this parameter both globally and per interface. We will configure it per interface as well.

SW1

SW1(config)# int e3/25-31 SW1(config-if)# dot1x max-reauth-req 4

Finally we should ensure that all activity is logged via accounting towards the RADIUS server.

SW1

SW1(config)# dot1x radius-accounting SW1(config)# aaa accounting dot1x default group RADIUS_SERVERS



With this last configuration we finished the 802.1X task. Next up is a true Data Center feature called Cisco TrustSec.

Task 6: Cisco TrustSec

The final task of this chapter is about configuring Cisco TrustSec. This technology was introduced within the Nexus series. It supports encryption of traffic in the data-plane. This means it has no impact on the forwarding performance while still begin highly secure.

Besides adding encryption it also supports multi-tenancy in the network. This is because the feature introduces a tag, called the Security Group Tag (SGT). This tag is added to the packet, like a 802.1Q header.

Cisco TrustSec (CTS) works together with 802.1X for full security control over the entire data center network.

The tagging requires support in hardware, so be aware of this when selecting platforms and CTS is a requirement.

Besides encrypting traffic and tagging traffic it can also be used to selectively add switches to the network as they are authenticated. This works almost the same as Port Security in MDS switches and Fibre Channel links. Switches send CHAP or EAP challenges to each other to authenticate.

All the verification can also be performed by a RADIUS server. Be aware that only Cisco ACS supports the TrustSec features at the time of this writing.



As we can not test the full features in this lab, because of limited available hardware and software. We will focus on the authentication of the switches themselves. So a secure data center network is created.

We first enable Cisco TrustSec and the authentication against the RADIUS server.

SW1

SW1(config)# feature dot1x SW1(config)# feature cts SW1(config)# cts device-id SW1 password SW1p@ssw0rd

When enabling the feature it’s required to have it configured with a local ID and password. We will use the password mentioned later in this task for the other questions.

SW1

SW1(config)# aaa authentication cts default group RADIUS_SERVERS SW1(config)# aaa authorization cts default group RADIUS_SERVERS

Do not forget to enable this on all the switches!

SW2

SW2(config)# feature cts SW2(config)# cts device-id SW2 password SW2p@ssw0rd SW2(config)# aaa authentication cts default group RADIUS_SERVERS SW2(config)# aaa authorization cts default group RADIUS_SERVERS

SW3

SW3(config)# feature cts SW3(config)# cts device-id SW3 password P@ssw0rdSW3 SW3(config)# aaa authentication cts default group RADIUS_SERVERS SW3(config)# aaa authorization cts default group RADIUS_SERVERS

Next we should enable Cisco TrustSec (CTS) on the interfaces that we enabled 802.1X on.

SW1

SW1(config)# interface e3/25-31 SW1(config-if)# cts dot1x SW1(config-if)# shut SW1(config-if)# no shut



The enabling of CTS requires a flap of the interface before it will work. In our case we have no hosts connected to the interface, so it doesn’t really matter. In a real life network the shutdown and no shutdown of the interface is required for the CTS feature to work properly.

The last few questions of this chapter are about another feature of Cisco TrustSec, which is the authentication of switches to ensure your datacenter network is only consisting of switches that you expect. Connections between switches do not come online automatically without configuring a proper authentication.

This feature is called the SGT Exchange Protocol (SXP). It is primarily used for giving out the Security Tags to other switches, but can therefore also be used for authenticating switches.

This is where we are going to use it for.

It requires manual configuration of the authentication keys as we cannot use RADIUS for this task. We also cannot use the mgmt0 interface for the SXP protocol, so we need to create SVI’s which we are allowed to.

SW1

SW1(config)# vlan 99 SW1(config-vlan)# exit SW1(config)# interface vlan 99 SW1(config-if)# ip address 198.18.99.1/24 SW1(config-if)# interface po101 SW1(config-if)# sw trunk allowed vlan add 99 SW1(config-if)# interface po102 SW1(config-if)# sw trunk allowed vlan add 99 SW1(config-if)# exit SW1(config)#

SW2


SW3




Finally we can configure the authentication parameters of the switches to ensure the SXP protocol will run successfully on all switches.

The switches will need to be statically configured as either speaker or listener switches. Not at the same time they should be doing both to each other.

SW1

SW1(config)# feature cts SW1(config)# cts role-based enforcement SW1(config)# cts sxp enable SW1(config)# cts sxp connection peer 198.18.99.2 password required SW2p@ssw0rd mode listener SW1(config)# cts sxp connection peer 198.18.99.3 password required P@ssw0rdSW3 mode listener

SW2

SW2(config)# cts role-based enforcement SW2(config)# cts sxp enable SW2(config)# cts sxp connection peer 198.18.99.1 password required SW1p@ssw0rd mode speaker SW2(config)# cts sxp connection peer 198.18.99.3 password required P@ssw0rdSW3 mode speaker

Don’t forget to configure the speaker parameter on this switch.

SW3

SW3(config)# cts role-based enforcement SW3(config)# cts sxp enable SW3(config)# cts sxp connection peer 198.18.99.1 password required SW1p@ssw0rd mode speaker SW3(config)# cts sxp connection peer 198.18.99.2 password required SW2p@ssw0rd mode listener

With this final configuration we completed the task and finished the chapter!

We prepared some configuration for the next chapter, so leave all configuration in place while moving on to the next chapter about management features!



Chapter 9: Management Features

Chapter 9: Management Features is intended to let you be familiar with the Management features which are available on the Nexus platform. You will be configuring Role Based Access Control (RBAC), SNMP, Syslog, NetFlow, NTP and many more.

We highly recommend to create your own diagram at the beginning of each lab so you are able to draw on your own diagram, making it much easier when you step into the real lab. Multiple topology drawings are available for this chapter.



Solutions

Task 1: Role Based Access Control (RBAC)

This chapter is about configuring management features of the NX-OS platform. The management features are equal to those that are available on the MDS switches as it’s running the same code internally.

One of those features is also a bit related to security and that is Role Based Access Control (RBAC).

The model of RBAC is completely different from the Privilege level style like was used in Classical IOS. In IOS users were connected to a privilege level, which is a number between 0 and 15, on which privileges are configured. Those privileges are configured in the form of which commands are allowed to be executed by that user.

The possibilities with this type of configuration is limited and this is changed in NX-OS. Now roles are configured instead of privilege levels. Inside of the role configuration it’s still possible to configure commands that are allowed to be executed, but they offer much more features.

One of the possibilities of role configurations is that you are able to allow features to be individually configured. This means you do not need to configure all the related commands in the configuration, but can allow a whole set of configuration commands with just a single line in the configuration by allowing a specific feature.

Per feature you are able to specify whether the user is allowed to only verify the working or also change the configuration.

This configuration can then even be applied only to interfaces and VRFs that are configured in the role. When a user has certain interfaces and/or VRFs specified, it’s not able to touch any configuration on other interfaces and in other VRFs.

The role configuration is specific to the VDC and never shared.

There are a number of default roles pre-configured on the NX-OS devices.



Role Privileges

network-admin Complete access to all commands and configuration items

network-operator Access to all show commands except show running-config, users can never change any configuration

vdc-admin Complete access to all commands and configuration items limited to a VDC

vdc-operator Access to all show commands except show running-config, users can never change any configuration limited to a VDC

It’s not possible to change any of the default roles.

When changes are required, you need to create new user roles for this use.

Users are not limited to a single user role, but can be a member of multiple roles. In this case it becomes very flexible what the user is allowed to do.

When features are overlapping in a role configuration or in a configuration where a user is connected to multiple roles. The feature or role that is applied later, so later in the chain of configuration, that rule is applied.

This is different from the rule that usually the first or the most restrictive rule is applied.

Our configuration will begin with the configuration of certain users connected to certain roles. Pay close attention to the questioning. It’s highly advisable to read ahead and check all the questions related to a certain role or user configuration.

You might find limitations that would otherwise mean you have to re-do some configuration afterwards, which costs time.

In the first question it’s stated to configure a user with a specific role. When we try to configure the user up front, it becomes impossible to do this. We can only configure the default roles and the special privilege roles, because we enabled backwards compatibility with the IOS-Style privilege levels in the previous chapter.

SW1

SW1(config)# username user1 role ? network-admin System configured role network-operator System configured role priv-0 Privilege role priv-1 Privilege role priv-10 Privilege role priv-11 Privilege role priv-12 Privilege role priv-13 Privilege role priv-14 Privilege role priv-15 Privilege role priv-2 Privilege role



priv-3 Privilege role priv-4 Privilege role priv-5 Privilege role priv-6 Privilege role priv-7 Privilege role priv-8 Privilege role priv-9 Privilege role vdc-admin System configured role vdc-operator System configured role SW1(config)# username user1 role

This means we have to do our user configuration in a later stage of the configuration. We first need to configure the role itself and the limitations.

We do not need to enable a certain feature, like with all options in NX-OS. The RBAC feature is by default enabled and can not be disabled.

As the question states. We are not able to configure features directly under the role configuration. In NX-OS it’s possible to combine features together in so-called feature-groups. These groups can then be used multiple times in different role configurations to prevent overlapping configuration items.

SW1

SW1(config)# role feature-group ? name Feature-group name SW1(config)# role feature-group name ? WORD Enter feature-group name (Max Size 32) SW1(config)# role feature-group name USER1_FEATURES SW1(config-role-featuregrp)# ? feature Feature name no Negate a command or set its defaults end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in

The feature names can be difficult to get to know while the CLI will not auto-complete you in this case. It will tell you when a faulty feature is configured.

There is a list of features available through a show command which might make it a lot easier to configure the feature names, although they are easy to guess as well!

SW1(config-role-featuregrp)# show role feature aaa (AAA service related commands) arp (ARP protocol related commands) cdp (Cisco Discovery Protocol related commands) l3vm (Layer 3 virtualization related commands) ping (Network reachability test commands) snmp (SNMP related commands) radius (Radius configuration and show commands) syslog (Syslog related commands) tacacs (TACACS configuration and show commands)



install (Software install related commands) license (License related commands) callhome (Callhome configuration and show commands) platform (Platform configuration and show commands) access-list (IP access list related commands) svi (Interface VLAN related commands) hsrp (Hot Standby Router Protocol related commands) igmp (Internet Group Management Protocol related commands) msdp (Multicast Source Discovery Protocol related commands) vlan (Virtual LAN related commands) dot1x (DOT1X related commands) ipfib (IP Forwarding Information Base related commands) eth-span (Ethernet SPAN related commands) router-bgp (Border Gateway Protocol related commands) router-rip (Routing Information Protocol related commands) ethanalyzer (Ethernet Analyzer) router-ospf (Open Shortest Path First protocol related commands) router-eigrp (Enhanced Interior Gateway Routing Protocol related commands) spanning-tree (Spanning Tree protocol related commands) acl (FC ACL related commands) sfm (ISCSI flow related commands) sme (Storage Media Encryption feature related commands) fcns (Fibre Channel Name Server related commands) fcsp (Fibre Channel Security Protocol related commands) fdmi (FDMI related commands) fspf (Fabric Shortest Path First protocol related commands) rlir (Registered Link Incident Report related commands) rscn (Registered State Change Notification related commands) span (SPAN session relate commands) vsan (VSAN configuration and show commands) wwnm (WorldWide Name related commands) zone (Zone related commands) fcanalyzer (FC analyzer related commands) sme-kmc-admin (SME commands authorized to kmc admin) sme-stg-admin (SME commands authorized to storage admin) sme-recovery-officer(SME commands authorized to recovery officer) SW1(config-role-featuregrp)#

Now we know enough to perform the configuration for the role of user1.

SW1

SW1(config-role-featuregrp)# feature vlan SW1(config-role-featuregrp)# feature svi SW1(config-role-featuregrp)# feature spanning-tree SW1(config-role-featuregrp)# feature hsrp

We need to configure access to FHRP protocols. In this case this only means HSRP, as that’s the only feature that we can configure. When we try to configure the other FHRP’s we are not able to perform this configuration.

SW1(config-role-featuregrp)# feature vrrp Invalid role feature name vrrp SW1(config-role-featuregrp)# feature glbp Invalid role feature name glbp SW1(config-role-featuregrp)# exit



Then we can start configuring the role itself with the feature group.

SW1

SW1(config)# role ? feature-group Configure role feature-group name Enter the role name SW1(config)# role name USER1_ROLE SW1(config-role)# ? description Add a description for the role interface Configure the interface policy for this role no Negate a command or set its defaults rule Enter the rule number vlan Configure the vlan policy for this role vrf Configure the vrf policy for this role vsan Configure the vsan policy for this role end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-role)# rule ? <1-256> Enter the rule number SW1(config-role)# rule 1 ? deny Deny rule permit Permit rule SW1(config-role)# rule 1 permit ? command Command line read Read access read-write Read and write access SW1(config-role)# rule 1 permit read-write ? <CR> feature Feature name feature-group Feature group name SW1(config-role)# rule 1 permit read-write feature-group ? WORD Enter the access entity name (Max Size 32) SW1(config-role)# rule 1 permit read-write feature-group USER1_FEATURES

The configuration is done by creating rules in the role. There you are able to specify the read-only or read-write functionality and then the feature or feature-group that should get access when connected to a username.

Next up is that the user should be limited to only configure a few interfaces.

This is done by first denying access to all interfaces and later specifying which interfaces should have access.

SW1

SW1(config-role)# interface ? policy Configure the interface policy for this role SW1(config-role)# interface policy ?



deny Deny access to a interface unless specifically permitted SW1(config-role)# interface policy deny SW1(config-role-interface)# ? no Negate a command or set its defaults permit Permit access to interfaces (applicable if interface policy is 'deny') end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-role-interface)# permit ? interface Enter the range of interfaces accessible the role SW1(config-role-interface)# permit interface ethernet3/1-10

Just like with normal interfaces you are able to configure interface ranges under the configuration. Which is applied accordingly.

SW1(config-role-interface)# ? no Negate a command or set its defaults permit Permit access to interfaces (applicable if interface policy is 'deny') end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-role-interface)# exit SW1(config-role)# exit

Now finally after configuring the features, the role and the interface limitation we can configure the user that is going to use this role.

SW1

SW1(config)# username user1 role USER1_ROLE password User1p@ssw0rd SW1(config)#

Now we can verify the user and role configuration.

SW1(config)# show user-account user1 user:user1 this user account has no expiry date roles:USER1_ROLE SW1(config)# show role name USER1_ROLE Role: USER1_ROLE Description: new role vsan policy: permit (default) Vlan policy: permit (default) Interface policy: deny Permitted interfaces: Ethernet3/1-10 Vrf policy: permit (default) ------------------------------------------------------------------- Rule Perm Type Scope Entity ------------------------------------------------------------------- 1 permit read-write feature-group USER1_FEATURES



SW1(config)# show role feature-group name USER1_FEATURES feature group: USER1_FEATURES vlan (Virtual LAN related commands) svi (Interface VLAN related commands) spanning-tree (Spanning Tree protocol related commands) hsrp (Hot Standby Router Protocol related commands) SW1(config)#

The best test is of course to log-in to the switch with this new username and verify what you are able to configure now.

TerminalServer#ssh -l user1 172.16.100.2 Nexus 5000 Switch Password: User1p@ssw0rd Bad terminal type: "xterm-256color". Will assume vt100. Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php SW1# SW1# SW1# ? clear Reset functions configure Enter configuration mode debug Debugging functions show Show running system information end Go to exec mode exit Exit from command interpreter SW1# show ? spanning-tree Show spanning tree information vlan Vlan commands SW1# conf t Enter configuration commands, one per line. End with CNTL/Z. SW1(config)# ? spanning-tree Spanning Tree Subsystem vlan Vlan commands end Go to exec mode exit Exit from command interpreter SW1(config)#






As is shown in the above outputs. The options for viewing and configuring items is very limited. This limitation is enforced by the role configuration so our first questions are completed.

Next up is the configuration of another user with other limitations.

This user should again be limited to a number of features. Plus it’s only allowed to review settings and execute show commands.

We are not allowed to create another feature-group or configure the features for all the routing protocols. This means that we should find another creative solution.

When we verify the feature-group configuration. We see that there is a default feature-group available on the switch in the OS.

SW1(config)# show role feature-group ERROR: Feature "router-isis" does not exist feature group: L3 router-bgp (Border Gateway Protocol related commands) router-eigrp (Enhanced Interior Gateway Routing Protocol related commands) router-ospf (Open Shortest Path First protocol related commands) router-rip (Routing Information Protocol related commands) feature group: USER1_FEATURES vlan (Virtual LAN related commands) svi (Interface VLAN related commands) spanning-tree (Spanning Tree protocol related commands) hsrp (Hot Standby Router Protocol related commands) SW1(config)#

As we need to configure the routing protocols we can use this default feature group for the role configuration.

We configure the role for this next user.

SW1

SW1(config)# role name USER2_ROLE SW1(config-role)# rule 1 permit read read read-write SW1(config-role)# rule 1 permit read feature-group ? WORD Enter the access entity name (Max Size 32) SW1(config-role)# rule 1 permit read feature-group L3 SW1(config-role)# rule 2 permit read feature access-list SW1(config-role)# rule 3 permit read feature license

Just like with permitting only selected interfaces, we can also limit the configuration of certain VRFs or VSANs. In this case we should limit the VRFs that the user is allowed to configure.

SW1

SW1(config-role)# vrf ? policy Configure the vrf policy for this role SW1(config-role)# vrf pol deny



SW1(config-role-vrf)# permit vrf VPN1 SW1(config-role-vrf)# permit vrf VPN2 SW1(config-role-vrf)# permit vrf VPN3 SW1(config-role-vrf)# exit SW1(config-role)# exit SW1(config)# username user2 role USER2_ROLE password User2User2 SW1(config)#

After configuring the username with the correct role assignment we finished the second user’s role configuration.

The next user is only allowed to configure management protocols. This means that we should create a feature-group consisting of all management protocols that we can find in the overview and add those to the group together with the possibility to upgrade software of the device.

SW1(config-role-featuregrp)# show role feature aaa (AAA service related commands) arp (ARP protocol related commands) cdp (Cisco Discovery Protocol related commands) l3vm (Layer 3 virtualization related commands) ping (Network reachability test commands) snmp (SNMP related commands) radius (Radius configuration and show commands) syslog (Syslog related commands) tacacs (TACACS configuration and show commands) install (Software install related commands) license (License related commands) callhome (Callhome configuration and show commands) platform (Platform configuration and show commands) access-list (IP access list related commands) svi (Interface VLAN related commands) hsrp (Hot Standby Router Protocol related commands) igmp (Internet Group Management Protocol related commands) msdp (Multicast Source Discovery Protocol related commands) vlan (Virtual LAN related commands) dot1x (DOT1X related commands) ipfib (IP Forwarding Information Base related commands) eth-span (Ethernet SPAN related commands) router-bgp (Border Gateway Protocol related commands) router-rip (Routing Information Protocol related commands) ethanalyzer (Ethernet Analyzer) router-ospf (Open Shortest Path First protocol related commands) router-eigrp (Enhanced Interior Gateway Routing Protocol related commands) spanning-tree (Spanning Tree protocol related commands) acl (FC ACL related commands) sfm (ISCSI flow related commands) sme (Storage Media Encryption feature related commands) fcns (Fibre Channel Name Server related commands) fcsp (Fibre Channel Security Protocol related commands) fdmi (FDMI related commands) fspf (Fabric Shortest Path First protocol related commands) rlir (Registered Link Incident Report related commands) rscn (Registered State Change Notification related commands) span (SPAN session relate commands) vsan (VSAN configuration and show commands) wwnm (WorldWide Name related commands) zone (Zone related commands) fcanalyzer (FC analyzer related commands) sme-kmc-admin (SME commands authorized to kmc admin)



sme-stg-admin (SME commands authorized to storage admin) sme-recovery-officer(SME commands authorized to recovery officer) SW1(config-role-featuregrp)#

The marked lines show all the management protocols that we can allow access to.

SW1

SW1(config)# role feature-group name MGMT_PROTOCOLS SW1(config-role-featuregrp)# feature aaa SW1(config-role-featuregrp)# feature radius SW1(config-role-featuregrp)# feature tacacs SW1(config-role-featuregrp)# feature cdp SW1(config-role-featuregrp)# feature snmp SW1(config-role-featuregrp)# feature syslog SW1(config-role-featuregrp)# feature callhome SW1(config-role-featuregrp)#

Next up we configure the role and we add the final step, which is to allow software upgrades. We could’ve also configured this together with the feature-group, but here we demonstrate that we can both use a feature-group together with normal features in all types of configuration.

SW1

SW1(config-role-featuregrp)# role name MAINTENANCE_ROLE SW1(config-role)# rule 1 permit read-write feature-group MGMT_PROTOCOLS SW1(config-role)# rule 2 permit read-write feature install SW1(config-role)# exit SW1(config)# username maintenance role MAINTENANCE_ROLE password MainTenanc3 SW1(config)#

Finally we create the username and we are done for this user. The installation of software should of course be done with the install command, which is the only command the user has access to.

The next user should only be allowed to configure Fibre Channel storage related features.

Again we check out the list of features and verify which should be added to a feature-group.

SW1(config-role-featuregrp)# show role feature aaa (AAA service related commands) arp (ARP protocol related commands) cdp (Cisco Discovery Protocol related commands) l3vm (Layer 3 virtualization related commands) ping (Network reachability test commands) snmp (SNMP related commands) radius (Radius configuration and show commands) syslog (Syslog related commands) tacacs (TACACS configuration and show commands) install (Software install related commands) license (License related commands) callhome (Callhome configuration and show commands) platform (Platform configuration and show commands) access-list (IP access list related commands) svi (Interface VLAN related commands) hsrp (Hot Standby Router Protocol related commands)



igmp (Internet Group Management Protocol related commands) msdp (Multicast Source Discovery Protocol related commands) vlan (Virtual LAN related commands) dot1x (DOT1X related commands) ipfib (IP Forwarding Information Base related commands) eth-span (Ethernet SPAN related commands) router-bgp (Border Gateway Protocol related commands) router-rip (Routing Information Protocol related commands) ethanalyzer (Ethernet Analyzer) router-ospf (Open Shortest Path First protocol related commands) router-eigrp (Enhanced Interior Gateway Routing Protocol related commands) spanning-tree (Spanning Tree protocol related commands) acl (FC ACL related commands) sfm (ISCSI flow related commands) sme (Storage Media Encryption feature related commands) fcns (Fibre Channel Name Server related commands) fcsp (Fibre Channel Security Protocol related commands) fdmi (FDMI related commands) fspf (Fabric Shortest Path First protocol related commands) rlir (Registered Link Incident Report related commands) rscn (Registered State Change Notification related commands) span (SPAN session relate commands) vsan (VSAN configuration and show commands) wwnm (WorldWide Name related commands) zone (Zone related commands) fcanalyzer (FC analyzer related commands) sme-kmc-admin (SME commands authorized to kmc admin) sme-stg-admin (SME commands authorized to storage admin) sme-recovery-officer(SME commands authorized to recovery officer) SW1(config-role-featuregrp)#

The marked lines show the storage related features that should be given access to. The final 3 features are related to a special storage feature, where dedicated users can only be given access to these commands. They are not in scope of this configuration and are not necessary to be configured.

SW1

SW1(config)# role feature-group name STORAGE SW1(config-role-featuregrp)# feature sfm SW1(config-role-featuregrp)# feature sme SW1(config-role-featuregrp)# feature fcns SW1(config-role-featuregrp)# feature fcsp SW1(config-role-featuregrp)# feature fdmi SW1(config-role-featuregrp)# feature fspf SW1(config-role-featuregrp)# feature rlir SW1(config-role-featuregrp)# feature rscn SW1(config-role-featuregrp)# feature vsan SW1(config-role-featuregrp)# feature wwnm SW1(config-role-featuregrp)# feature zone SW1(config-role-featuregrp)# feature fcanalyzer SW1(config-role-featuregrp)# SW1(config-role-featuregrp)# exit SW1(config)# SW1(config)# SW1(config)# role name STORAGE_ROLE SW1(config-role)# rule 1 permit read-write feature-group STORAGE SW1(config-role)# exit



After configuring the role and the feature-group. We complete the configuration by creating the username attached to the role.

SW1

SW1(config)# username storage-admin role STORAGE_ROLE password st0rage-@Dmin SW1(config)#

The final role that should be configured is a NOC user. This user can have all rights like the one of the ‘network-operator’, but is also allowed to issue Telnet and SSH commands from the box.

Here we need to use command authorization and wildcards!

SW1

SW1(config)# role name NOC SW1(config-role)# rule 1 permit command show SW1(config-role)# rule 2 permit command telnet * SW1(config-role)# rule 3 permit command ssh * SW1(config-role)# exit SW1(config)# username nocuser password NOCus3r role NOC SW1(config)#

Now we verify if the user indeed has the correct rights.

TerminalServer#ssh -l nocuser 172.16.100.2 Nexus 5000 Switch Password: NOCus3r Bad terminal type: "xterm-256color". Will assume vt100. Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php SW1# conf t Enter configuration commands, one per line. End with CNTL/Z.

We are allowed to ensure configuration mode, but we cannot change anything in here.

SW1(config)# ? no Negate a command or set its defaults username Configure user information. end Go to exec mode exit Exit from command interpreter SW1(config)# exit SW1# show ? aaa Show aaa information access-lists List access lists accounting Show accounting configuration banner Show current motd banner message






boot Show Bootvar Variables callhome Show callhome information cdp Show Cisco Discovery Protocol information cfs CFS Show Command handler checkpoint Show configuration rollback checkpoints cimserver Show cimserver configuration class-map Show class maps cli Show CLI information clock Display current Date

All show commands are available!

SW1# telnet ? A.B.C.D Enter a valid IPv4 address WORD Enter hostname (Max Size 64) SW1# telnet SW1# ssh ? WORD Enter hostname or user@hostname (Max Size 64) SW1# ssh

And the user is able to access Telnet and SSH features.

The final question of this task is to ensure that all switches share the same role configuration. The big advantage here is that RBAC supports distribution through CFS and therefore the configuration is easily done by enabling CFS distribution and ensuring all role configuration is available across all switches.

Pay attention as the distribute command is not automatically completed by the CLI, but it does exist and is supported.

SW1

SW1(config)# role distribute

SW2


SW3


The final step is to distribute all roles of SW1 to the switches by issuing a commit.

SW1

SW1(config)# role commit 2012 Nov 20 21:18:55 SW1 %CFS-2-MTS_REJECT: Verification failed reject MTS message SAP 37:RR-token 0x3fe696b Please check Status with show command SW1(config)# SW1(config)# show role status



Distribution: Enabled Session State: Locking SW1(config)#

Now we finished the first task of this management chapter!

Task 2: Traffic monitoring

This task is about configuring monitoring sessions or SPAN sessions. Monitoring sessions are configurations for monitoring traffic on certain interfaces or VLANs.

This is used for troubleshooting purposes and for interception purposes. Traffic is fully copied from one interface to another.

These sessions are called SPAN sessions. You can monitor traffic from the following sources:

Ethernet interfaces

Port Channel interfaces

Control-Plane interface to the forwarding plane

VLANs

Satellite and port-channel interfaces on Fabric Extenders

The traffic is then copied to a destination interface. Multiple sources can be sent to the same destination interface. Be aware that it’s possible to oversubscribe the destination interface with configuring multiple sources.

There are ways to prevent that destination interfaces are oversubscribed, by configuring rate limiting or truncation of packets.

All Nexus switches support up to 16 SPAN sessions to be configured. Only 2 can be active simultaneous.

Virtual SPAN sessions are sessions that do not monitor a full interface, but only specific VLANs on this physical interface.

Our configuration starts with configuring these Virtual SPAN sessions on the port-channels connecting to SW2 and SW3.

Let’s start with configuring the switch.

SW1

SW1(config)# monitor session 1 SW1(config-monitor)# source interface port-channel101 SW1(config-monitor)# source interface port-channel102



First configure the sources which are the port-channels.

SW1(config-monitor)# filter vlan 50,99

Then we filter out the VLANs that we really need.

SW1(config-monitor)# destination interface ethernet3/19

The destination interface is specified. This can not be a port-channel member, only a physical interface

SW1(config-monitor)# no shut

By default are SPAN sessions in a shutdown state, so they need to be enabled.

SW1(config-monitor)# int e3/19 SW1(config-if)# sw monitor SW1(config-if)# sw mode trunk SW1(config-if)# no shut SW1(config-if)#

Finally we should configure the destination interface to support the SPAN traffic, which it by default won’t.

We verify the working of the session and if we configured everything properly.

The interface will remain down as there is no cable connected, so the session will also show up as being down, which is not a problem for our task as it is the expected result.

SW1(config-if)# do sh int e3/19 Ethernet3/19 is down (SFP not inserted) Hardware: 10000 Ethernet, address: 0005.9b73.5752 (bia 0005.9b73.5752) MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA Port mode is trunk auto-duplex, 10 Gb/s Beacon is turned off Input flow-control is off, output flow-control is off Switchport monitor is on SW1(config-if)# show monitor session 1 session 1 --------------- type : local state : down source intf : rx : Po101,Po102 tx : Po101,Po102 both : Po101,Po102 source VLANs : rx : 50,99 source VSANs : rx : destination ports : Eth3/19



Legend: f = forwarding enabled, l = learning enabled SW1(config-if)#

The next question is related to changing the MTU size of the monitoring traffic. This is done to prevent the monitoring server for having too much traffic sent at too large MTU sizes.

The packet after the MTU size which is configured is deleted, so not the entire packet is sent to the destination interface.

SW1

SW1(config)# monitor session 1 SW1(config-monitor)# mtu 1100

Now the MTU size will always be the same and never larger, not dependent on the source packet.

The next question asks for another approach. The question asks for transportation of the traffic across the network.

This technology is called Remote SPAN or RSPAN. It’s been around for a long time and it means that traffic being monitored on one switch, is sent into a special dedicated VLAN which is then picked up at another switch where the destination interface is located.

This is exactly what we need for this next question.

The traffic is being monitored on another switch and is sent to SW1 where the destination interface is located.

SW1

SW1(config)# vlan 601 SW1(config-vlan)# remote-span

This is the trick for RSPAN sessions. The VLAN should be marked with the ‘remote-span’ keyword. Then the traffic is correctly picked up and received well.

SW1(config-vlan)# exit SW1(config)# monitor session 2 SW1(config-monitor)# source vlan 601 SW1(config-monitor)# destination interface ethernet3/20 SW1(config-monitor)# no shut

The configuration of the RSPAN session is equal to a normal session for the rest. Keep in mind to use another number for the session so you do not overlap with the previous question.

SW1(config-monitor)# int e3/20 SW1(config-if)# sw monitor SW1(config-if)# no shut SW1(config-if)#



Now the traffic is being picked up and sent to the destination interface.

The next question is related to a technology introduced in the Nexus platform. Instead of transporting traffic across a Layer 2 domain it’s now also possible to transport it across a layer 3 domain inside a GRE tunnel.

This technology is called ERSPAN. It supports transport across a routed Layer 3 network.

The Nexus 5000 switches only support Source based sessions where as the Nexus 7000 also supports destination sessions.

There is no specification on which IP addresses should be used for this session. So we will use any routed network between the switches. For example VLAN 99 which is configured in a previous chapter.

Then the configuration is pretty straight forward by just configuring the right IP addresses, interfaces and some special values related to the ERSPAN session.

SW2

SW2(config)# monitor session 3 type erspan-source

The type of session is very important. Otherwise the ERSPAN configuration parameters could not be configured.

SW2(config-monitor)# source interface ethernet1/17 SW2(config-monitor)# destination ip 198.18.99.1 SW2(config-monitor)# erspan-id 100

The ERSPAN ID is a unique identifier to ensure the ERSPAN session is not overlapped with another session, which could cause mistakes in traffic monitoring.

SW2(config-monitor)# vrf default SW2(config-monitor)# no shut

We should also configure the source address that the switch uses, as that is required for configuration on the destination switch.

SW2(config)# monitor erspan origin ip-address 198.18.99.2 SW2(config-erspan-src)# exit

Next up is the configuration of the destination.

SW1

SW1(config)# monitor session 3 type erspan-destination SW1(config-monitor)# source ip 198.18.99.2 SW1(config-monitor)# destination interface ethernet3/17 SW1(config-monitor)# erspan-id 100

The ERSPAN ID should match on both sides of the SPAN session.

SW1(config-monitor)# vrf default SW1(config-monitor)# no shut



SW1(config-monitor)# int e3/17 SW1(config-if)# sw monitor SW1(config-if)# no shut SW1(config-if)#

Then the destination interface should be marked as a SPAN destination port and we’re done!

The final tasks are related to tuning the ERSPAN configuration.

The first is about giving priority to ERSPAN traffic. This is done by giving the ERSPAN traffic a higher DSCP bit, making it a higher priority traffic in Quality of Service configurations.

This configuration is done on the source.

SW2

SW2(config)# monitor session 3 type erspan-source SW2(config-monitor)# ip dscp 11

Finally the granularity should be configured. This means in how much precision the traffic is monitored. Configuration changes for this are done on the source as well.

SW2

SW2(config)# monitor erspan granularity ns

Nanosecond granularity is the finest we can get.

With this configuration we finished the task about SPAN ports!

Task 3: NetFlow

In this task about NetFlow we will be configuring traffic monitoring based on the flow of packets. This information is then used to form statistics about the network usage.

NetFlow works in a way that after it’s been enabled it will look for various packet streams that have identical packet headers. Meaning they belong to a flow. This flow is a unidirectional stream of packets. The flows are temporarily saved in the NetFlow cache en is then exported to a Flow server where it can be used for analysis and accounting on network traffic.

This process can take up a lot of resources. Which is why it’s done in hardware on the Nexus platforms. NetFlow uses a sampling rate to capture traffic according to a configured rate. For example every 1000 packets one is picked out and analyzed. The higher the sampling rate is, the more precise the NetFlow information becomes, but also the required resources on the device are getting higher.



There are several versions of the NetFlow export protocol. The Nexus switches support version 5 and version 9. Where the highest version is recommended due to flexibility and support for IPv6 and layer 2 information.

NetFlow support is very specific to the platform and the hardware. Currently only the Nexus 7000 supports the NetFlow feature.

The feature works by configuring various profiles. These profiles are bundled together and applied to an interface. On this interface the specified information is then monitored.

We start by configuring the Nexus 7000 switch for the NetFlow record that the question states.

The initial task is about creating a flow record based on IP address information. Together with that the Flow ID and packets per second should be captured.

SW1

SW1(config)# feature netflow SW1(config)# flow record IPX SW1(config-flow-record)# ? collect Specify a non-key field description Provide a description for this Flow Record match Specify a key field no Negate a command or set its defaults end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-flow-record)# match ? datalink Datalink (Layer 2) attributes ip IP attributes ipv4 IPv4 attributes ipv6 IPv6 attributes transport Transport layer fields SW1(config-flow-record)# match ipv4 ? destination Destination Address source Source Address SW1(config-flow-record)# match ipv4 source ? address Address SW1(config-flow-record)# match ipv4 source address SW1(config-flow-record)# match ipv4 destination address

We first configure by which fields this record should be matched. Which are the IPv4 addresses like our question stated.

SW1

SW1(config-flow-record)# collect flow ? sampler Sampler



SW1(config-flow-record)# collect flow samp ? id Identifer for sampler used for the flow SW1(config-flow-record)# collect flow samp id

Then the NetFlow ID should be captured.

SW1

SW1(config-flow-record)# collect ? counter Counters to collect flow Flow identifying fields ip IP attributes routing Routing attributes timestamp Timestamp fields transport Transport layer fields SW1(config-flow-record)# collect counter ? bytes Total number of bytes packets Total number of packets SW1(config-flow-record)# collect counter packets ? <CR> long Long counter (64 bits) SW1(config-flow-record)# collect counter packets long

As well as the Packets per Second counter. Be aware that you need to use 64-bit counters as the question stated.

SW1(config-flow-record)#

In the next question it is asked that the information should be exported to a dedicated flow server. Read ahead on the next question, because you will need some crucial information to configure this export.

The last question of this task states that we should be able to export Layer 2 fields to the Flow Server.

This automatically means that we need to use NetFlow version 9 in the Exporter configuration.

SW1

SW1(config)# flow exporter IPX_EXPORT SW1(config-flow-exporter)# version ? 5 Version 5 Export 9 Version 9 Export SW1(config-flow-exporter)# version 9 SW1(config-flow-exporter-version-9)# ? no Negate a command or set its defaults option Version 9 Option Templates and Data template Version 9 Template end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name



where Shows the cli context you are in SW1(config-flow-exporter-version-9)# exit SW1(config-flow-exporter)# ? description Provide a description for this Flow Exporter destination Specify the destination address dscp Optional DSCP no Negate a command or set its defaults source Source Interface for this destination transport Transport Destination Port version Specify the export version end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-flow-exporter)# destination 172.16.100.109 ? <CR> use-vrf Optional VRF label SW1(config-flow-exporter)# destination 172.16.100.109 SW1(config-flow-exporter)# exit SW1(config)#

After configuring the version and the destination we finished configuring our exporter. Now the configuration hasn’t finished!

We continue with configuring the sampling rate of the packets.

SW1

SW1(config)# flow ? exporter Define a Flow Exporter monitor Define a Flow Monitor record Define a Flow Record timeout Define a Flow Timeout SW1(config)# sampler IPX_SAMPLE SW1(config-flow-sampler)# ? description Provide a description for this Flow Sampler mode Define the Sampler mode no Negate a command or set its defaults end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-flow-sampler)# mode ? <1-64> Number of samples per sampling - 64 max SW1(config-flow-sampler)# mode 5 ? out-of M out of N packets SW1(config-flow-sampler)# mode 5 out-of ? <1-8192> Number of packets in each sampling - 8192 max SW1(config-flow-sampler)# mode 5 out-of 150 SW1(config-flow-sampler)# exit SW1(config)#



Now we configured our sampling profile for 5 out of 150 packets to be sampled.

Finally we should combine all this information together so the port-channels, like stated in the first question, are monitoring flows and exporting them correctly.

SW1

SW1(config)# flow monitor IPX_MON SW1(config-flow-monitor)# ? description Provide a description for this Flow Monitor exporter Add an Exporter to use to export records no Negate a command or set its defaults record Specify Flow Record to use end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-flow-monitor)# exporter IPX_EXPORT SW1(config-flow-monitor)# record IPX SW1(config-flow-monitor)# exit SW1(config)#

Within a monitor configuration, all items are combined together (exporter and record).

SW1

SW1(config)# SW1(config)# int port-chan101 SW1(config-if)# ip flow monitor ? WORD Name of Flow Monitor (Max Size 63) SW1(config-if)# ip flow ? monitor Apply a Flow Monitor to this interface SW1(config-if)# ip flow monitor IPX_MON ? input Apply Flow Monitor on input traffic output Apply Flow Monitor on output traffic SW1(config-if)# ip flow monitor IPX_MON input ? <CR> sampler Optional Sampler to apply to this Flow Monitor

Finally the flow monitor should be configured under the interface where it’s necessary. Do not forget to apply the sampler which we just configured!

SW1

SW1(config-if)# ip flow monitor IPX_MON input sampler ? WORD Name of Sampler (Max Size 63) SW1(config-if)# ip flow monitor IPX_MON input sampler IPX_SAMPLE



There was no direction specified in which the flow monitor should be applied so we are going to configure it on both directions.

SW1

SW1(config-if)# ip flow monitor IPX_MON output sampler IPX_SAMPLE SW1(config-if)# int port-chan102 SW1(config-if)# ip flow monitor IPX_MON input sampler IPX_SAMPLE SW1(config-if)# ip flow monitor IPX_MON output sampler IPX_SAMPLE SW1(config-if)#

The final step is to apply the flow configuration to both directions and to both interfaces.

And after the application of the interfaces. We finished the configuration of NetFlow!

Of course there are many other combinations that can be configured, but for now we have configured all properties, which should be enough for you to understand all variants that are possible within the configuration.

Task 4: Management protocols

This next task will be about configuring all the various management protocols that are available on the Nexus platform. The management protocols that we are configuring are the following.

SNMP

Syslog

NTP

CDP

These are no surprises as these are the management protocols which have been around for a long time and are configured on many different platforms and software versions.

There is one important difference between IOS and NX-OS and that is that SNMP version 3 is by default enabled on the device. This is done, because the Data Center Network Manager (DCNM) uses SNMP v3 to communicate with the Nexus switches.

When you create a user-account the device automatically creates a SNMP version 3 user account, making it possible to log-in using SNMPv3 as well.

Still NX-OS is compatible with SNMPv2c, because the entire world is still using it to manage their networks.

We start by configuring the classical SNMP environment for our switch.

SW1

SW1(config)# snmp-server host ? A.B.C.D|A:B::C:D|WORD IPv4 or IPv6 address or DNS Name of SNMP notification host



SW1(config)# snmp-server host 172.16.100.110 ? WORD SNMP community string or SNMPv3 user name (Max Size 32) filter-vrf Filters notifications to the notification host receiver based on the configured VRF informs Send Inform messages to this host source-interface Source interface to be used for sending out SNMP notifications to this host traps Send Traps messages to this host use-vrf Configures SNMP to use the selected VRF to communicate with the host receiver version SNMP version to use for notification messages SW1(config)# snmp-server host 172.16.100.110 traps ? WORD SNMP community string or SNMPv3 user name (Max Size 32) version SNMP version to use for notification messages

Ensure traps are sent to the host as the question stated.

SW1(config)# snmp-server host 172.16.100.110 traps version ? 1 Use SNMPv1 2c Use SNMPv2c 3 Use SNMPv3 SW1(config)# snmp-server host 172.16.100.110 traps version 2c ? WORD SNMP community string or SNMPv3 user name (Max Size 32)

We also need to configure the version at which the traps should be sent. In this case we will be using the Version 2c.

SW1

SW1(config)# snmp-server host 172.16.100.110 traps version 2c IPexpert SW1(config)# snmp-server enable traps ? <CR> aaa Module notifications enable bridge Enable SNMP STP Bridge MIB traps callhome Module notifications enable cfs Module notifications enable config Module notifications enable entity Module notifications enable fcdomain Module notifications enable fcns Module notifications enable fcs Module notifications enable fctrace Module notifications enable fdmi Module notifications enable feature-control Module notifications enable fspf Module notifications enable license Module notifications enable link Module notifications enable rf Module notifications enable rmon Module notifications enable rscn Module notifications enable scsi Module notifications enable snmp Module notifications enable stpx Enable SNMP STPX MIB traps sysmgr Module notifications enable upgrade Module notifications enable vsan Module notifications enable vtp Module notifications enable zone Module notifications enable



SW1(config)# snmp-server enable traps SW1(config)#

Before the switch will actually send traps. They need to be enabled first. By default the switch will not send any traps, so then the question would not be answered correctly.

Ensure that traps are enabled when is stated in the question that a server needs to receive traps. It’s not only the possibility, but it actually needs to receive them.

In this case we enable all the possible traps

The next question specifies that the server should also be able to access the switch via a specific community string.

SW1

SW1(config)# snmp-server community ? WORD SNMP community string (Max Size 32) SW1(config)# snmp-server community IPexpert ? <CR> group Group to which the community belongs ro Read-only access with this community string rw Read-write access with this community string use-acl Acl name to filter snmp requests SW1(config)# snmp-server community IPexpert ro

Read the question carefully as it clearly specified that the server should be able to ‘read’ information. So we can configure the read-only feature here.

The server is now able to access the switch to read out all information and it will receive traps.

The next questions are related to some contact information which is used in management systems to identify devices based on their location. It’s a free text format, so you can fill out anything you like.

SW1

SW1(config)# snmp-server location Utrecht, Netherlands, Europe SW1(config)# snmp-server ? aaa-user Set duration for which aaa-cached snmp user exists community Set community string and access privs contact Modify sysContact context SNMP context to be mapped enable Enable SNMP Traps globalEnforcePriv Globally enforce privacy for all the users host Specify hosts to receive SNMP notifications location Modify sysLocation mib Mib access parameters protocol Snmp protocol operations source-interface Source interface to be used for sending out SNMP notifications tcp-session Enable one time authentication for snmp over tcp session. user Define a user who can access the SNMP engine



SW1(config)# snmp-server con contact context SW1(config)# snmp-server contact Rick Mur, IPexpert SW1(config)#

As the next question states, we want the switch to enforce encryption for SNMPv3 traffic. One of the biggest complaints about SNMPv1 and SNMPv2c is that it has no support for any authentication other than the community strings and that it has no support for encryption. SNMPv3 offers all of that, but it’s not required by default.

With a single command we can turn the switch so it will only accept encrypted SNMPv3 requests.

SW1

SW1(config)# snmp-server ? aaa-user Set duration for which aaa-cached snmp user exists community Set community string and access privs contact Modify sysContact context SNMP context to be mapped enable Enable SNMP Traps globalEnforcePriv Globally enforce privacy for all the users host Specify hosts to receive SNMP notifications location Modify sysLocation mib Mib access parameters protocol Snmp protocol operations source-interface Source interface to be used for sending out SNMP notifications tcp-session Enable one time authentication for snmp over tcp session. user Define a user who can access the SNMP engine SW1(config)# snmp-server globalEnforcePriv SW1(config)#

We then should configure a user for SNMPv3 with rights. Now here is an overlap with the RBAC configuration as SNMPv3 uses roles from RBAC to give access to what the management device can access.

When you create a username, the switch will automatically create an SNMPv3 username with the same rights as well. You can individually delete the SNMPv3 usernames when you like, but it’s not possible to turn it off that the user is automatically created in the first place.

When an SNMPv3 user is created it also works vice versa. It will automatically create a user-account as well.

SW1

SW1(config)# snmp-server user ? WORD Name of the user (Max Size 32) SW1(config)# snmp-server user version3 ? <CR> WORD Group name (ignored for notif target user) (Max Size 32) auth Authentication parameters for the user SW1(config)# snmp-server user version3 auth ? md5 Use HMAC MD5 algorithm for authentication



sha Use HMAC SHA algorithm for authentication SW1(config)# snmp-server user version3 auth md5 ? WORD Authentication password for user (Max Size 130)

First we need to specify the password of the user.

SW1(config)# snmp-server user version3 auth md5 version3password ? <CR> engineID EngineID for configuring notif target user (for V3 informs) localizedkey Specifies whether the passwords are in localized key format priv Encryption parameters for the user SW1(config)# snmp-server user version3 auth md5 version3password priv ? WORD Privacy password for user (Max Size 130) aes-128 Use 128-bit AES algorithm for privacy SW1(config)# snmp-server user version3 auth md5 version3password priv version3password ? <CR> engineID EngineID for configuring notif target user (for V3 informs) localizedkey Specifies whether the passwords are in localized key format

The next step is to configure the encryption key that is used. There is no problem in using the same password for both keys. Which is also the default when a normal username is created.

SW1

SW1(config)# snmp-server user version3 auth md5 version3password priv version3password SW1(config)# snmp-server user version3 ? <CR> WORD Group name (ignored for notif target user) (Max Size 32) auth Authentication parameters for the user SW1(config)# snmp-server user version3 storage-admin

Finally the user should be given the same rights as a currently existing user role. Now at first it seems strange where to configure this, but the roles are called groups in this configuration.

You should use normal role names as the group name in this user configuration. After which it will take over the rights the user has and work as the question stated.

Now we finished the part for configuring SNMP related configurations.

We can verify if everything is configured at is should.

SW1

SW1(config)# show snmp ? <CR> > Redirect it to a file >> Redirect it to a file in append mode community Show snmp community strings context Show snmp context mapping entries engineID Show snmp engineID group Show snmp group host Show snmp hosts



internal Show snmp internal information sessions Show snmp sessions source-interface Show source-interface through which notifications are sent trap Show snmp traps user Show SNMPv3 users | Pipe command output to filter SW1(config)# show snmp user ______________________________________________________________ SNMP USERS [global privacy flag enabled] ______________________________________________________________ User Auth Priv(enforce) Groups ____ ____ _____________ ______ rick md5 des(no) network-admin admin md5 des(no) network-admin nocuser md5 des(no) network-operator NOC readonly md5 des(no) network-operator version3 md5 des(no) network-operator storage-admin maintenance md5 des(no) MAINTENANCE_ROLE storage-admin md5 des(no) STORAGE_ROLE ______________________________________________________________ NOTIFICATION TARGET USERS (configured for sending V3 Inform) ______________________________________________________________ User Auth Priv ____ ____ ____

Here it’s shown that all the users that have a regular user account are also configured as SNMPv3 users.

SW1

SW1(config)# show snmp host ------------------------------------------------------------------- Host Port Version Level Type SecName ------------------------------------------------------------------- 172.16.100.110 162 v2c noauth trap IPexpert

Here you can verify which servers receive the traps that the switch sends.

SW1(config)# show snmp community Community Group / Access context acl_filter --------- -------------- ------- ---------- IPexpert network-operator SW1(config)#



Finally you can verify which SNMP communities are configured on the switch and how they are operating.

The next management protocol which we will be configuring is the Syslog protocol. This protocol is used for troubleshooting and debug messages and not so much for alarming.

For alarming SNMP traps are used and Syslog is used mostly during troubleshooting processes.

There are several levels of message priority you are able to see and which you are able to process in separate log files when desired.

There are 7 levels of syslog priorities.

0: Emergency

1: Alert

2: Critical

3: Error

4: Warning

5: Notification

6: Informational

7: Debugging

All messages that the system can give have a priority attached. Usually up to level 4 is shown as errors of the system. Level 5 and level 6 are messages related to processes that perform actions. For example an OSPF neighbor comes online. When a neighbor goes down, it’s possibly an error so this get’s a higher priority.

Level 7 is only used for debug messages and is therefore never shown. In IOS they were always shown on the Console port, but this could cause serious issues as every console character had impact on the CPU. Therefore in NX-OS this has been disabled and you need to turn logging for debugging specifically on.

By default the following levels of logging are enabled. The logging monitor is the logging which is shown in Telnet and SSH sessions.

SW1

SW1(config)# show logging Logging console: enabled (Severity: critical) Logging monitor: enabled (Severity: notifications) Logging linecard: enabled (Severity: notifications) Logging fex: enabled (Severity: notifications) Logging timestamp: Seconds Logging server: disabled Logging logfile: enabled



Currently it’s not even supported to raise the logging level of the console, without having changed the console speed itself.

SW1(config)# logging console 6 Baud rate of console should be at least 38400 to increase logging level

The first Syslog question is so that Telnet and SSH sessions see informational level messages. This is done by changing the logging level of monitor sessions.

SW1

SW1(config)# logging monitor ? <CR> <0-7> 0-emerg;1-alert;2-crit;3-err;4-warn;5-notif;6-inform;7-debug SW1(config)# logging monitor 6

With the question mark it’s verifiable which level should be configured on the logging level.

SW1(config)# show logging Logging console: enabled (Severity: critical) Logging monitor: enabled (Severity: information) Logging linecard: enabled (Severity: notifications) Logging fex: enabled (Severity: notifications) Logging timestamp: Seconds Logging server: disabled Logging logfile: enabled

Now the informational messages are logged in Telnet and SSH sessions.

The next question states that debugging messages should be logged into a logfile.

SW1

SW1(config)# logging ? abort Flushes cached data without committing and releases the lock commit Commits cached data (of all msg types) and releases the lock console Set console logging distribute Enables/disables fabric distribution using cfs. event Interface events level Facility parameter for syslog messages logfile Set File logging message Interface events module Set module logging monitor Set terminal line(monitor) logging level server Enable forwarding to Remote Syslog Server timestamp Set logging timestamp granularity SW1(config)# logging logfile ? WORD Enter the logfile name SW1(config)# logging logfile DEBUG ? <0-7> 0-emerg;1-alert;2-crit;3-err;4-warn;5-notif;6-inform;7-debug SW1(config)# logging logfile DEBUG 7 SW1(config)#



We can configure logfiles in which we can enable separate severities to be logged.

SW1(config)# show logging Logging console: enabled (Severity: critical) Logging monitor: enabled (Severity: information) Logging linecard: enabled (Severity: notifications) Logging fex: enabled (Severity: notifications) Logging timestamp: Seconds Logging server: disabled Logging logfile: enabled Name - DEBUG: Severity - debugging Size - 4194304

Next we should be ensuring that the log files are marked with timestamps that are very precise.

SW1

SW1(config)# logging timestamp ? microseconds Timestamp in micro-seconds milliseconds Timestamp in milli-seconds seconds Timestamp in seconds (Default) SW1(config)# logging timestamp microseconds SW1(config)#

Microseconds is the most precise value we can use for our log messages. From now on the log messages are logged with timestamps including a second, millisecond and microsecond value.

Finally we should configure a Syslog server which should receive all notifications that are generated by the switch.

SW1

SW1(config)# logging ? abort Flushes cached data without committing and releases the lock commit Commits cached data (of all msg types) and releases the lock console Set console logging distribute Enables/disables fabric distribution using cfs. event Interface events level Facility parameter for syslog messages logfile Set File logging message Interface events module Set module logging monitor Set terminal line(monitor) logging level server Enable forwarding to Remote Syslog Server timestamp Set logging timestamp granularity SW1(config)# logging server 172.16.100.110 ? <CR> <0-7> 0-emerg;1-alert;2-crit;3-err;4-warn;5-notif;6-inform;7-debug

We should specify the correct level, which is 5 in this case where we want the notifications to be forwarded to the server.

SW1(config)# logging server 172.16.100.110 5 ? <CR> facility Facility to use when forwarding to server



use-vrf Enter VRF name, default is default VRF SW1(config)# logging server 172.16.100.110 5 facility ? auth Use auth facility authpriv Use authpriv facility cron Use Cron/at facility daemon Use daemon facility ftp Use file transfer system facility kernel Use kernel facility local0 Use local0 facility local1 Use local1 facility local2 Use local2 facility local3 Use local3 facility local4 Use local4 facility local5 Use local5 facility local6 Use local6 facility local7 Use local7 facility lpr Use lpr facility mail Use mail facility news Use USENET news facility syslog Use syslog facility user Use user facility uucp Use Unix-to-Unix copy system facility

The last step is configuring the correct facility that should be used. Facilities in Syslog are used for grouping messages together on the server itself and have no technical feature.

SW1

SW1(config)# logging server 172.16.100.110 5 facility local3 ? <CR> facility Facility to use when forwarding to server use-vrf Enter VRF name, default is default VRF SW1(config)# logging server 172.16.100.110 5 facility local3 SW1(config)#



This finalizes the syslog configuration.

Next up is the configuration of the Network Time Protocol (NTP). This protocol is widely used for distributing time information throughout a network. Many things rely on the configuration of the correct timing, it’s especially necessary to have synchronized timing on your network devices, because you require correct timestamps when you are troubleshooting a problem.

The NTP protocol is separated in servers, clients, peers and masters. The servers are authoritative over their connecting clients, but do not hold a master clock. That is only for the clocks configured as master. Peers are devices which are synchronizing towards each other and neither of them is really authoritative.

Not all switches can become master clocks. In the case of the Nexus family. Only the Nexus 7000 can become a master clock.

The trust worthiness of a clock is dependent on its ‘stratum’. This can be seen as a ‘hop-count’ as each time a server is in between, the stratum count increases of the clock. This means that the lower the number, the closer the clock is to the source.

The first 2 questions are creating a master, server and 2 clients. It means that SW1 should be the master clock for the network and SW2 and SW3 are the clients that synchronize their time against it.

SW1

SW1(config)# ntp ? abort Abort the ntp configuration authenticate Enable/Disable authentication authentication-key NTP authentication key commit Commit the ntp configuration distribute Enable NTP configuration distribution enable Enable NTP logging Enable/Disable logging of NTPD Events peer NTP Peer address server NTP server address source Source of NTP packets source-interface Source interface sending NTP packets trusted-key NTP trusted-key SW1(config)# ntp master 1

The master clock is now configured, so we can continue with the clients. We can use any Layer 3 address active on the switch for this use. So we will be using the VLAN 99 IP addresses.

SW2

SW2(config)# ntp server 198.18.99.1

SW3

SW3(config)# ntp server 198.18.99.1



After the configuration we can verify that the time is synchronizing against the master clock. Keep in mind that this can take a long time sometimes. During the CCIE Lab it’s even better to move on to the next task as you might need to wait up to 10 minutes before the NTP has been synchronized correctly.

SW2(config)# show ntp peer-status Total peers : 1 * - selected for sync, + - peer mode(active), - - peer mode(passive), = - polled in client mode remote local st poll reach delay ------------------------------------------------------------------------ =198.18.99.1 0.0.0.0 16 16 0 0.00000 SW1(config)#

We should now protect SW1 against unwanted NTP clients to be synchronizing time with us.

SW1

SW1(config)# ip access-list NTP_CLIENTS SW1(config-acl)# permit ? <0-255> A protocol number ahp Authentication header protocol eigrp Cisco's EIGRP routing protocol esp Encapsulation security payload gre Cisco's GRE tunneling icmp Internet Control Message Protocol igmp Internet Group Management Protocol ip Any IP protocol nos KA9Q NOS compatible IP over IP tunneling ospf OSPF routing protocol pcp Payload compression protocol pim Protocol independent multicast tcp Transmission Control Protocol udp User Datagram Protocol SW1(config-acl)# permit ip host 198.18.99.2 any SW1(config-acl)# permit ip host 198.18.99.3 any SW1(config-acl)# exit SW1(config)# ntp access-group serve NTP_CLIENTS

The serve keyword enables the NTP server to filter out requests from various connecting NTP systems. When serve is used, it will only accept requests and will never synchronize its own time against the specified IP addresses in the ACL.

To protect the NTP process even more, the requests should be authenticated!

SW1

SW1(config)# ntp authentication-key 1 md5 TimeIPX SW1(config)# ntp trusted-key 1 SW1(config)# ntp authenticate



We first configure the master clock for authentication. Do not forget to enable the authentication process itself, otherwise NTP will continue without authenticating anything.

The clients should be configured a little different, because they should authenticate against the server.

SW2

SW2(config)# ntp authentication-key 1 md5 TimeIPX SW2(config)# ntp server 198.18.99.1 key 1 SW2(config)# ntp authenticate

SW3

SW3(config)# ntp authentication-key 1 md5 TimeIPX SW3(config)# ntp server 198.18.99.1 key 1 SW3(config)# ntp authenticate

Here we finished the configuration of NTP.

The last time related question is that we should set the timezone to our current location. This is very important when you are using Time of Day time synchronization like NTP.

SW1

SW1(config)# clock timezone CET 1 0 SW1(config)#

SW2


SW3


You first configure a name which identifies the timezone that you are in. Then you specify the time offset in hours and minutes that is different from GMT time.



The final questions in this management task are related to a protocol proprietary to Cisco. This protocol is known as the Cisco Discovery Protocol.

It is used to exchange information about the Cisco devices which are connected to the interfaces. It’s extremely useful in your network environment as you immediately know what is connected behind an interface and also at which interface on that other device. Additionally you can even see what kind of device it is and when possible the management IP address to connect to.

It’s also extremely dangerous when used in environments where a lot of third parties connect, like Service Provider environments or Multi-Tenant Data Centers.

We should first set SW1 to identify itself via its serial number compared to its hostname which is the default.

SW1

SW1(config)# cdp ? advertise Highest CDP version supported on the switch enable Enable/disable CDP on all interfaces format Device ID format for CDP holdtime CDP hold time advertised (in seconds) timer CDP refresh time interval (in seconds) SW1(config)# cdp format? format Device ID format for CDP SW1(config)# cdp format ? device-id Device ID format for CDP SW1(config)# cdp format device-id ? mac-address Mac-address of the Chassis serial-number Chassis Serial Number/OUI system-name System name/Fully Qualified Domain Name (Default) SW1(config)# cdp format device-id serial-number SW1(config)#

This is of course totally unusable in a production world, but in this case it’s what the question asks for.

Next we should configure that switches should not use the default 60 second advertisements of CDP packets, but should change this to 10 second intervals.

SW1

SW1(config)# cdp timer 10 SW1(config)#

SW2




SW3


And the last question is that we want to disable CDP to be enabled on certain interfaces that are connected to a third party.

SW2

SW2(config)# int e1/10-20 SW2(config-if-range)# no cdp ? enable Enable/disable CDP on the interface SW2(config-if-range)# no cdp enable SW2(config-if-range)#

SW3

SW3(config)# int e1/10-20 SW3(config-if-range)# no cdp enable SW3(config-if-range)#

If you want to disable CDP to run on the entire device you should globally disable it with the command ‘no cdp run’.

Herewith we finished the task about management protocols!

Task 5: Device management

The fifth task is not about management protocols, but about managing the device itself and that means how file systems are handled, how configuration can be backed up (archived) and other features related to management of the device and the command-line of NX-OS.

In the first task we are asked to store configuration of the switch so it can be re-used. Now this can be done in multiple ways. One of them is by just storing the configuration on the flash.

This is not what we want, because we are also required to let it be compared with the current (new) configuration.

NX-OS knows a special feature for this, called ‘rollback’. It’s not the same as in other operating systems like IOS-XR, where the configuration is edited in a text file and then applied to the device.

In the NX-OS we can create configuration checkpoints. In which the configuration is saved. Then you are able to compare the differences with a newer configuration and when necessary even perform a rollback to the previous configuration.

SW1

SW1# checkpoint ? <CR>



WORD Checkpoint name (Max Size 32) description Checkpoint description for the given checkpoint file Create configuration rollback checkpoint to file SW1# checkpoint CONFIG_BACKUP ? <CR> description Checkpoint description for the given checkpoint SW1# checkpoint CONFIG_BACKUP ....Done SW1#

After the checkpoint has been generated it’s possible to compare differences with the other configuration files.

SW1

SW1# show diff ? rollback-patch Show rollback patch between configuration files or checkpoints SW1# show diff rollback-patch ? checkpoint Use checkpoint as source configuration file Src Checkpoint file running-config Use running configuration as source startup-config Use startup configuration as source SW1# show diff rollback-patch checkpoint ? CONFIG_BACKUP Checkpoint name

The available rollbacks are shown which you can use for the comparing.

SW1# show diff rollback-patch checkpoint CONFIG_BACKUP ? checkpoint Use checkpoint as destination configuration file Dst Checkpoint file running-config Use running configuration as destination startup-config Use startup configuration as destination SW1# show diff rollback-patch checkpoint CONFIG_BACKUP startup-config ? <CR> > Redirect it to a file >> Redirect it to a file in append mode | Pipe command output to filter

Then you should specify to which configuration the compare should be done. In this case we are using the startup-config.

SW1# show diff rollback-patch checkpoint CONFIG_BACKUP startup-config Collecting Startup-Config #Generating Rollback Patch ..... !! no logging timestamp microseconds no logging server 172.16.100.110 5 use-vrf management facility local3 no monitor session 2 no monitor session 1 no clock timezone CET 1 0 interface Ethernet1/20



cdp enable exit interface Ethernet1/19 cdp enable exit interface Ethernet1/18 cdp enable exit interface Ethernet1/17 cdp enable exit interface Ethernet1/16 cdp enable exit interface Ethernet1/15 cdp enable exit interface Ethernet1/14 cdp enable exit interface Ethernet1/13 cdp enable exit interface Ethernet1/12 no switchport monitor cdp enable exit interface Ethernet1/11 no switchport monitor no switchport mode trunk cdp enable exit interface Ethernet1/10 cdp enable exit no interface port-channel1011 no interface port-channel110 no cdp format device-id serial-number no cdp timer 10 no vlan 601 no ntp trusted-key 1 no ntp authentication-key 1 md5 WeqjAPS 7 no ntp authenticate no ntp server 198.18.99.1 no snmp-server community IPexpert group network-operator no snmp-server host 172.16.100.110 traps version 2c IPexpert no snmp-server globalEnforcePriv no snmp-server location Utrecht, Netherlands, Europe no snmp-server contact Rick Mur, IPexpert no ip access-list NTP_CLIENTS no username version3 role storage-admin no username version3 no username readonly no role name storage-admin ! interface vfc1 switchport mode E no shutdown ! logging monitor SW1#



The configuration output that is shown then includes a patch that you can use as a configuration template. The configuration is made so when it is applied to the current configuration (the one you are comparing with). Then it would rollback to that configuration.

It’s a very useful feature before doing changes, to automatically create your rollback script. The script also ensures that commands are typed in the correct order, so you can easily copy and paste it to the device when there are problems with your change or your current configuration.

The next questions states that the configuration should be saved to a TFTP server on a weekly basis on Sunday night at 10PM or 22:00.

The file-name should include some variables like the hostname and the date and time.

For this feature we are going to use a simple copy command, but we need to schedule it.

NX-OS knows a scheduler feature in which you are able to schedule CLI commands and also use different variables to ensure the command executed includes things like the hostname or the date and time.

We configure the scheduler job first, containing the commands that need to be executed.

SW1

SW1(config)# feature scheduler SW1(config)# scheduler ? aaa-authentication Password for AAA authentication(of logged in user) job Define a job logfile Scheduler log file configuration schedule Define a schedule transport Configure transport related configuration SW1(config)# scheduler job ? name Specify a name for the job SW1(config)# scheduler job name SAVE_CONFIG SW1(config-job)# copy run tftp://172.16.100.103/config-$(SWITCHNAME)-$(TIMESTAMP).txt SW1(config-job)# exit

Within the scheduler job you have access to the full CLI. This CLI access gives you the possibility to change anything in the configuration or any show commands necessary.

Ensure you are using the correct variables in the job. These can be found in the Cisco Documentation. For now the hostname and timestamp are the only variables available in the scheduler, but new ones can be added in newer versions of software.

SW1

SW1(config)# scheduler schedule name CONFIG_BACKUP_SCHED SW1(config-schedule)# job name SAVE_CONFIG SW1(config-schedule)# ?



email-addr Add email addr to send output of jobs configured for this schedule job Assign a job to the schedule no Negate a command or set its defaults time Specify a schedule to run jobs end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-schedule)# time ? daily Specify a daily schedule monthly Specify a monthly schedule start Specify a future time schedule weekly Specify a weekly schedule SW1(config-schedule)# time weekly ? WORD Day-of-week {[[dow:]HH:]MM} dow = 1 (Sun) .. 7 (Sat) (Max Size 10) SW1(config-schedule)# time weekly Sun:22:00 SW1(config-schedule)# exit

After the creation of the job, a scheduler needs to be created. This finalizes the configuration and ensures that the selected job is run at the given time and date.

We can verify if the job was executed successfully and what results where. When errors occur during the execution of scheduler jobs. These are logged in a logfile.

It’s also possible to configure a username and password which is used for executing the jobs so the accounting records are filled as well.

By default the admin user is used as the user to execute the jobs.

SW1(config)# show scheduler job Job Name: SAVE_CONFIG --------------------- copy running-config tftp://172.16.100.103/config-$(SWITCHNAME)-$(TIMESTAMP).txt ============================================================================== N7K11-SW1(config)# show scheduler schedule Schedule Name : CONFIG_BACKUP_SCHED ----------------------------------------- User Name : admin Schedule Type : Run on every Sunday at 22 Hrs 0 Mins Last Execution Time : Yet to be executed ----------------------------------------------- Job Name Last Execution Status ----------------------------------------------- SAVE_CONFIG -NA- ============================================================================== SW1(config)#



Now the schedule job is scheduled and the configuration is backed up to a TFTP server every week.

The next question is related to a banner that should be shown to users when they log in. As with classical IOS. This is done using the Message Of The Day (MOTD) banner.

SW1

SW1# conf t Enter configuration commands, one per line. End with CNTL/Z. SW1(config)# banner ? motd Configure banner motd message SW1(config)# banner motd "IPexpert CCIE Data Center Lab" SW1(config)# end SW1# exit Connection to 172.16.100.1 closed. RetinaRick:~ rick$ ssh 172.16.100.1 IPexpert CCIE Data Center Lab Bad terminal type: "xterm-256color". Will assume vt100. Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php SW1#

After configuring the banner it is shown when users are logging in to the device.

The last questions are related to the archiving features of the NX-OS platform. You are able to save files and compress them to GZIP files. After which they can then be combined to TAR files.

This is especially good to do with tech-support files, as they are very lengthy and can generate megabytes of clear text.

In the first question we need to save the tech-support and create a zip file manually.

SW1# show tech-support > bootflash:techsupport.txt SW1# dir bootflash: 18623 Nov 23 11:24:39 2012 CONFIG_BACKUP_2 1256 Mar 18 16:33:53 2011 aaa_cnv.log 364 Mar 18 16:33:53 2011 assoc_mgr_cnv.log 505 Mar 21 17:26:12 2010 license_SSI140806X2_6.lic 49152 Mar 18 16:14:02 2011 lost+found/ 3785 Nov 07 20:01:14 2012 mts.log 21778944 May 18 13:50:04 2010 n5000-uk9-kickstart.4.2.1.N1.1.bin 25164288 Feb 21 11:36:01 2011 n5000-uk9-kickstart.5.0.2.N2.1.bin 25152512 Mar 18 15:59:51 2011 n5000-uk9-kickstart.5.0.3.N1.1a.bin 181095489 May 18 13:46:13 2010 n5000-uk9.4.2.1.N1.1.bin 156932426 Feb 21 11:37:53 2011 n5000-uk9.5.0.2.N2.1.bin 188758273 Mar 18 16:02:21 2011 n5000-uk9.5.0.3.N1.1a.bin 4939579 Nov 23 12:55:41 2012 techsupport.txt 4096 Jan 01 01:02:43 2005 vdc_2/






4096 Jan 01 01:02:43 2005 vdc_3/ 4096 Jan 01 01:02:43 2005 vdc_4/ Usage for bootflash://sup-local 666238976 bytes used 217112576 bytes free 883351552 bytes total

After the output of the tech-support is saved to a text file on the flash. We see that the size is almost 5 megabytes. Now we need to compress it.

SW1# gzip bootflash:techsupport.txt SW1# dir bootflash: 18623 Nov 23 11:24:39 2012 CONFIG_BACKUP_2 1256 Mar 18 16:33:53 2011 aaa_cnv.log 364 Mar 18 16:33:53 2011 assoc_mgr_cnv.log 505 Mar 21 17:26:12 2010 license_SSI140806X2_6.lic 49152 Mar 18 16:14:02 2011 lost+found/ 3785 Nov 07 20:01:14 2012 mts.log 21778944 May 18 13:50:04 2010 n5000-uk9-kickstart.4.2.1.N1.1.bin 25164288 Feb 21 11:36:01 2011 n5000-uk9-kickstart.5.0.2.N2.1.bin 25152512 Mar 18 15:59:51 2011 n5000-uk9-kickstart.5.0.3.N1.1a.bin 181095489 May 18 13:46:13 2010 n5000-uk9.4.2.1.N1.1.bin 156932426 Feb 21 11:37:53 2011 n5000-uk9.5.0.2.N2.1.bin 188758273 Mar 18 16:02:21 2011 n5000-uk9.5.0.3.N1.1a.bin 347146 Nov 23 12:55:41 2012 techsupport.txt.gz 4096 Jan 01 01:02:43 2005 vdc_2/ 4096 Jan 01 01:02:43 2005 vdc_3/ 4096 Jan 01 01:02:43 2005 vdc_4/ Usage for bootflash://sup-local 661639168 bytes used 221712384 bytes free 883351552 bytes total SW1#

After compression the file is only 347 kilobytes.

In this question we compressed the file manually. It’s also possible to create automatically compressed files from the output.

SW1

SW1# terminal ? alias Show aliases (if no arugments); create 'exec' aliases (not persistent). Persistent aliases are in config mode, see 'cli alias' color Enable colorization of prompt(green if last command ok, red if error), command line (blue), output (default color) dont-ask Don't ask 'are you sure' questions, take default answer instead edit-mode Set command line edition keys (vi or emacs; emacs is default) event-manager Event manager cli event history Configure terminal history properties length Set number of lines on a screen log-all Accounting log all commands including the show commands monitor Copy Syslog output to the current terminal line no Negate a command or set its defaults output How output of show commands should be formated



redirection-mode Set the redirection mode session-timeout Set session timeout terminal-type Set the terminal type tree-update Updates the main parse tree verify-only Verify command and do not execute width Set width of the display terminal SW1# terminal redirection-mode ? ascii Set the redirection mode to ASCII zipped Set the redirection mode to ZIPPED SW1# terminal redirection-mode zipped SW1# SW1# show interface > interface.txt

With the redirection-mode configuration (the > command) we can set how files should be saved. By default they are saved as standard ASCII files, but now we changed it to ZIP files.



While the file is still saved as .txt file. The output is unreadable as it is zipped.

SW1# dir 18623 Nov 23 11:24:39 2012 CONFIG_BACKUP_2 1256 Mar 18 16:33:53 2011 aaa_cnv.log 364 Mar 18 16:33:53 2011 assoc_mgr_cnv.log 3101 Nov 23 12:59:27 2012 interface.txt 505 Mar 21 17:26:12 2010 license_SSI140806X2_6.lic 49152 Mar 18 16:14:02 2011 lost+found/ 3785 Nov 07 20:01:14 2012 mts.log 21778944 May 18 13:50:04 2010 n5000-uk9-kickstart.4.2.1.N1.1.bin 25164288 Feb 21 11:36:01 2011 n5000-uk9-kickstart.5.0.2.N2.1.bin 25152512 Mar 18 15:59:51 2011 n5000-uk9-kickstart.5.0.3.N1.1a.bin 181095489 May 18 13:46:13 2010 n5000-uk9.4.2.1.N1.1.bin 156932426 Feb 21 11:37:53 2011 n5000-uk9.5.0.2.N2.1.bin 188758273 Mar 18 16:02:21 2011 n5000-uk9.5.0.3.N1.1a.bin 347146 Nov 23 12:55:41 2012 techsupport.txt.gz 4096 Jan 01 01:02:43 2005 vdc_2/ 4096 Jan 01 01:02:43 2005 vdc_3/ 4096 Jan 01 01:02:43 2005 vdc_4/ Usage for bootflash:// 661643264 bytes used 221708288 bytes free 883351552 bytes total SW1# tail interface.txt ???.SH??K?22@)?pLc??,c?fV????7h??#k?`?rX??!h?p?T҇?^hh탐????=????`>J?$4??5pτQ??q??ɬ?D ?C ߹9?Rn4W\?)4O*?cQ?He???`SV?M?{Ԑ)?|??Hv??!0wrd4-תJ? Ak??v4?Z?&??l:W??$?'E?L\_???Y???????C?K??¢?UJ?t?x?0?뙙??A?f?3?ӕr??sd?W??X|V?a??B??ێG??6??FZ???w?T ??x???+Y???? sj?8w?.?l??f????vٴHN}??n?I?? ? O| 2 x;v۶?w?Z?jyZ?7#ڱ#?i??4??y?! +?#?%?A?\?!]Pc;~?@ ???i{Zۻ?O`K?K?5?�|?>?`U??U ?;??Pdk:2?F6??ˑڄF?????>w ???R|l?}??|z?zK?\?h{K? &??zK4S?p? ?&??z??(i,????YI?(-a?F ?ҶoKKì&???-??b??????? ÿ?oh> F-????qΊ?}??{?;?lpOl,?f??????ν???ڌ? yi??&@??????Z?8?-???v?{?k?? ???߹?^?Fz?n`t?۟3Z???`??)?ջ ?\KΥT??Iо?T?ϥ`??B??Ń? ?1???WJ???????Z?ݝw??*\|x???ɔ????+???.}??~I/???/S?8?Ee?V?+??z??k#??+?ş???:?? ߹9v?v??-?Vc?&?fLӡ?[??m??J+??5?H?[u.?T?R3#?ٶ??; ?Q?j?̀??U[B1j?g? Z}?Zo???g?=%u,?j??s SW1# ?R?

As the output above shows the file is unreadable unless unzipped.

The final question states that both files that are generated should be combined in a TAR file.

SW1

SW1# tar ?



append Append some files to an existing archive create Create an archive (merge several files together) extract Extract files from archive (unmerge them) list Shows the list of files which are part of the archive SW1# tar create ? bootflash: The name of the archive (extension will be added if none of tar/tgz/tar.gz/tar.bz2/tbz2/tar.Z specified) volatile: The name of the archive (extension will be added if none of tar/tgz/tar.gz/tar.bz2/tbz2/tar.Z specified) SW1# tar create bootflash:combined ? absolute Don't strip leading `/'s from file names bootflash: Name of file to be added into archive bz2-compress Compress archive with bzip2 -> .tar.bz2 gz-compress Compress archive with gzip, the default -> .tar.gz remove Remove files after adding them to the archive uncompressed Dont compress archive -> .tar verbose Display files while merging/extracting volatile: Name of file to be added into archive SW1# tar create bootflash:combined bootflash:interface.txt bootflash:techsupport.txt.gz SW1#

We first specify the name of the archive and then specify which files should be included in the archive.

SW1# SW1# dir 18623 Nov 23 11:24:39 2012 CONFIG_BACKUP_2 1256 Mar 18 16:33:53 2011 aaa_cnv.log 364 Mar 18 16:33:53 2011 assoc_mgr_cnv.log 330925 Nov 23 13:05:32 2012 combined.tar.gz 3101 Nov 23 12:59:27 2012 interface.txt 505 Mar 21 17:26:12 2010 license_SSI140806X2_6.lic 49152 Mar 18 16:14:02 2011 lost+found/ 3785 Nov 07 20:01:14 2012 mts.log 21778944 May 18 13:50:04 2010 n5000-uk9-kickstart.4.2.1.N1.1.bin 25164288 Feb 21 11:36:01 2011 n5000-uk9-kickstart.5.0.2.N2.1.bin 25152512 Mar 18 15:59:51 2011 n5000-uk9-kickstart.5.0.3.N1.1a.bin 181095489 May 18 13:46:13 2010 n5000-uk9.4.2.1.N1.1.bin 156932426 Feb 21 11:37:53 2011 n5000-uk9.5.0.2.N2.1.bin 188758273 Mar 18 16:02:21 2011 n5000-uk9.5.0.3.N1.1a.bin 347146 Nov 23 12:55:41 2012 techsupport.txt.gz 4096 Jan 01 01:02:43 2005 vdc_2/ 4096 Jan 01 01:02:43 2005 vdc_3/ 4096 Jan 01 01:02:43 2005 vdc_4/ Usage for bootflash:// 661979136 bytes used 221372416 bytes free 883351552 bytes total SW1# tar list bootflash:combined.tar.gz interface.txt techsupport.txt.gz SW1#



We finally see the file is created and that the archive contains the correct files just like we wanted.

Just like the question specified, the TAR file should be compressed. Which is the default when not specifying anything about the archive in the CLI.

This finished the task about the system management!

Task 6: Smart Call Home and GOLD

The final task is about configuring the possibility of the device to report critical issues to an operator or even to Cisco TAC. This feature is called CallHome or Smart Call Home.

Smart Call Home offers the possibility to send critical information about the Nexus device to either an operator or on-call engineer. The other option is that the feature will analyze the information and file a request with the correct Cisco TAC team.

When your Nexus is not connected to an internet connection. So a connection to a mail server or to Cisco TAC is not possible. You are able to use a Transport Gateway (TG) aggregation point. This TG is connected to the internet and connected to the internal management network to ensure a secure connection between the Nexus device and Cisco TAC.

It’s possible to distribute Callhome configuration through CFS between all Nexus and MDS switches in the network.

Once a Call Home message is generated you are able to configure what kind of information should be shared with Cisco. This can come from default pre-sets of configurations or you can enter certain CLI commands to be entered to collect more information.

Then there is an option to send the messages in brief text messages, full text messages or XML messages. The XML messages are only used for filing TAC cases.

The pre-sets that are available for a default set of commands to be reported are called Alert-Groups. The following table explains which alert-groups execute which command to use in the alert.

Alert-Group Description Commands Cisco-TAC All critical alerts from the other

alert groups destined for Smart Call Home.

Execute commands based on the alert group that originates the alert.



Diagnostic Events generated by diagnostics.

show diagnostic result module all detail

show moduleshow version

show tech-support platform callhome

Supervisor hardware

Events related to supervisor modules.




Linecard hardware

Events related to standard or intelligent switching modules.




Configuration Periodic events related to configuration.

show version

show module

show running-config all

show startup-config

System Events generated by failure of a software system that is critical to unit operation.

show system redundancy status

show tech-support

Environmental Events related to power, fan, and environment-sensing elements such as temperature alarms.

show environment

show logging last 1000

show module show version


Inventory Inventory status that is provided whenever a unit is cold booted, or when linecards are inserted or removed. This alert is considered a noncritical event, and the information is used for status and entitlement.

show module

show version

show license usage

show inventory

show sprom all

show system uptime



Configuration of the Call-Home feature requires a number of components.

You first need to configure SNMP-Server Contact information. Without this configured CallHome will not work! This is also the only option which is not distributed with CallHome. You need to manually configure the SNMP server contact information before CallHome actually works.

We start our configuration with a boot-up configuration. The feature that is intended here is called GOLD. GOLD ensures that the right amount of diagnostics are collected. By default all notifications are enabled, so the only option to configure is in this case the amount of diagnostics that should be collected during the boot process of the switch.

SW1

SW1(config)# diagnostic ? bootup Configure Diagnostic for bootup SW1(config)# diagnostic bootup ? level Select diagnostic level SW1(config)# diagnostic bootup level ? bypass Skip all bootup test complete Complete level extra Extra level SW1(config)# diagnostic bootup level complete

SW2

SW2(config)# diagnostic bootup level complete

SW3

SW3(config)# diagnostic bootup level complete

The results can be made visible with a show diagnostic results command.

Next we can start by configuring the Call Home feature.

We do not need to enable the feature, because it’s always enabled.

SW1

SW1(config)# feature call ^ % Invalid command at '^' marker. SW1(config)# SW1(config)# SW1(config)# snmp-server contact Rick Mur, IPexpert



As explained, the SNMP contact (SysContact) information should be supplied, before CallHome will work.

SW1

SW1(config)# callhome SW1(config-callhome)# ? alert-group Alert group contract-id Service contract id of the customer customer-id Customer id destination-profile Configure destination profiles duplicate-message Configure throttling of duplicate callhome alert messages email-contact Email address of the contact person enable Enable callhome no Negate a command or set its defaults periodic-inventory Configure periodic software inventory message dispatch phone-contact Contact person's phone number site-id Site id of the network where switch is deployed streetaddress Configure replacement part shipping address. switch-priority Priority of the switch(0-highest 7-lowest) transport Configure transport related configuration end Go to exec mode exit Exit from command interpreter pop Pop mode from stack or restore from name push Push current mode to stack or save it under name where Shows the cli context you are in SW1(config-callhome)# alert-group ? Configuration Events related to Configuration Diagnostic Events related to Diagnostic Environmental Power,fan,temperature related events Inventory Inventory status events License Events related to licensing Linecard-Hardware Linecard related events Supervisor-Hardware Supervisor related events Syslog-group-port Events related to syslog messages filed by port manager System Software related events Test User generated test events

Alert-groups can be configured to be expanded with several CLI commands.

SW1

SW1(config-callhome)# destination-profile ? CiscoTAC-1 Configure destination profile for XML message WORD User defined destination profile name (Max Size 31) full-txt-destination Configure destination profile for plain txt message short-txt-destination Configure destination profile for short txt message

We are configuring a destination-profile, just like our questions state.

We should be configuring a profile that sends messages to an e-mail address and specifying all messages to be sent.

SW1

SW1(config-callhome)# destination-profile CRITICAL_MAILS ? <CR>



alert-group Add alert group email-addr Add email addr format Callhome message format (default XML) http Add http or https url message-level Callhome message level(0-lowest urgency, 9-highest urgency) message-size Configure maximum message size (default 2500000) transport-method Callhome message sending transport-method SW1(config-callhome)# destination-profile CRITICAL_MAILS email-addr ? WORD Provide email address, example: [email protected] (Max Size 255) SW1(config-callhome)# destination-profile CRITICAL_MAILS SW1(config-callhome)# destination-profile CRITICAL_MAILS email-addr [email protected] SW1(config-callhome)# destination-profile CRITICAL_MAILS ? <CR> alert-group Add alert group email-addr Add email addr format Callhome message format (default XML) http Add http or https url message-level Callhome message level(0-lowest urgency, 9-highest urgency) message-size Configure maximum message size (default 2500000) transport-method Callhome message sending transport-method SW1(config-callhome)# destination-profile CRITICAL_MAILS transport-method ? email Email transport-method http Http transport-method SW1(config-callhome)# destination-profile CRITICAL_MAILS transport-method email

Now the destination e-mail address and method are specified. Next up is configuring the source.

SW1

SW1(config-callhome)# email-contact ? WORD Provide email address, example: [email protected] (Max Size 255) SW1(config-callhome)# email-contact [email protected] ? SW1(config-callhome)# streetaddress Cisco Street 111

The street address should also be specified as that is used as a field in the Callhome message.

SW1

SW1(config-callhome)# transport ? email Configure email transport related configuration SW1(config-callhome)# transport email from reply-to smtp-server SW1(config-callhome)# transport email ? from Configure from email address reply-to Configure replyto email address smtp-server Configure SMTP server address SW1(config-callhome)# transport email from [email protected] SW1(config-callhome)# transport email from sm? WORD Provide from email address, example: [email protected] (Max Size 255) SW1(config-callhome)# transport email smtp-server ? A.B.C.D IPV4 address of SMTP server

mailto:[email protected]








A:B::C:D IPV6 address of SMTP server WORD SMTP server(DNS name or IPv4 or IPv6 address) SW1(config-callhome)# transport email smtp-server mail.ciscocallhome.com ? <CR> port Configure SMTP server port (default:25) use-vrf Configure vrf name SW1(config-callhome)# transport email smtp-server mail.ciscocallhome.com SW1(config-callhome)# exit

We configure the SMTP server and the source e-mail address that should be used in our configuration. Pay attention that you are only able to configure 1 source address and SMTP server, but multiple destination addresses. The destination address is configured under the destination-profile.

SW1

SW1(config)# ip name-server 172.16.100.111 Cannot write to the dns configuration -DNS is disabled SW1(config)# ip domain-lookup SW1(config)# ip name-server 172.16.100.111

Because we are configuring the name of the SMTP server in our configuration, we should use DNS name resolution to ensure that the IP address of the mail server is resolved.

Pay attention to enable name-resolution globally.

SW1

SW1(config)# callhome SW1(config-callhome)# SW1(config-callhome)# SW1(config-callhome)# destination-profile ? CiscoTAC-1 Configure destination profile for XML message WORD User defined destination profile name (Max Size 31) full-txt-destination Configure destination profile for plain txt message short-txt-destination Configure destination profile for short txt message SW1(config-callhome)# SW1(config-callhome)# SW1(config-callhome)# destination-profile CRITICAL_MAILS ? <CR> alert-group Add alert group email-addr Add email addr format Callhome message format (default XML) http Add http or https url message-level Callhome message level(0-lowest urgency, 9-highest urgency) message-size Configure maximum message size (default 2500000) transport-method Callhome message sending transport-method SW1(config-callhome)# destination-profile CRITICAL_MAILS mess message-level message-size SW1(config-callhome)# destination-profile CRITICAL_MAILS message-level ? <0-9> SW1(config-callhome)# destination-profile CRITICAL_MAILS message-level 0 SW1(config-callhome)# destination-profile CRITICAL_MAILS message-size ? <0-5000000> Provide maximum possible message size



SW1(config-callhome)# destination-profile CRITICAL_MAILS message-size 5000000 SW1(config-callhome)# destination-profile CRITICAL_MAILS format ? XML XML message format full-txt Plain text message format short-txt Short text message format SW1(config-callhome)# destination-profile CRITICAL_MAILS format full-txt SW1(config-callhome)#

Finally we should specify the maximum message size. We want to sent all messages of any urgency therefore we configured the level 0 urgency.

Next we should configure periodic inventory reports, which is another feature of CallHome.

SW1

SW1(config-callhome)# SW1(config-callhome)# periodic-inventory ? notification Enable periodic software inventory message dispatch SW1(config-callhome)# periodic-inventory notification ? <CR> interval Configure the time period for periodic inventory timeofday Configure the timeofday for periodic inventory in HH:MM format SW1(config-callhome)# periodic-inventory notification interval ? <1-30> Time period in days (default is 7 days) SW1(config-callhome)# periodic-inventory notification interval 1 SW1(config-callhome)# periodic-inventory ? notification Enable periodic software inventory message dispatch SW1(config-callhome)# periodic-inventory notification ? <CR> interval Configure the time period for periodic inventory timeofday Configure the timeofday for periodic inventory in HH:MM format SW1(config-callhome)# periodic-inventory notification SW1(config-callhome)#

The next question specifies that SW1 is a very important switch which should be noticed in the CallHome configuration. This is specified through the priority configuration.

SW1

SW1(config-callhome)# switch-priority ? <0-7> Priority of the switch(0-highest 7-lowest) SW1(config-callhome)# switch-priority 0 SW1(config-callhome)#



Finally we should ensure that Cisco TAC receives information, both via e-mail and HTTP requests.

This is configured with XML destination-profiles. However we can only configure one method per destination-profile. We can only configure one additional profile, so we need to use one predefined profile.

The pre-defined profiles are configured with a set of alert-groups, but also give you room for additional configuration, like the transport-methods.

We first configure the HTTP version using the pre-defined Cisco TAC profile.

SW1

SW1(config-callhome)# callhome SW1(config-callhome)# destination-profile ? CiscoTAC-1 Configure destination profile for XML message WORD User defined destination profile name (Max Size 31) full-txt-destination Configure destination profile for plain txt message short-txt-destination Configure destination profile for short txt message SW1(config-callhome)# destination-profile CiscoTAC-1 ? alert-group Add alert group email-addr Add email addr http Add http or https url transport-method Callhome message sending transport-method SW1(config-callhome)# destination-profile CiscoTAC-1 transport-method http SW1(config-callhome)#

Now we used the pre-defined TAC profile for sending the HTTP requests. As you can see the configuration is limited of this profile, because it uses a number of defaults.

Next we can finish the configuration of this chapter with configuring an additional destination-profile for sending e-mails.

SW1

SW1(config-callhome)# destination-profile CiscoTAC-2 transport-method email SW1(config-callhome)# destination-profile CiscoTAC-2 email-addr [email protected] SW1(config-callhome)# destination-profile CiscoTAC-2 ? <CR> alert-group Add alert group email-addr Add email addr format Callhome message format (default XML) http Add http or https url message-level Callhome message level(0-lowest urgency, 9-highest urgency) message-size Configure maximum message size (default 2500000) transport-method Callhome message sending transport-method SW1(config-callhome)# destination-profile CiscoTAC-2 format xml SW1(config-callhome)# destination-profile CiscoTAC-2 SW1(config-callhome)#




Now every destination-profile is correctly configured so the final task is to enable the callhome configuration.

SW1(config-callhome)# enable contact phone is not configured callhome cannot be enabled on the switch, because necessary configuration has not been done Please check if all of following configuration is done contact person name(sysContact) contact person's email contact person's phone number street addr To configure sysContact, please use snmp-server command SW1(config-callhome)# SW1(config-callhome)# SW1(config-callhome)# phone-contact +111333444555666 SW1(config-callhome)# SW1(config-callhome)# enable SW1(config-callhome)#

The enabling of Callhome will warn you when something has been forgotten. In this case we forgot to configure the phone number. After the phone number has been configured

Now we finished the configuration. Let’s verify if everything is working as expected.

SW1(config-callhome)# show run callhome !Command: show running-config callhome !Time: Sun Nov 25 17:45:52 2012 version 5.0(3)N1(1a) callhome switch-priority 0 email-contact [email protected] phone-contact +111333444555666 streetaddress Cisco Street 111 destination-profile CRITICAL_MAILS destination-profile CRITICAL_MAILS format full-txt destination-profile CiscoTAC-1 transport-method http destination-profile CiscoTAC-2 destination-profile CiscoTAC-2 format XML destination-profile CRITICAL_MAILS email-addr [email protected] destination-profile CiscoTAC-1 email-addr [email protected] destination-profile CiscoTAC-2 email-addr [email protected] transport email smtp-server mail.ciscocallhome.com port 25 transport email from [email protected] enable periodic-inventory notification interval 1 SW1(config-callhome)# show callhome callhome enabled Callhome Information: contact person name(sysContact):Rick Mur, IPexpert contact person's email:[email protected] contact person's phone number:+111333444555666 street addr:Cisco Street 111 site id: customer id: contract id: switch priority:0









duplicate message throttling : enabled periodic inventory : enabled periodic inventory time-period : 1 day periodic inventory timeofday : 08:00 (HH:MM) Distribution : Disabled SW1(config-callhome)# show callhome destination-profile full_txt destination profile information maximum message size:2500000 message format:full-txt message-level:0 transport-method:email email addresses configured: url addresses configured: alert groups configured: all short_txt destination profile information maximum message size:4000 message format:short-txt message-level:0 transport-method:email email addresses configured: url addresses configured: alert groups configured: all CiscoTAC-1 destination profile information maximum message size:5000000 message format:XML message-level:0 transport-method:email and http email addresses configured: [email protected] url addresses configured: alert groups configured: cisco-tac CiscoTAC-2 destination profile information maximum message size:2500000 message format:XML message-level:0 transport-method:email email addresses configured: [email protected] url addresses configured: alert groups configured: CRITICAL_MAILS destination profile information maximum message size:2500000 message format:full-txt message-level:0 transport-method:email email addresses configured: [email protected]






url addresses configured: alert groups configured: SW1(config-callhome)#

After verification we finished the task and we finished the chapter.

This chapter concluded the configuration chapters of NX-OS equipment. The next chapter will all be about configuring the Cisco UCS system. This has been divided in several chapters related to Networking, Storage and Compute.

ipexpert’s lab preparation detailed solution...

Documents