network visibility using advancedd2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/brkdct-1890.pdfnetwork...

Network Visibility using Advanced Analytics in Nexus Switches

Oliver Ziltener - Technical Marketing Engineer

BRKDCT-1890

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Session Abstract

• Session ID : BRKDCT – 1890

• Title : Network visibility using advanced Analytics in Nexus switches

• Abstract:• Learn how to get the most visibility from your Nexus-based network with new monitoring

capabilities and advanced enhancements to traditional features like SPAN, ERSPAN and NetFlow. We will delve deeper into platform specific features like ERSPAN, Virtual SPAN to leverage multi destination SPAN, SPAN filters, In-Band SPAN, Extended SPAN/ERSPAN, Rule based SPAN, SPAN with MTU truncation, SPAN rate limiting, Exception SPAN on Nexus 7000 & Microburst monitoring, latency monitoring, line rate SPAN, SPAN on drop, SPAN on latency, buffer usage histogram etc. in Nexus 5000/6000. NetFlow and its unique aspects on Nexus switches will be discussed as well. These features help you understand the network and the applications running on the network better, and quickly pinpoint trouble spots in the network. We will go over what each feature is capable of, what proper real world use cases are, proper configurations, how to interpret the outputs and use the data collected. This session will focus on analytics and monitoring. It will not focus on other management aspects like SNMP, Syslog, RMON etc.

BRKDCT-1890 3


Session Goal

• Create awareness of the Analytics and Monitoring tools available in the Nexus family (N3k, N5K, N6K, N7K and N9K) in NX-OS standalone mode

• Provide the ability to choose the right tool to analyze, which helps in timely resolution of the problem

• It will NOT focus on other management aspects like SNMP, Syslog, RMON, troubleshooting, QOS, architecture and packet flows

Reference Slide

BRKDCT-1890 4

• Introduction

• Quick Product Overview

• Advanced Visibility

• SPAN / ERSPAN

• Flexible NetFlow

• Conclusion

Agenda


Network Bandwidth Explosion

M2M

Trillions of new

“connected

events” will

occur over IP

networks

throughout the

next decade

Cloud

Global cloud

traffic will grow

6X by 2016

4G Mobile Adoption

4G will account

for 45% of

global mobile

data traffic

Global IP traffic

will grow 3X to

1.4 zettabytes

annually by

2017

IP Traffic

By 2017, the

world will reach

3 trillion Internet

video minutes

per month

Video

BRKDCT-1890 6


If not handled well....

• Degrading performance

• Difficulty to troubleshoot

• Improper planning of resources

BRKDCT-1890 7


What is Analytics?

The systematic

computational analysis

of data or statistics

statistics.

Discovery and

communication of

meaningful patterns in

data

Studying past historical

data to research potential

trends

BRKDCT-1890 8


Advanced Analytics on Nexus Switches

• Collection of various features and enhancements to the traditional monitoring tools

• Latency Monitoring, Buffer Monitoring, SPAN-on-drop, Exception SPAN, SPAN filters, Microburst Monitoring and a LOT MORE!

• Advantages: Microbursts, Congestion, find malicious source, filter SPAN packets etc...

BRKDCT-1890 9

• Introduction



• SPAN / ERSPAN


• Conclusion

Agenda


Nexus Switches Family

Nexus 5000

Nexus 7000

Nexus 3000

Nexus 2000

Nexus 3100

Nexus1000V

Nexus 9000Nexus 5600/6000

Nexus 2300Nexus 7700

Nexus 3500

BRKDCT-1890 11

• Introduction



• SPAN / ERSPAN


• Conclusion

Agenda

Latency Monitoring


• Many applications can get impacted because of high latency

• Website download

• Video streaming

• Video conferencing

• Online gaming

• Banking

• Airline reservation

• Stock Market

• Web hosting

Why do we need to correct latency problems?

BRKDCT-1890 14


How does Latency Monitoring work?

Packet

Packet

INGRESS TIMESTAMPING

EGRESS TIMESTAMPING

Latency Monitoring Feature measure: T2 – T1 in ns

Packet Time T1

Packet Time T2

BRKDCT-1890 15


• Latency Monitoring provides {min, average, max} latency between a specified port pair and also maintains latency histogram (accuracy in few nanoseconds)

• By default instantaneous Latency Monitoring is enabled between pair of ports

• Latency Histogram can be enabled for specific port-pair to provide histogram instead of instantaneous mode

• Measures switch latency for each packet, no sampling required

• Fully implemented in HW, no CPU impact, no traffic impact

How does Latency Monitoring work?

BRKDCT-1890 16


Modes of Latency monitoring

• Instantaneous - Enabled by default on all pairs of ports

• No configuration required

• The latency measured is after the packet enters Port ASIC (Bigsur)

NEXUS# show hardware profile latency monitor interface e1/7 interface ethernet 1/14

--------------------------------------------------------------------------------

Egress Port: Ethernet1/7 Ingress Port: Ethernet1/14 Mode: Inst

--------------------------------------------------------------------------------

| | Minimum | Maximum | Average |

--------------------------------------------------------------------------------

| cnt | 912| 936| 923|

--------------------------------------------------------------------------------

Egress Interface Ingress Interface

cnt denotes the latency of packets entering e1/14 and egressing e1/7

BRKDCT-1890 17


Modes of Latency Monitoring

• Custom histogram – Counts packets in defined range. Needs below configurations

NEXUS# show hardware profile latency monitor interface e1/3 interface e1/1

--------------------------------------------------------------------------------

Egress Port: Ethernet1/3 Ingress Port: Ethernet1/1 Mode: Custom Histogram

--------------------------------------------------------------------------------

| Range| 800 <= Latency < 10000| 800 > Latency >= 10000|

--------------------------------------------------------------------------------

| cnt | 3542903| 56792|

--------------------------------------------------------------------------------

NEXUS(config)# interface e1/3

NEXUS(config-if)# packet latency interface e1/1 mode custom low-latency 800 high-latency 10000

Ingress Interface

cnt denotes the number of packet in the specific range

Egress Interface Time in nano seconds

BRKDCT-1890 18

Microburst monitoring


• Spike of high activity

• Passes under the radar of traditional load-monitoring tools

• Traffic spike that causes that system to saturate

• How short and how high? – Capacity of worst system in N/W

Microburst – A Concern

BRKDCT-1890 20


Microburst in Reality

• UW-Madison & Microsoft Research Paper: “Understanding Data Center Traffic Characteristics” http://research.microsoft.com/pubs/136788/wren09.pdf

• Results: “we find only a small fraction of losses do not belong to any microburst. This indicates that, more often that not, when losses happen at the edge or aggregation links, they happen in bursts.”

BRKDCT-1890 21

http://research.microsoft.com/pubs/136788/wren09.pdf


Challenge: It’s Very Hard to see Microbursts

NEXUS# show interface ethernet 1/2

Ethernet1/2 is up

[…]

Last clearing of "show interface" counters 00:00:58

0 interface resets

30 seconds input rate 96315720 bits/sec, 1331 packets/sec

30 seconds output rate 0 bits/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 77.00 Mbps, 1.05 Kpps; output rate 0 bps, 0 pps

RX

200000 unicast packets 0 multicast packets 0 broadcast packets

200000 input packets 1800000000 bytes

200000 jumbo packets 0 storm suppression bytes

0 runts 0 giants 0 CRC 0 no buffer

[…]

BRKDCT-1890 22


Solution: Burst Monitoring

• Configure your own burst filter per port per direction

• This command essentially enables micro burst detection on a port

• This command defines the maximum number of bursts that should happen over a time window before firing an syslog

burst threshold {ingress | egress} {limit percent | size max_bytes} interval

interval_time

[no] burst maximum {ingress | egress} burst-count max-burst

BRKDCT-1890 23


Feature guideline

• Supported on physical ports, port-channel members, and FEX fabric ports

• Not supported on sub interfaces, FEX HIF ports and port-channels

BRKDCT-1890 24


Burst Monitoring CLI

• To monitor bursts

• Example:

• To clear counters:

clear burst-counters [interface {all | ethernet interface}] {both | egress |

ingress }

show interface [ethernet slot/port]] burst-counters

NEXUS# show interface e1/14 burst-counters

--------------------------------------------------------------------

| Interface | Ingress Bursts | Egress Bursts | Total Bursts |

--------------------------------------------------------------------

| Ethernet1/14| 2| 0| 2|

BRKDCT-1890 25


Real World Example

• Troubleshooting Methodology: Detect micro bursty traffic

• Enable Micro burst detection to provide syslog notification

interface Ethernet1/13

burst threshold ingress size 10000 interval 100

burst maximum ingress burst-count 100

burst threshold egress size 10000 interval 100

burst maximum egress burst-count 100

!

2016 Feb 8 12:10:05 NEXUS %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Micro

Burst has been detected on ingress side on Ethernet1/13 - bigsurusd

Time in micro seconds

Define how many bursts to

be detected, before send

syslog

Both commands are recommend per direction

limit: Threshold size as percentage of link speed

size: Threshold size in bytes

BRKDCT-1890 26

Buffer monitoring


Why do we need to monitor buffers?

• Is my network congested?

• Can I add a new server?

• Will the performance be impacted?

• Why are the drops happening?

BRKDCT-1890 28


What is Buffer monitoring on Nexus?

• Buffer utilization is on a per port basis

• Buffer utilization shows buffer for unicast traffic in ingress and unicast and multicast in egress directions

• Histogram mode – slow (1sec) or fast (250ms) sampling

BRKDCT-1890 29


• Supported on physical ports, port-channel members, and FEX fabric ports

• Not supported on sub interfaces, FEX HIF ports and port-channels

Feature Guideline

BRKDCT-1890 30


Configuration

• Buffer utilization must be enabled on interface

• Fast sampling must be enabled in global configuration mode

• Default sampling is slow = 1 second

NEXUS(config)# inter e1/10

NEXUS(config-if)# hardware profile buffer monitor

NEXUS(config-if)#

NEXUS(config)# hardware profile buffer monitor sampling fast

NEXUS(config)#

BRKDCT-1890 31


Configuration

• To see buffer utilization and/or the buffer utilization histogram*, the next command must be executed

• To clear buffer utilization history use

show hardware profile buffer monitor { interface <ifid> | all } history {

brief | detail }

clear hardware profile buffer monitor [ interface <ifid> ]

*History up to 1 hour

BRKDCT-1890 32


Output of Buffer Monitoring tool

NEXUS# show hardware profile buffer monitor interface ethernet 1/21 history brief

--------------------------------------------------------------------------------

Interface : Eth1/21

--------------------------------------------------------------------------------

--------------------------------------------------------------------------------

Sampling Mode : Slow (1 second)

--------------------------------------------------------------------------------

--------------------------------------------------------------------------------

Ingress Buffer Utilization Detected(in KB)

Per asic Ingress Total Usage (15.628800MB)

--------------------------------------------------------------------------------

1 sec | 5 sec | 1 min | 5 min | 1 hour |

--------------------------------------------------------------------------------

0.6| 0.6| 0.6| 0.6| 0.6|

--------------------------------------------------------------------------------

Egress Buffer Utilization Detected(Unicast|Multicast)(in KB)

Per asic Egress Total Usage (8.611850MB)

--------------------------------------------------------------------------------

1 sec | 5 sec | 1 min | 5 min | 1 hour |

--------------------------------------------------------------------------------

112.6| 0.0| 177.2| 0.0| 158.0| 0.0| 164.1| 0.0| 164.3| 0.0|

--------------------------------------------------------------------------------

BRKDCT-1890 33


Real World Example

• Slow Application Response – Port Oversubscription

• Interface and queueing statistics verification ingress discards due oversubscription egress port

• Check buffer utilization

• Determine the egress port that is congested using virtual output queue (VoQ) statistics

e1/25

10G

e1/5

10G

e1/4

10G

Destination

BRKDCT-1890 34


Real World Example

• We spot input discards on interfaces

Slow Application Response – Port Oversubscription

NEXUS#show inter e1/5

Ethernet1/5 is up

---snip---

RX





0 input error 0 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 57491175 input discard

0 Rx pause...


Ethernet1/25 is up

---snip---

RX








0 Rx pause...

BRKDCT-1890 35


Real World Example

• We spot ingress discards on interface (RX queuing)


NEXUS#show queuing interface e1/5

Ethernet1/5 queuing information:

TX Queuing

qos-group sched-type oper-bandwidth

0 WRR 100

RX Queuing

qos-group 0

q-size: 100160, HW MTU: 1500 (1500 configured)

drop-type: drop, xon: 0, xoff: 0

Statistics:

Pkts received over the port : 112068891

Ucast pkts sent to the cross-bar : 54577716

Mcast pkts sent to the cross-bar : 0

Ucast pkts received from the cross-bar : 0

Pkts sent to the port : 0

Pkts discarded on ingress : 57491175

Per-priority-pause status : Rx

(Inactive),Tx(Inactive)

Only default queue is used

(default) here

BRKDCT-1890 36


Real World Example

• Egress interface is ok



Ethernet1/4 is up

---snip---

TX


228498277 output packets 30161765824 bytes

0 jumbo packets

0 output error 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard

0 Tx pause

BRKDCT-1890 37


Real World Example

• What is about buffer utilization?


NEXUS# show hardware profile buffer monitor interface e1/5

+---------------------------------------------------------------------------+

| Instant Ingress Buffer utilization per class per port. Every line |

| displays the number of cells utilized for a given port for each class |

| One cell represents 320 bytes |

+---------------------------------------------------------------------------+

-----------------------------------------------------------------------------

Interface : Eth1/5

-----------------------------------------------------------------------------

Total Port Instant Usage 17744 (5.678080MB)

Remaining Asic Instant Usage 31096 (9.950720MB)

Per asic ingress cell count 48840 (15.628800MB)

+----------+-------+-------+--------+-------+-------+-------+-------+-------+

port| class0| class1| class2| class3| class4| class5| class6| class7|

+----------+-------+-------+--------+-------+-------+-------+-------+-------+

Eth1/5| 0| 0| 0| 17744| 0| 0| 0| 0|

+----------+-------+-------+--------+-------+-------+-------+-------+-------+

---snip---

Note:

Class0 is control

traffic

Class1 is internetwork

control traffic

Class2 is FCoE traffic

Class3 is QoS group

0 (default queue)

Class4-7 are QoS

group 2-5 sequentially

Real-time buffer/cell allocation

of buffer/cell at ingressBRKDCT-1890 38


Real World Example

• To get additional information about the oversubscribed port, we will need to look at the virtual output queue (VoQ) statistics for the ingress ASIC group


NEXUS# show platform fwm info pif e1/5 | grep global_asic_num

Eth1/5 pd: slot 0 logical port num 4 slot_asic_num 1 global_asic_num 1 fw_inst 4 phy_fw_inst 1 fc 0

NEXUS# show platform fwm info pif e1/25 | grep global_asic_num

Eth1/25 pd: slot 0 logical port num 24 slot_asic_num 3 global_asic_num 3 fw_inst 0 phy_fw_inst 0 fc 0

ASIC group is the

«global_asic_num»

BRKDCT-1890 39


Real World ExampleSlow Application Response – Port Oversubscription

NEXUS# show platform software qd info counters voq asic-num 1

+----------+------------------------------+------------------------+-----------+

| port| TRANSMIT| TAIL DROP| HEAD DROP|

+----------+------------------------------+------------------------+-----------+

Eth1/4

QUEUE-3 54577716 57491175 0

---snip--

+----------+------------------------------+------------------------+-----------+

NEXUS# show platform software qd info counters voq asic-num 3

+----------+------------------------------+------------------------+-----------+

| port| TRANSMIT| TAIL DROP| HEAD DROP|

+----------+------------------------------+------------------------+-----------+

Eth1/4

QUEUE-3 173917190 1457036 0

---snip---

+----------+------------------------------+------------------------+-----------+

• VoQ statistics indicates that QUEUE-3 of Eth1/4 is oversubscribed (tail drops)

QoS Group 0

Egress Interface

Note: Internal queue

numbers are mapped

as follows:

QUEUE-0 is control

traffic

QUEUE-1 is

internetwork control

traffic

QUEUE-2 is FCoE

traffic

QUEUE-3 is QoS

group 0 (default

queue)

QUEUE 4-7 are QoS

group 2-5 sequentially

BRKDCT-1890 40


Real World Example

• The same drops on the egress interface using the following command:

• The output indicates that ASIC1 and ASIC3 are dropping traffic destined to Eth1/4 as seen in the initial VoQ output


NEXUS# show platform software qd info counters voq interface e1/4

+----------+------------------------------+------------------------+-----------+

|slot asic| TRANSMIT| TAIL DROP| HEAD DROP|

+----------+------------------------------+------------------------+-----------+

---snip--

0 1

QUEUE-3 54577716 57491175 0

0 3

QUEUE-3 173917190 1457036 0

+----------+------------------------------+------------------------+-----------+

BRKDCT-1890 41

• Introduction



• SPAN / ERSPAN


• Conclusion

Agenda


Switch Port Analyzer (SPAN)

• A SPAN session is an association of source ports/vlans to one or more destination ports

• Once the traffic is identified for replication, switch copies the matching traffic to the SPAN destination port(s)

• The SPAN (copied) packets are created in hardware without overloading the CPU

SPAN Source

Host B

SPAN Destination

Sniffer Device

e1/1 e5/1

e2/1

SPAN all the packets

ingressing e1/1

Spanned (copied)

traffic

Host A

BRKDCT-1890 43


SPAN Sources

• Switchports

• Access ports

• Trunk ports

• Private VLAN ports

• Port-channels

• Routed interfaces

• Physical interfaces

• Port-channels

• VLANs and PVLANs

• Supervisor inband interface

• Up to 128 physical interfaces and/or up to 32 VLANs per session

• Mix of interface types allowed in single session

• For example, SPAN source of VLAN 10 and interface e1/1 in same session

• Individual subinterfaces cannot be SPAN source

BRKDCT-1890 44


Layer 3 Network

Encapsulated Remote SPAN (ERSPAN)

• ERSPAN supports source and destinations on different switches*

• It uses a GRE tunnel to carry traffic

• Packets replicated in hardware

ERSPAN Source

Sniffer Device

Packets are replicated and

GRE encapsulated at

ERSPAN source device

ERSPAN

Destination

At ERSPAN Destination

device, GRE packet is

decapsulated

*Not all HW supports ERSPAN destination, e.g. N9272 BRKDCT-1890 45


ERSPAN with IEEE1588 timestamp – Find Network Latency

ID

N5K-C56-72UP

STAT

2

5 6 7 8

1 3 4 10

13 14 15 16

9 11 12 18

21 22 23 24

17 19 201 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 4825 26 27 28 29 30 31 32

ID

N5K-C56-72UP

STAT

2

5 6 7 8

1 3 4 10

13 14 15 16

9 11 12 18

21 22 23 24

17 19 201 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 4825 26 27 28 29 30 31 32

PTP messages

Data

Switch A

Switch N

ERSPAN type III

Sniffer

Latency from Switch A

to Switch N = T2 – T1

GPS

PTP grandmaster

BRKDCT-1890 46


ERSPAN Type III – Packet Capture Example

• Timestamp information in the Type III header to be used to calculate the packet latency

monitor session 1 type erspan-source

header-type 3

erspan-id 1

vrf default

destination ip 104.104.104.21

source interface ethernet1/1 both

rate-limit auto

no shut

!

monitor erspan granularity 1588

ERSPAN Type III use a new GRE

Protocol Type 0x22EB

--------

ERSPAN II use 0x88BE

Direction (0xxx) and Granularity x10x = 1588*

Time Stamps

*This wireshark version does not decode properly BRKDCT-1890 47


ERSPAN Type III – Configuration Example (N7700)

hostname Node1

interface loopback0

ip address 1.1.1.1/32

!

monitor session 2 type erspan-source

header-type 3

erspan-id 1

vrf default

destination ip 3.3.3.3

source interface Ethernet1/3 both

rate-limit auto

no shut

!

! Admin VDC

monitor erspan origin ip-address 1.1.1.1 global

monitor erspan granularity 1588

hostname Node3

interface loopback0

ip address 3.3.3.3/32

!

monitor session 2 type erspan-destination

erspan-id 1

vrf default

source ip 3.3.3.3

destination interface Ethernet1/19

no shut

e1/3 e1/19

*only the relevant configuration is shown

Layer 3 Network

ERSPAN

BRKDCT-1890 48


Nexus 2000 (FEX) with Nexus Parent Switch

• Parent Switch support: please check latest release notes

++ x F

EX

Parent Switch Nexus 2000 Fabric Extenders

BRKDCT-1890 49


SPAN with FEX

• FEX Host ports can be SPAN source ports*

• FEX Fabric ports can be SPAN source ports with all parent switches

• FEX Host ports can be SPAN destination port with Nexus 5K**

Nexus Fabric Extender

*Except Nexus 7000 parent switch with F2/F2e Line card

Sniffer Device

Host A

Nexus Parent Switch

**Supported since NX-OS 7.2

BRKDCT-1890 50

Nexus 5600/6000 SPAN Features


Use Case - Packet Drops

• What packets are dropped?e1/5

10G


Ethernet1/5 is up

---snip---

RX








0 Rx pause...

BRKDCT-1890 52


SPAN-on-Drop

• SPAN-on-Drop allows SPAN’ning of the packets which were dropped due to unavailable buffer on ingress

Dedicated

SPAN Buffer

Ingress

Data Buffer

SP

AN

-on

-Dro

pTail-

Dro

p

Port 3 is

congested

Sniffer Device

N5600/N6000

BRKDCT-1890 53


SPAN-on-Drop Information

• Works for unicast packets only

• Supports both local SPAN and ERSPAN

• One SPAN-on-Drop session is supported

• Can have multiple source ports, and multiple destination ports

• Source port(s) can be a part of a SPAN-on-Drop session, and a local SPAN session simultaneously

• Note: SPAN-on-Drop is supported on N9K since 7.0(3)I4(1) on 2nd Generation N9K (N9200-X and N9200-Q/C). Hardware support N9300-EX, as Standalone NX-OS is not shipped yet.

BRKDCT-1890 54


SPAN-on-Drop Configuration

• The source interface is the ingress port for which we want to monitor drops

NEXUS(config)# monitor session 1 type span-on-drop

NEXUS(config-span-on-drop)# source interface e1/1 rx

NEXUS(config-span-on-drop)# source interface e1/2 rx

NEXUS(config-span-on-drop)# destination interface e1/4

NEXUS(config)# monitor session 2 type span-on-drop-erspan

NEXUS(config-span-on-drop-erspan)# source interface e1/1 rx

NEXUS(config-span-on-drop-erpsan)# source interface e1/2 rx

NEXUS(config-span-on-drop-erspan)# destination ip 100.1.1.2

Always Rx :

Ingress interface

– Packets

dropped at

ingress

BRKDCT-1890 55


SPAN-on-Drop Guidelines

• The source interfaces can only be Ethernet. They can be port-channel members, but port-channel as source is not supported

• Fabric extender (HIF) interfaces are not supported as sources; however, fabric (NIF) interfaces are supported. Setting a fabric interface as a source allows SPAN-on-Drop to be enabled on all fabric extender ports associated with that fabric interface.

• One SPAN-on-drop or SPAN-on-drop ERSPAN session can be active at a time

BRKDCT-1890 56


Use Case – Identify delayed flows

• Is a packet delayed?

e1/14

10G

e1/7

10G

BRKDCT-1890 57


SPAN-on-Latency

Sniffer Device

Data

Timestamp

Data

Port 3 is congested

Latency monitoring

If Latency Threshold

> 10 usec:

SPAN to 4

N5600/N6000

BRKDCT-1890 58


SPAN-on-Latency Information

• Replicated traffic uses the SPAN buffer so it doesn't impact the production traffic

• Supports both local SPAN and ERSPAN

• Latency threshold is per-port

• One SPAN-on-Latency session is supported in hardware

BRKDCT-1890 59


SPAN-on-Latency Configuration

• SPAN-on-Latency session makes a copy of all high-latency packets egressing on this port, coming from any ingress port

NEXUS(config)# monitor session 1 type span-on-latency

NEXUS(config-span-on-latency)# source interface Ethernet1/7 tx

NEXUS(config-span-on-latency)# destination interface Ethernet1/23


packet latency threshold 10000


switchport monitor

Always Tx:

packets

egressing on 1/7

(any source) with

latency >10us

will be replicated

to the SPAN

destination 1/23

BRKDCT-1890 60


SPAN-on-Latency Guideslines

• Support for one SPAN-on-latency session

• Multiple sources can be configured – latency threshold is per SPAN-on-drop TX source port

• A SPAN-on-Latency source port cannot be in another SPAN session

BRKDCT-1890 61


SPAN-on-Latency Guideslines

• Source port can be an regular Ethernet port, not a port-channel. Can be a port-channel member

• Source port cannot be FEX HIF port. But FEX fabric port is supported

• Destination is only a single Ethernet port, not port-channel

BRKDCT-1890 62


Real World Example

• Troubleshooting Methodology

• Verification of interface errors and Switch CPU

• Maybe congestion?

• Use Analytics Latency monitoring & Span-on-Latency

Slow Download Rate

e1/14

10Ge1/7

10G

BRKDCT-1890 63


Real World Example

• Instantaneous Latency Monitoring (no configuration required)

Slow Download Rate


--------------------------------------------------------------------------------


--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

| cnt | 912| 936| 923|

--------------------------------------------------------------------------------


--------------------------------------------------------------------------------


--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

| cnt | 904| 7526784| 4047543|

--------------------------------------------------------------------------------

When no

congestion

on e1/7

When heavy

congestion

on e1/7

BRKDCT-1890 64


Real World Example

• Optional: Configure Latency Monitoring

Slow Download Rate

NEXUS(config)#interface Ethernet1/7

NEXUS(config-if)# packet latency int e1/14 mode custom low-latency 800 high-latency 10000

NEXUS#show hardware profile latency monitor interface e1/7 interface e1/14

--------------------------------------------------------------------------------

Egress Port: Ethernet1/7 Ingress Port: Ethernet1/14 Mode: Custom Histogram

--------------------------------------------------------------------------------

| Range| 800 <= Latency < 10000| 800 > Latency >= 10000|

--------------------------------------------------------------------------------

| cnt | 203029| 8193520|

--------------------------------------------------------------------------------

Ingress

Interface

Egress

Interface

Out of the required

latency > 10us

BRKDCT-1890 65


Real World Example

• Find which application is impacted - SPAN-on-Latency

Slow Download Rate

NEXUS(config)# monitor session 1 type span-on-latency

NEXUS(config-span-on-latency)# source interface Ethernet1/7 tx

NEXUS(config-span-on-latency)# destination interface Ethernet1/23


packet latency threshold 10000


switchport monitor

Always Tx:

packets

egressing on 1/7

(any source) with

latency >10us

will be replicated

to the SPAN

destination 1/23

BRKDCT-1890 66


SPAN with ACL filter

• Selectively monitor traffic in a SPAN session using Access-Control-List (ACL) to avoid destination sniffer overload

• SPAN session ignores any permit/deny actions specified in the ACL

• SPANs packets that match (permit) the ACL filter criteria

NEXUS(config)# ip access-list ACL-IP-01

NEXUS(config-acl)# 10 permit ip host 192.168.111.11 host 192.168.112.12

NEXUS(config-acl)# end

NEXUS(config)# monitor session 1

NEXUS(config-monitor)# source interface ethernet 1/3

NEXUS(config-monitor)# destination interface ethernet 1/9

NEXUS(config-monitor)# filter access-group ACL-IP-01

NEXUS(config-monitor)# no shut

*Supported with SPAN local and ERSPAN BRKDCT-1890 67

Nexus 7000/9000 SPAN Features


SPAN VLAN Filters

• VLAN filters allow monitoring subset of VLANs on trunk ports

• Filter specifies list of VLANs to capture

• Traffic for other VLANs not sent to SPAN destination


NEXUS(config-monitor)# source interface e1/17 both

NEXUS(config-monitor)# destination interface e1/32

NEXUS(config-monitor)# filter vlan 77,88

BRKDCT-1890 69


SPAN ACL Filtering Nexus 9000

• Configuration Example

NEXUS(config)# ip access-list match_my_pkts

NEXUS(config-acl)# permit ip 11.0.0.0 0.255.255.255 any

NEXUS(config)# vlan access-map span_filter 5

NEXUS(config-access-map)# match ip address match_my_pkts

NEXUS(config-access-map)# action forward


NEXUS(config-monitor)# filter access-group span_filter

BRKDCT-1890 70


Multi-Destination Virtual SPAN

• Use Case:«Breakout» high speed source

• To monitor multiple VLAN sources and choose only VLANs of interest to transmit on multiple destination ports

SPAN Destination802.1Q, Allowed VLAN 10

e2/1

802.1Q Trunk

VLANs 10-20




e1/1

e1/2

e1/3

e1/4

monitor session 1

source interface e2/1 both

destination interface e1/1




filter vlan 10-13

High-speed

Interface

Multiple SPAN destination configured

as trunk and allow vlan list

BRKDCT-1890 71


SPAN with ACL Capture

• Selectively monitor traffic on an interface or VLAN

• Packets that match ACL rule are permitted or denied and/or sent to an monitor destination

NEXUS(config)# monitor session 1 type acl-capture

NEXUS(config-acl-capture)#destination interface Ethernet1/32

NEXUS(config-acl-capture)#no shut

NEXUS# show monitor session 1

session 1

---------------

type : acl-capture

state : up

destination ports : Eth1/32

BRKDCT-1890 72



• Enable a capture session for an ACL's access control entries (ACEs) and then apply the ACL to an interface or VLAN filter-list (VACL)

• Capture session ID matches with the monitor session ID

• An example with the capture option applied to a VLAN-List

• Note: The ACL rule with the capture option can be also applied to an interfaces

ip access-list MY-ACL

10 permit udp any any capture session 1

vlan access-map MY-VACL 10

match ip address MY-ACL

action forward

vlan filter MY-VACL vlan-list 77

BRKDCT-1890 73



• The ACL Capture filter feature requires to enable hardware access-list command in the admin VDC or default VDC*

hardware access-list capture

*ACL capture is not supported with ACL logging

BRKDCT-1890 74


Packet Injection

• Allows device connected to SPAN destination interface to inject traffic into the network

• Specify input packets option when configuring SPAN destination:

e1/3

switchport monitor ingress

SMAC: 0000.0000.2222

e1/4

switchport monitor ingress learning

SMAC: 0000.0000.3333

e1/3 –

e1/4 0000.0000.3333

interface Learned MAC

MAC Table

e1/2

SMAC: 0000.0000.1111

switchport monitor

X

NEXUS(config)# interface ethernet 1/2

NEXUS(config-if)# switchport monitor

NEXUS(config-if)# interface ethernet 1/3

NEXUS(config-if)# switchport monitor ingress

NEXUS(config-if)# interface ethernet 1/4

NEXUS(config-if)# switchport monitor ingress learning

Allow inject packets, but do not learn the MAC

Allow inject packets and learn MAC

Normal SPAN session

BRKDCT-1890 75


Inband SPAN – Monitor control traffic

• Supervisor CPU sends/receives traffic via dedicated interface to Fabric using inband interface

• Monitoring direction is from perspective of switch fabric, not CPU

• Tx SPAN monitors traffic from switch fabric to CPU

• Rx SPAN monitors traffic from CPU to switch fabric

• One Inband SPAN session per switch supported


NEXUS(config-monitor)# source interface sup-eth 0

BRKDCT-1890 76


Inband SPAN – Monitor control traffic

• Inband SPAN Packet Trace Example

monitor session 1

source interface sup-eth0 both

rate-limit auto


no shut


switchport

switchport monitor

speed 1000

no shutdown

BRKDCT-1890 77


Real World ExampleHigh CPU – Use INBAND SPAN to find out!

NEXUS# show processes cpu sort

CPU utilization for five seconds: 100%/100%; one minute: 99%; five minutes:98%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

6131 11367100 1497150 7 78.02% 77.12% 76.35% - X

5615 44622720 3059816 14 15.121% 14.13% 14.59% - Y


NEXUS(config-monitor)# source interface sup-eth 0

BRKDCT-1890 78


Rule Based SPAN – SPAN-Filter

• Filter applied selectively on a session results in desired subset of traffic

• Filter by L2/L3/L4 fields

BRKDCT-1890 79



• Configure a filter within the session configuration mode

• Simple SPAN-Filter Configuration:

monitor session 1

source interface Ethernet1/17 both

rate-limit auto


filter frame-type ipv4 src-ip 10.10.77.113/32

filter frame-type ipv4 dest-ip 10.10.77.114/32

no shut

Boolean AND between

different filters

BRKDCT-1890 80



• How is ACL different from SPAN-Filter?

• ACL

• Applied on interfaces & vlans

• Requires large TCAM size

• SPAN-Filter

• Applied on a SPAN session

• Limited TCAM space

BRKDCT-1890 81


SPAN – Filters, ACL etc…Confused?

• Nexus 5600/6000 :

• ACL Filters for SPAN (Use Access lists to filter SPAN)

• Nexus 7000/7700 :

• VLAN filters (Filter by VLAN)

• Rule based SPAN (Filter by L2/L3/L4 fields)

• Nexus 3100/9200/9300/9500 :

• VLAN filters (Filter by VLAN)

• ACL Filters for SPAN (Use Access lists to filter SPAN)

BRKDCT-1890 82


SPAN Rate Limiting

• Limits the number of SPAN copies made on ingress

• In manual mode, the rate limit will be in 1-100 range, i.e., 1%, 2%, 3% …100% of 10G SPAN rate

• In auto mode, the rate limit will automatically calculated as follows:

• Rate limit = Destination Bandwidth / Source Bandwidth

• Rate Limiting is by default in auto-modeNEXUS(config-monitor)# [no] rate-limit [auto | manual [1..100]]

BRKDCT-1890 83


Sampled SPAN

• Used to provide an accurate count of the SPAN source packets

• Sampling and MTU truncation can be enabled at the same time and have no precedence over each other because they are applied to different aspects of the source packet (packet count versus size)

• Sampling takes precedence over SPAN source rate limiting. Rate limiting takes effect after sampling is completed on SPAN source packets

NEXUS(config-monitor)# sampling [2..1023]

BRKDCT-1890 84


Exception SPAN

• Exception SPAN enables you to span exception packets

• Packets that have failed an build-in Nexus 7x00 intrusion detection system (IDS); as example for Layer 3 IP verification

• Rate limiters, MTU truncation, and sampling are supported in the exception SPAN session

• Exception SPAN is supported in TX direction only

BRKDCT-1890 85


Exception SPAN – Verify CLI (IP IDS)

NEXUS# show hardware ip verify

IPv4 IDS Checks Status Packets Failed

-----------------------------+---------+------------------

address source broadcast Enabled 65536

address source multicast Enabled 65536

address destination zero Enabled 65536

address identical Enabled 65536

checksum Enabled 768

protocol Enabled 0

fragment Enabled 0

length minimum Enabled 0

length consistent Enabled 0

length maximum max-frag Enabled 0

length maximum max-tcp Enabled 0

tcp flags Enabled 0

tcp tiny-frag Enabled 0

version Enabled 0

BRKDCT-1890 86


Exception which lead to SPAN

• length minimum = if the packet length is smaller than 64 bytes

• length consistent = when L2 frame size is shorter than the expected length to include the IP packet + MAC header.

• length maximum max-frag = if the packet fragment exceeds allowed fragmentation count

• length maximum udp = if the UDP payload is larger than specified

• length maximum max-tcp = if the TCP payload is larger than specified

• tcp flags = if incorrect flags are set in the TCP packet

• tcp tiny-frag = if TCP payload is smaller or is fragmented unexpectedly

• Version = if the IP header version is incorrect

BRKDCT-1890 87


Exception SPAN

• Each VDC supports one exception SPAN session

• Configuration Example


NEXUS(config-monitor)# source exception all

NEXUS(config-monitor)# destination interface ethernet 2/5

NEXUS(config-monitor)# no shut

BRKDCT-1890 88


Exception which lead to SPAN Exceptions Brief explanation

No route in hardware This is seen when adjacency is not yet formed

Unicast/Multicast route error

(incoming/outgoing interface)

This is seen when the outgoing interface is not available (say,

when the LC is reloaded)

Multicast DF failure Seen when the designated forwarder is not available.

SMAC IP check failure Incorrect SMAC / DMAC combinations, like multicast SRC

MAC or SRC.IP = DST.IP or SRC.IP is a broadcast address or

DST.IP is all zeros

Protocol field failure Incorrect IP protocol specified in the IP header

FCS / CRC errors Errors related to incorrect FCS or CRC

BRKDCT-1890 89


Exception which lead to SPAN Exceptions Brief explanation

TTL expiry When the number of hops in the header exceeds TTL

configured

SPAN replication before L2/L3 ACL deny If the copy is made before the decision engine takes a

decision, it is Ingress replication

IPV6 scope check fail Seen when there are multiple link-local addresses tied to an

interface and the route does not exist for the packet through

either one of them,.

MTU fail When packet size exceeds the link MTU

Stale adjacency When the adjacency does not exist / is not updated for a long

time / fails refresh

CoPP violations Any packets that violated CoPP rate-limits

BRKDCT-1890 90


Real World ExampleCRC errors – Use Exception SPAN

Packet dropped in hardware

Packet which came in didn't

make it to the egress

Use Exception SPAN

– Find reason for

drop and what was

dropped!

Packets coming into interface

were mishandled by Transceiver

leading to CRC errorsReceive packet from

wire

BRKDCT-1890 91


Real World ExampleMalfunction NIC – Use Exception SPAN

Use Exception SPAN

– Find reason for

drop and what was

dropped!

Receive packet from

wire

Packet dropped in hardware

Packets from the server were

sent with BROADCAST

SOURCE IP because of

MALFUNCTIONING NIC

I didn’t

receive the

data!

BRKDCT-1890 92


SPAN Sessions on Nexus 7000

• 14 active unidirectional SPAN session

• 2 bidirectional local SPAN sessions per system

• 11 unidirectional extended sessions with F2/F2E/F3 modules present in addition to 2 traditional SPAN sessions

• 12 unidirectional extended sessions with F1/M2 modules present in addition to 2 traditional SPAN sessions

• M1 supports only 2 bidirectional local SPAN sessions per system

BRKDCT-1890 93



• 16 active unidirectional SPAN session (F2E, F3 and M3)

• All SPAN sessions are unidirectional and any two can be combined to create a bidirectional session

• The Cisco Nexus 7700 switch does not have standard and extended sessions

BRKDCT-1890 94



• Nexus 9500: up to 32 active SPAN session

• Scale is based on the number of linecards and the SPAN source interface to ASIC mapping

• Nexus N9200-X / N9200-Q/C / N9300 / N9300-EX): 4 active SPAN session

• Up to 3 bidirectional session plus 1 unidirectional

BRKDCT-1890 95

Review SPAN


SPAN Overview Nexus 5600/6000 and 7x00

SPAN Features* Nexus 5600/6000 Nexus 7000 Nexus 7700

ERSPAN destination session Yes All except F1 All LC’s

Prioritize data over SPAN Yes Yes (F2E/F3/M1/M2) Yes (F2E/F3/M3)

Line-rate SPAN throughput Yes No No

ERSPAN (v3) with 1588 PTP

timestampYes M2/F2/F2E/F3 F2E/F3/M3**

Number of SPAN destinations 16 32 N/A

SPAN with MTU truncation Yes Yes (Except M1) Yes

Virtual SPAN Yes Yes Yes

ACL filters Yes Rule based SPAN Rule based SPAN

SPAN source as VLAN Receive only Bidirectional Bidirectional

*Please check release notes for additional details and support

**M3 has HW support for ERSPAN III with IEEE15888, SW

support is pending BRKDCT-1890 97


SPAN Overview – Nexus 3000/9200/9300

SPAN Features* Nexus 3100Nexus 9300

1st Generation

Nexus 9200-X

Nexus 9200-Q/CNexus 9300-EX**

SPAN source as VLAN Receive only Receive only Receive/Transmit Receive/Transmit

ERSPAN destination

session (V2 and V3)Yes No HW support HW support

ERSPAN with V2 header Yes Yes-Only on

uplink portsYes Yes

Prioritize data over SPAN Yes Yes Yes Yes

Line-rate SPAN throughput Yes Yes Yes Yes

ERSPAN V3 with 1588 PTP

timestampsNo

Yes-Only on

uplink portsYes Yes

Number of SPAN session 1 1 4 4

ACL filters for SPAN Yes Yes Yes Yes

*Check SPAN/ERSPAN Configuration Documentation for details on CCO

** NX-OS Standalone will be available Q3 CY2016 BRKDCT-1890 98


SPAN Overview – Nexus 9500

SPAN Features*Nexus 9500

Linecards 9400/9500/9600

Nexus 9500

Linecard 9700-EX**

SPAN source as VLAN Receive only Receive/Transmit

ERSPAN destination session No HW support

ERSPAN with V2 header No Yes

Prioritize data over SPAN Yes Yes

Line-rate SPAN throughput Yes Yes

ERSPAN V3 with 1588 PTP timestamps No Yes

Number of SPAN destinations per session 32 32

ACL filters for SPAN Yes Yes

*Check SPAN/ERSPAN Configuration Documentation for

details on CCO

** NX-OS Standalone will be available Q3 CY2016 BRKDCT-1890 99

• Introduction



• SPAN / ERSPAN


• Conclusion

Agenda


Flexible NetFlow

• Enhanced network anomaly

• Customized user configurable flow (records)

• Monitor a wider range of packet information

BRKDCT-1890 101


NetFlow = Visibility A single NetFlow Record provides a wealth of information

switch# show flow monitor MONITOR-1 cache

…

IPV4 SOURCE ADDRESS: 192.168.100.100

IPV4 DESTINATION ADDRESS: 192.168.20.6

TRNS SOURCE PORT: 47321

TRNS DESTINATION PORT: 443

INTERFACE INPUT: E1/1

IP TOS: 0x00

IP PROTOCOL: 6

ipv4 next hop address: 192.168.20.6

tcp flags: 0x1A

interface output: Gi0/1.20

counter bytes: 1482

counter packets: 23

timestamp first: 12:33:53.358

timestamp last: 12:33:53.370

ip dscp: 0x00

ip ttl min: 127

ip ttl max: 127

application name: nbar secure-http

…

BRKDCT-1890 102


Layer 2 NetFlow versus Bridged NetFlow

• Layer 2 NetFlow - ability to collect IP traffic statistics based on the packet’s Layer 2 header and thus allowing for SRC/DST MAC accounting

• Bridged NetFlow - ability to collect statistic for IP traffic being bridged within a given VLAN. The flow information will be based on the packet’s Layer 3 and Layer 4 headers, allowing for applications visibility

DMAC SMAC VLAN Ethertype

000A:ABCD:00EF 001E:A12D:1287 16 0x86DD

Layer2 NetFlow

Flow Information

IP SA IP DA IP ProtoLayer4 SRC

PortLayer4 DST

Port

115.12.34.2 115.12.34.3 6 1023 5230

Bridged NetFlow

Flow Information

BRKDCT-1890 103


Seven Steps of Flow Creation

1 Packet

Extract relevant fieldsFlow

Flow

Flow

Flow

Flow Statistics

Statistics

Statistics

Statistics

Statistics

DMAC SMAC VLAN Ethertype

000A:ABCD:00EF 001E:A12D:1287 16 0x86DD

43

I/O module collects the flows and their

statisticsonce the flow ages out

5

2

6

Formatted into

NetFlow Export

7 Collector

I/O Module

BRKDCT-1890 104


Full versus Sampled NetFlow

• NetFlow collects full or sampled flow data

• Full NetFlow: Accounts for every packet of every flow on interface

• Available on M1/M2 modules only on Nexus 7000

• Flow data collection up to capacity of hardware NetFlow table

• Sampled NetFlow: Accounts for M in N packets on interface

• Available on M1/M2 and F3/M3 in Nexus 7x00 and Nexus 5600/6000

• M2: Flow data collection up to capacity of hardware NetFlow table

• F3: Flow data collection for up to ~500pps per ASIC(SOC) module before NX-OS 7.2

• F3/M3: Increased per-module sampling rate leveraging on-board Fabric Services Accelerator (FSA) complex to ~50kpps with NX-OS 7.2

• Nexus 5600/6000: Flow data collection for up to ~120kpps per chassis

BRKDCT-1890 105


NetFlow on M2 Modules

Fabric

ASIC

VOQs

MgmtEnet

Supervisor

Engine

Forwarding

Engine

LC

CPU

NetFlow

Table

M2 Module

Forwarding

Engine

LC

CPU

NetFlow

Table

M2 Module

Forwarding

Engine

LC

CPU

NetFlow

Table

M2 Module

Hardware

Flow Creation

Hardware

Flow Creation

Hardware

Flow Creation

Aged Flow Info

Aged Flow Info

Aged Flow Info

Generate NetFlow v5

or v9 export packets

Main

CPU

To NetFlow Collector


Switched

EOBC

via Supervisor

Inband

via mgmt0

BRKDCT-1890 106


NetFlow on F3/M3 Modules

F3/M3 Module

FSA

CPU

SoC

Decision

Engine

DRAM

NetFlow

Cache

F3/M3 Module

Fabric

ASIC

VOQs

MgmtEnet

Supervisor

Engine

FSA

CPU

SoC

Decision

Engine

Main

CPU



Switched

EOBC

via mgmt0

DRAM

NetFlow

Cache

Populate cache based

on received samples

Age flows and

generate NetFlow v5

or v9 export packets

F3/M3 Module

FSA

CPU

SoC

Decision

Engine

DRAM

NetFlow

Cache

Data Flow

Data Flow

Data Flow

via Module

Inband

via Module

Inband

via Module

Inband

Sampled

Packets

Sampled

Packets

Sampled

Packets

Aged

Flows

Aged

Flows

Aged

Flows

BRKDCT-1890 107


NetFlow - Traffic StatisticsConfiguration Steps for Full Netflow*

flow exporter FLOW-EXPORT

description NetFlow v9 Exporter

destination 11.1.1.1 use-vrf management

source Loopback0

transport udp 2055

version 9

flow monitor FLOW-MONITOR

description NetFlow v9 Monitor

record FLOW-RECORD

exporter FLOW-EXPORT

interface eth 1/1

ip address 172.16.0.1 255.255.255.0

ip flow monitor FLOW-MONITOR input

ip flow monitor FLOW-MONITOR output

flow record FLOW-RECORD

match ipv4 source address

match ipv4 destination address

match transport source-port

match transport destination-port

collect counter bytes

collect counter packets

collect timestamp sys-uptime first

collect timestamp sys-uptime last

NON-KEY

Flexible NetFlow

KEY

1. Create Flow Record

2. Create Flow Exporter

3. Associate Record and Exporter to a Flow Monitor

4. Apply to the interface

*command “feature netflow” is not shown BRKDCT-1890 108


NetFlow - Traffic StatisticsConfiguration Steps for Sampled Netflow*

flow exporter FLOW-EXPORT

description NetFlow v9 Exporter


source Loopback0

transport udp 2055

version 9

flow monitor FLOW-MONITOR

description NetFlow v9 Monitor

record FLOW-RECORD

exporter FLOW-EXPORT

interface eth 1/1

ip address 172.16.0.1 255.255.255.0

ip flow monitor FLOW-MONITOR input sampler FLOW-SAMPLER

ip flow monitor FLOW-MONITOR output sampler FLOW-SAMPLER

flow record FLOW-RECORD





collect counter bytes


collect timestamp sys-uptime first

collect timestamp sys-uptime last

NON-KEY

Flexible NetFlow

KEY

1. Create Flow Record

2. Create Flow Exporter

3. Associate Record and Exporter to a Flow Monitor

4. Create Flow Sampler

5. Apply Flow Monitor and Flow Sampler to the

interface

*Command “feature netflow” is not shown

sampler FLOW-SAMPLER

description Netflow v9 Sampler

mode 1 out-of 1200

BRKDCT-1890 109


NetFlow – Traffic Statistics

• Example with a N77-F348XP-23 linecard (6 SOC/ASICs per Linecard), FSA enabled

• Assumption: average packetsize 512bytes, average traffic rate max 50%

• Per F3 Linecard sampling rate (FSA) = 50kpps, per SOC/ASIC = 8’333pps*

• Calculation

• 50% x 10Gbps / 512 Bytes = ~1’221kpps

• 8 Ports per SOC: 8 x 1’221kpps = 9’768kpps

• Calculated Sampling Rate: 9’768kpps / 8’333pps = ~ 1172

• Recommended sampling rate to be configured = 1 : 1’200

Calculating the sampling rate

*Exceeding per SOC sampling rate will result tail-dropping

packets due the rate limiter BRKDCT-1890 110


Nexus 7000 Control-Plane NetFlow

• Creates NetFlow records for control-traffic* punted to the CPU on the Supervisor

• Traffic flows from Linecard to CPU, hence the NetFlow monitor could be applied in egress (output) direction only

• Flow monitor is applied on control-plane interface

• Only sampled Netflow is supported

• Configuration applied in the default VDC

• Linecard specific NetFlow capabilities and resources are used for creating this internal control-plane flows

NX-OS

7.3

*today unicast control-plan traffic, only multicast control-

traffic should be supported from the next 7.3

Maintenance Release (Q3/4 CY2016) BRKDCT-1890 111


Nexus 7000 Control-Plane NetFlow Resolving High CPU using CoPP NetFlowNEXUS# show processes cpu sort

CPU utilization for five seconds: 65%/8%; one minute: 63%; five minutes: 61%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

310 30544 189234 81 47.12% 45.11% 45.23% 0 IP Input

flow record NF-RECORD






High CPU due to process “IP Input”

Building a NetFlow record, matching L3 and L4

parameters (key fields) and collecting details on

Input interface and packet count (non-key fields)

NX-OS

7.3

flow exporter NF-EXPORT-1


transport udp 2055

source mgmt0

version 9

Optional: Building a NetFlow exporter

sampler NF-SAMPLER

mode 1 out-of 1Create a sampler

BRKDCT-1890 112


Nexus 7000 Control-Plane NetFlow Resolving High CPU using CoPP NetFlow

NX-OS

7.3

flow monitor NF-MONITOR

record NF-RECORD

exporter NF-EXPORT-1

Create the flow monitor and associate Record

and Exporter

control-plane

ip flow monitor NF-MONITOR output sampler NF-SAMPLER

NEXUS# show hardware flow ip

---snip---

D - Direction; L4 Info - Protocol:Source Port:Destination Port

IF - Interface: (Eth)ernet, (S)vi, (V)lan, (P)ortchannel, (T)unnel

TCP Flags: Ack, Flush, Push, Reset, Syn, Urgent

D IF SrcAddr DstAddr L4 Info PktCnt TCP Fl

--+-----------+---------------+---------------+---------------+----------+------

CP sup-eth1 104.104.104.011 104.104.104.021 000:00000:00000 0000000100 ......

Applying to the control-plane interface the NetFlow

monitor in egress direction with a sampler

Check your control-plan flow entries

Troubleshooting Methodology:

Once the flow is identified, further action could be (1) blocking the flow with an

Access List (ACL) (Infrastructure or CoPP) or/and (2) rate-limiting the flow using

CoPP depending on the criticality of the flow to the production


NetFlow Overview

M2 (N7000) F3 (Nexus 7x00) M3 (Nexus 7700) Nexus 5600/6000

Per-interface NetFlow Yes Yes Yes Yes

NetFlow direction Ingress/Egress Ingress / Egress* Ingress / Egress Ingress only

Full NetFlow Yes No No No

Sampled NetFlow Yes Yes Yes Yes

FSA assisted for Sampled

NetFlowNo Yes* Yes No

Bridged NetFlow Yes Yes Yes Yes

Hardware Cache Yes No No No

Software Cache No Yes Yes Yes

Hardware Cache Size512K entries per

forwarding engineN/A N/A N/A

NDE (v5/v9) Yes Yes Yes Yes

*supported since NX-OS 7.2

Note: Nexus 9K (N9200-X/N9300-EX/N9700-EX) supports full NetFlow; software support is on the roadmap

BRKDCT-1890 114

• Introduction



• SPAN/ ERSPAN

• NetFlow

• Conclusion

Agenda


Tools designed with you in mind

• Advanced feature rich analytics tools

• Visibility into the products helping to validate the path-of-the-packet

• Analytics tools can help in isolating problems we see in Datacenters today

• Reduce the time to resolution of network issues

Netflow

SPAN ERSPAN

ACL CaptureLatency

monitoringSPAN-on-drop

Microburst

monitoring

BRKDCT-1890 116


Call to Action

• Attend the following related sessions

• BRKDCN-3020 - Network Analytics using Nexus 3000/9000 Switches

• BRKARC-3452 - Cisco Nexus 5600/6000 Switch Architecture

• BRKARC-3470 - Cisco Nexus 7000/7700 Switch Architecture

• BRKARC-2222 - Cisco Nexus 9000 Architecture

• BRKARC-2011 - Overview of Packet Capturing Tools in Cisco Switches and Routers

• Visit the World of Solutions for

• Cisco Campus | Walk in Labs | Technical Solution Clinics

• Meet the Engineer

• Lunch and Learn Topics

• DevNet zone related sessions -

BRKDCT-1890 117

https://cisco.rainfocus.com/scripts/catalog/cleu16.jsp?search.sessiontype=lunchandlearn

https://cisco.rainfocus.com/scripts/catalog/cleu16.jsp?search.sessiontype=devnetzone


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

BRKDCT-1890 118

CiscoLive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/us

http://ciscolive.com/us


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKDCT-1890 119

Please join us for the Service Provider Innovation Talk featuring:

Yvette Kanouff | Senior Vice President and General Manager, SP Business

Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

11:30 am - 12:30pm, In the Oceanside A room

What to expect from this innovation talk

• Insights on market trends and forecasts

• Preview of key technologies and capabilities

• Innovative demonstrations of the latest and greatest products

• Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com

Thank you


White Papers on Cisco Connection Online (CCO)

• Monitor Microbursts on Cisco Nexus 5600 Platform and Cisco Nexus 6000 Series Switches

• http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/white-paper-c11-733020.html

• SPAN-on-Latency Feature on Cisco Nexus Switches: Troubleshoot Network Latency


BRKDCT-1890 123

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/white-paper-c11-733020.html



White Papers on Cisco Connection Online (CCO)

• SPAN-on-Drop Feature on Cisco Nexus Switches: Troubleshoot Network Congestion


• Latency Monitoring Tool on Cisco Nexus Switches: Troubleshoot Network Latency


BRKDCT-1890 124



network visibility using advancedd2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/brkdct-1890.pdfnetwork...

Documents