traffic monitoring and engineering for...

Traffic Monitoring and Engineering for UCSBRKCOM-2004

Steve McQuerry, CCIE# 6108

Technical Marketing Engineer

© 2013 Cisco and/or its affiliates. All rights reserved.BRKCOM-2004 Cisco Public

Agenda

UCS Networking Overview Network Statistics in UCSM Understanding Collection Policies Hotspot Detection Engineering to Avoid Hotspots

3

UCS Networking Overview


System Components: High-level Overview

5

• 61XX• 62XXUP

• 2104XP• 2204XP• 2208XP

• Cisco (VIC1280, VIC1225)• 3rd party

FABRIC INTERCONNECT

CHASSIS IO MODULE (FEX)

INTERFACE CARDS

• 2232PP



Fabric Interconnects–(10GE ports)

Chassis–Up to 8 half width blades or 4 full width blades

Fabric Extender (FEX or I/O Module)–Host to uplink traffic engineering–Up to 160Gb Flexible bandwidth allocation

Adapters–Virtualized adapter for single OS and hypervisor systems–Dual connected

Blade or Stand Alone Server

6



1st Generation Hardware

7

vpc

Mezz

Fabric A Fabric B

Mezz Mezz

FEX AFEX A FEX BFEX B

ni

c ni

c hb

a hb

a ni

c ni

c hb

a hb

a ni

c ni

c hb

a hb

a

SAN A SAN B

Port-Channel Port-Channel


UCS 2104 (1st Gen FEX) — Server to Fabric Pinning

8

Fabric Interconnect1 link

2 links

4 links

Server slots pinned to uplink

Uplink: slots 1,2,3,4,5,6,7,8

Uplink 1: slots 1,3,5,7Uplink 2: slots 2,4,6,8

Uplink 1: slots 1,5Uplink 2: slots 2,6Uplink 3: slots 3,7Uplink 4: slots 4,8

slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8



NIF

NIF

NIF

FEX

FEX

Fabric Interconnect

Fabric Interconnect

FEX



1st Generation Hardware

9

Fabric A Fabric B

Mezz

IOM AIOM A IOM BIOM B

ni

c ni

c hb

a hb

a



2nd Generation Hardware

10

Mezz

Fabric A Fabric B

Mezz Mezz


ni

c ni

c hb

a hb

a ni

c ni

c hb

a hb

a ni

c ni

c hb

a hb

a

SAN A SAN B

vpc



UCS 2204 (2nd Gen FEX) — Sever to Fabric Pinning

11


2 links

4 links


Uplink: slots 1,2,3,4,5,6,7,8






NIF

NIF

NIF

FEX

FEX

Fabric Interconnect

Fabric Interconnect

FEX


UCS 2204 (2nd Gen FEX) — Sever to Fabric Pinning

12

4 links

Server slots channeled acrossall uplinks

Uplink 1: slots 1-8Uplink 2: slots 1-8Uplink 3: slots 1-8Uplink 4: slots 1-8


NIF

FEX

Fabric Interconnect


UCS 2208 (2nd Gen FEX) — Server to Fabric Pinning

13


2 links

4 links


Uplink: slots 1,2,3,4,5,6,7,8






NIF

NIF

NIF

FEX

FEX

Fabric Interconnect

Fabric Interconnect

FEX


UCS 2208 (2nd Gen FEX) — Server to Fabric PinningServer slots pinned to uplink

8 linksUplink 1: slot 1Uplink 2: slot 2Uplink 3: slot 3Uplink 4: slot 4Uplink 5: slot 5Uplink 6: slot 6Uplink 7: slot 7Uplink 8: slot 8


NIF

FEX

Fabric Interconnect

8 links Uplink 1: slots 1-8Uplink 2: slots 1-8Uplink 3: slots 1-8Uplink 4: slots 1-8Uplink 5: slots 1-8Uplink 6: slots 1-8Uplink 7: slots 1-8Uplink 8: slots 1-8


NIF

FEX

Fabric Interconnect

Server slots channeled acrossall uplinks

14



2nd Generation Fabric Interconnect

15

Fabric A Fabric B

Mezz


ni

c ni

c hb

a hb

a

Network Statistics in UCSM


UCS CLI

DMEUCSM AG blade

Events {xml}

HTTP{xml}

DB

AGn

Polling/notification

XML API

CIM Object Translation

IBM Tivoli

Syslog

AG FI AG chassis

SNMPCIM-XML

HPNNM HPOM Misc. SNMP

Smash

Events{SNMPTraps}

Polls{SNMPGET}

UCSM Core

17


5 most recent stats history records are contained by the current stats instance.

Stats history are subclasses of the stats.

Root

topSyssys

equipmentChassissys/chassis-1

equipmentChassisStatssys/chassis-1/stats

equipmentChassisStatsHistsys/chassis-1/stats/1

... equipmentChassisStatsHistsys/chassis-1/stats/5

Statistics are contained by monitored object

Statistics are instances of statsCurr

Statistics dn is often deducible.

UCSM Statistics

18


Stats Collection Policy

A Systemwide Set of Stats Collection Policies (One per Application Domain) Allow Configuration of Collection and Reporting Intervals.Domains: adapter, chassis, host, port, serverWhen a Collection Interval Is Changed, the reportingInterval Is Restarted collectionInterval: frequency that endpoints will send stats updates to DME reportingInterval: frequency that DME will report stats updates to external

collectors and update stats history

19



Network statistics are collected by UCSM from the NX-OS software in the Fabric Interconnects. These are counters that are available for networking components Because of NIV technology the Fabric has visibility to the Cloud (LAN/SAN

uplinks), the IOM and the server NIC.

20



Access Statistics through:– LAN or SAN tab (port-group) – Devices tab ( server ports, network uplink, storage ports, and mezz ports,)– Server Tab (vNIC)

21



FI to LAN Network Uplink (Cloud)

22

Fabric A Fabric B

SAN A

vpc




Port Channel is the Aggregate of all interfaces in the channel Statistic for the channel are the sum of the statistics of the members Individual member statistics are also visible in the system Network usage is measured against TX Total bytes and RX Total bytes

23


Network Statistics in UCSMFI to Uplink Port Channel Statistics

24


Network Statistics in UCSMFI to Uplink Individual Port Statistics

25


Network Statistics in UCSMFI to SAN Port Statistics

26

Fabric A Fabric B

SAN A SAN B



A FC Port Channel is the Aggregate of all interfaces in the channel

Statistic for the channel are the sum of the statistics of the members

Individual member statistics are also visible in the system

FC usage is measured against Bytes RX and Bytes TX

FI to SAN Uplink (Cloud) Port Statistics

27


Network Statistics in UCSMFI to SAN Port Statistics

28


Network Statistics in UCSMFI to IOM (Internal LAN)

29

Fabric A Fabric B



Network Statistics in UCSMFI to IOM (Internal LAN)

30


UCS Networking OverviewServer to IOM

31

Mezz

ni

c ni

c hb

a hb

a


Network Statistics in UCSMServer to IOM vNIC Port Statistics

32


Using SNMP tools to Monitor Traffic

The network statistics in UCSM are pulled from the NX-OS statistics for the networking components.

It is possible to monitor the traffic for the networking components using an external SNMP collection tool such as MRTG.

The advantage is long term analysis, however most tools have no alerting capabilities.

33

Understanding Collection Policies



A collection Policy consist of a collection interval and a reporting interval The collection policy is set under the admin tab in Stats Management ->

Collection Policies -> Collection Policy name A unique policy can be set for, Adapters, Chassis, FEX, Port, Server, and Host* Not all policies involve network components

35

*Host is an unused policy in UCSM at present



36



The collection interval is how often the system will query a device for statistics.

The default collection interval is 60 seconds

The more frequent the interval the more granular the data. We will use 30 seconds.

The timing of the collection interval is important because it will be used in BW calculations for hotspot detection

Collection Interval

37



The reporting interval is internal to UCSM and determines how often UCSM will store data from the collection interval.

This data is stored in tables and the last 5 reporting intervals are available for inspection in the system

Reporting interval data is used to calculate minimum, maximum and average values shown in the statics view.

Reporting Interval

38



Select the Policy you want to change. Make selections and press the save changes button

39

Hotspot Detection


Hotspot Areas

41

Fabric Interconnects to LAN/SAN

FEX to Host

FEX to FI

Mezz

Fabric A Fabric B

Mezz Mezz

FEX AFEX A FEX BFEX B

ni

c ni

c hb

a hb

a ni

c ni

c hb

a hb

a ni

c ni

c hb

a hb

a

SAN A SAN B


There are three potential hotspot locations for UCS network connectivity.

1.) Send and Receive between Fabric Interconnect and LAN/SAN

2.) Send and Receive between FEX and Fabric Interconnect

3.) Send and Receive between Host and FEX


Hotspot Detection

To identify hot spots we will use Threshold policies in conjunction with collection policies to alert as we pass thresholds.

A threshold is calculated by measuring a statistic against a policy.

The policy measures change against a user defined normal value and turns on the alert between a users set high/low threshold and turns off the alert below the user set low threshold.

Threshold Policies

42

High - Up

Low - Up

Normal

High - Up

Low - Up

Normal

High - Up

Low - Up

Normal


Hotspot Detection

The statistic we use to calculate bandwidth is the delta in bytes.

This should be measured for both TX and RX

This delta is calculated in bytes changed over a period of time defined by the collection interval, for example 30 seconds

Calculating BW Threshold Limits for an Element

43

‟… if standardizing on one size was a good thing, standardizing on several sizes would be even better.”

• Radia Perlman• Interconnections: Bridges and Routers

44


Kilo, Mega, etc

What is a Kilobit?– “Once upon a time… Computer professionals noticed that 210 was very nearly equal to 1000

and started using the SI prefix “kilo” to mean 1024.– The rest of the world uses the term Kilo in the SI format to mean 1000.– The result is that no one seems to know what kilo and mega mean.

The IEEE standards board decided that the IEEE standards will use the conventional, internationally adopted, definitions of the SI prefixes. Therefore Mega will mean 1,000,000 and Kilo will mean 1,000.*

In Telecom and by extension networking Kilo means 1000 and NOT 1024

The SNMP reported speed for a 10Gig link (on any platform) will be reported by SNMP is 1,250,000,000 – Bytes (MaxBytes[10.29.148.6_436207616]: 1250000000)

*Sources: NIST and IEEE

IEEE Networking Standards for Prefix Values

45


Hotspot Detection

For Ethernet the BW of a single link is 10Gbps

First we determine our desired threshold for example 8Gbps

We need to calculate the expected change in bytes over the collection interval.

To calculate divide the desired BW by 8 bits per byte and then multiply by the collection interval by the time to get the expected delta in bytes for our collection period.

Example 8 Gbps over 30 seconds = 30,000,000,000 bytes– 8Gbps / 8bits per byte = 1,000,000,000 bytes per second – 1,000,000,000 bytes per second * 30 seconds = 30,000,000,000 bytes

Calculating BW Threshold Limits for an Element

46


Hotspot DetectionThreshold Calculations

47

Speed in Gbps Percentage of BW Conversion to BytesDelta expected over 30 second collection interval

10 100% 1,250,000,000 37,500,000,000

9 90% 1,125,000,000 33,750,000,000

8.5 85% 1,062,500,000 31,875,000,000

8 80% 1,000,000,000 30,000,000,000

7.5 75% 937,500,000 28,125,000,000

7 70% 875,000,000 26,250,000,000

6.5 65% 812,500,000 24,375,000,000

6 60% 750,000,000 22,500,000,000

5 50% 625,000,000 18,750,000,000

4 40% 500,000,000 15,000,000,000

3 30% 375,000,000 11,250,000,000


Hotspot DetectionThreshold Alerts

48


Hotspot Detection

Threshold policies can be configured for the following:– Internal LAN – IOM to FI– LAN Cloud – FI to Upstream Ethernet switches– SAN Cloud – FI to Upstream SAN switches– Server – Between the server NIC and the IOM

Threshold Policies Placement

49


Hotspot Detection

Navigate to admin->stats management and expand the fabric.

Select thr-policy-default and create a threshold class

Choose Ether Tx Stats from the stat class and click next.

Threshold Policies Internal LAN

50


Hotspot DetectionThreshold Policies Internal LAN

51



52


Hotspot Detection

Click the add button and select Ether Tx Stats Total Bytes Delta as the property type enter 0.0 as the normal value

Select the Alarm triggers you want to get and enter your values and click OK

Click finish to be returned to the policy.

Click the classes tab to see your policy.

Threshold Policies

53



54


Hotspot Detection

You will need to add another class for RX traffic.

Click the + bottom to the right and repeat the steps from the previous policy choosing Eter Rx Stats as the stats class this time

Click save changes once you have completed the steps

Threshold Policies

55


Hotspot DetectionThreshold Policies

56


Hotspot Detection

For Uplinks you can repeat this process for the LAN cloud.

For SAN use a single Stats class fcstats and create a definition for rx and tx stas under the same stats class

Threshold Policies

57



58


Hotspot Detection

For the vNIC port you will need to create a threshold policy to be used with a service profile.

Go the the appropriate organization level and select create threshold policy

Threshold Policies

59



60


Hotspot Detection

Give the policy an applicable name and description and press next

Choose the vnic stats class and create a single threshold with the rx bytes delta and the tx bytes delta.

Threshold Policies

61



62


Hotspot Detection

Apply the threshold policy to the Service profile of the servers you want to monitor

Threshold Policies

63


Hotspot Detection

When a server reaches a threshold you will receive an alert on UCSM.

This will exist while threshold is exceeded once it drops below the definition the alert will disappear

Alerts show as system faults as defined by the threshold policy

UCSM will send a TRAP when an alert is generated.

The TRAP will be sent regardless of what level of alert has been set.

Threshold Policies

64



65


Hotspot Detection

When a threshold policy has been crossed UCSM will send out an SNMP Trap alert

This assumes that UCSM has been configured for SNMP operation and the Trap Receiver has been defined

This feature was added in UCSM version 2.x.x

66


Hotspot Detection

UCSM has a limited amount of space to store statistics and can only provide the last 5 instances of collection intervals. It provides a good tool for monitoring thresholds, but does not provide a long term monitoring solution.

Tools like MRTG are great for Long Term analysis of BW usage, but may not provide the required alerting and usually require a great deal of customization to be usable

UCS Central version (TBD) will provide tools for both alerting as well as long term analysis of traffic within systems it manages.

Long Term Monitoring vs. Point in Time

67


Automating Threshold Policies

Creating threshold policies for the systems require time and effort. Isn’t there some way to push standard configurations to the system?

GoUCS is an automation tool available from the Cisco Developer Network (CDN) that allows users to build XML configuration scripts that can be sent to UCS.

PowerTool can also be used to automate this process

http://developer.cisco.com/

GoUCS and CDN

69

Engineering to Avoid Hotspots


Traffic Engineering

71

FI-1

Class-A Class-B Class-C

vNIC-1

vNIC-2

vNIC-3

FEX-1

FI-2

Blade-2, VIC-1vNIC-1

vNIC-2

vNIC-3

FEX-2

VIC with 3 vNICs

2 Fabric Extenders in chassis, each with 1 link to the FI.

2 FI, both with 1 connection to each FEX

Blade-1, VIC-1

vNICs can be pinned to a specific FI when created (with configurable failover to other switch)

Depending on requirements, vNICs could be pinned to one interconnect or distributed evenly

vNICs in System Class C pinned to one interconnectvNICs in System Class C pinned to one interconnect


Controlling Pinning in Profile

72

From the each server a fabric interconnect can be chosen to balance the traffic


Traffic Engineering

73

FI-1

Class-A Class-B Class-C

vNIC-1

vNIC-2

vNIC-3

FEX-1

FI-2

Blade-2, VIC-1vNIC-1

vNIC-2

vNIC-3

FEX-2

Blade-1, VIC-1

vNICs can be pinned to a specific FI when created (with configurable failover to other switch)

Depending on requirements, vNICs could be pinned to one interconnect or distributed evenly

vNICs in System Class C distributed across interconnectsvNICs in System Class C distributed across interconnects


QoS Architecture

No packet drops within the array– Largest buffers are on switch and

host memory, so congestion pushed to edges

– Priority Flow Control (PFC) used to ensure packet drops are at vNIC or Switch

All traffic in a CA system belongs to 1 of 6 System Classes

– Four are user configurable while the other two are for FCoE and standard Ethernet

QoS parameters can be configured at a per system class level, or a per vNIClevel.

74

Compute ChassisCompute Chassis

erx86 Computer x86 ComputerX

IIx8x8x8x8

B

MGMT

SS

B

X X X X X

CC

AAGG GG

GG GG

SAN

GG

RR

AAGG

GG GG

GG

RR

GG

PPMM PP

SANLAN

FabricSwitchFabricSwitch

FabricSwitchFabricSwitch

FabricExtender

FabricExtender

FabricExtenderFabricExtender

Compute Blade(Half slot)

Compute Blade(Half slot)

AdapterAdapter

Compute Blade(Full slot)

Compute Blade(Full slot)

AdapterAdapterAdapterAdapter


System Buffering/Queuing

75


User Configuration

Globally for each System Class

Globally for each System Class

COS value for packets in this class

COS value for packets in this class

Drop/No-drop behavior

Drop/No-drop behavior

Strict PriorityStrict Priority

Bandwidth/WeightBandwidth/Weight

Users configure QoS parameters at two levelsUsers configure QoS parameters at two levels

Class Name FC Bronze

COSValue 3 0

Drop/No-Drop No-Drop Drop

Strict Priority No No

Bandwidth/Weight 20% 30%

Example

76


User Configuration

77

For each vNIC (Egress properties)

For each vNIC (Egress properties)

System Class for traffic from this vNIC

System Class for traffic from this vNIC

Rate limit (Mbps)Rate limit (Mbps)

Burst Size (Kbytes)Burst Size (Kbytes)

Users configure QoS parameters at two levelsUsers configure QoS parameters at two levels

vNIC1 vNIC2 vNIC3

Class FC FC Bronze

Rate 4000 4000 5000

Burst 300 400 100

Example: Logical Server A


User Configuration – Example

78

Class Name FC Gold Ethernet BE

COS Value 3 1 0

Drop/No-Drop No-Drop Drop Drop

Strict Priority No No No

Bandwidth/Weight 1 (20%) 3 (60%) 1 (20%)

vNIC1 vNIC2 vNIC3

Class FC FC Eth. BE

Rate 4000 4000 5000

Burst 300 400 100

Logical Server A

Global System Class Definitions

vNIC1 vNIC2

Class Gold Eth. BE

Rate 600 4000

Burst 100 300

Logical Server B

FC TrafficHigh PriorityEthernet

Best EffortEthernet


QoS Tools

79

Priority Flow ControlPriority Flow Control

• Enables lossless Fabrics for each class of service

• PAUSE sent per virtual lane when buffers limit exceeded

Transmit Queues Ethernet Link Receive Buffers

EightVirtualLanes

OneOne OneOneTwoTwo TwoTwo

ThreeThree ThreeThreeFourFour FourFourFiveFive FiveFive

SevenSeven SevenSevenEightEight EightEight

SixSix SixSixSTOP PAUSE

COS based Bandwidth ManagementCOS based Bandwidth Management

• Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth

•802.1Qaz Enhanced Transmission

10 GE Link Realized Traffic Utilization

3G/s HPC Traffic3G/s

2G/s

3G/sStorage Traffic3G/s

3G/s

LAN Traffic4G/s

5G/s3G/s

t1 t2 t3

Offered Traffic

t1 t2 t3

3G/s 3G/s

3G/s 3G/s 3G/s

2G/s

3G/s 4G/s 6G/s

Among the tools used are aggregate shapers at the vNICs (VIC Adapter), ETS, Policers at the switch for each vNIC.


QoS Configuration in UCSMEnable QoS Classes in UCSM

80


QoS Configuration in UCSMCreate a QoS Policy

81


Applying QoS to a PolicyApply Policy to Adapter

82


Summary

UCSM is designed for optimized traffic flow Stats Management and Threshold policies allow you to monitor traffic levels QoS and Traffic engineering tools allow you to manage potential bottlenecks in

UCS

83


Maximize your Cisco Live experience with your free Cisco Live 365 account. Download session PDFs, view sessions on-demand and participate in live activities throughout the year. Click the Enter Cisco Live 365 button in your Cisco Live portal to log in.

Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Cisco Daily Challenge

points for each session evaluation you complete. Complete your session evaluation

online now through either the mobile app or internet kiosk stations.

84

traffic monitoring and engineering for...

Documents