traffic monitoring and engineering for...

Traffic Monitoring and Engineering for UCSBRKCOM-2004

Steve McQuerry, CCIE# 6108

Technical Marketing Engineer

Agenda

UCS Networking Overview Network Statistics in UCSM Understanding Collection Policies Hotspot Detection Engineering to Avoid Hotspots

UCS Networking Overview

System Components: High-level Overview

• 61XX• 62XXUP

• 2104XP• 2204XP• 2208XP

• Cisco (VIC1280, VIC1225)• 3rd party

FABRIC INTERCONNECT

CHASSIS IO MODULE (FEX)

INTERFACE CARDS

• 2232PP

Fabric Interconnects–(10GE ports)

Chassis–Up to 8 half width blades or 4 full width blades

Fabric Extender (FEX or I/O Module)–Host to uplink traffic engineering–Up to 160Gb Flexible bandwidth allocation

Adapters–Virtualized adapter for single OS and hypervisor systems–Dual connected

Blade or Stand Alone Server

1st Generation Hardware

Fabric A Fabric B

Mezz Mezz

FEX AFEX A FEX BFEX B

SAN A SAN B

Port-Channel Port-Channel

UCS 2104 (1st Gen FEX) — Server to Fabric Pinning

Fabric Interconnect1 link

2 links

4 links

Server slots pinned to uplink

Uplink: slots 1,2,3,4,5,6,7,8

Uplink 1: slots 1,3,5,7Uplink 2: slots 2,4,6,8

Uplink 1: slots 1,5Uplink 2: slots 2,6Uplink 3: slots 3,7Uplink 4: slots 4,8

slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8

Fabric Interconnect

1st Generation Hardware

Fabric A Fabric B

IOM AIOM A IOM BIOM B

2nd Generation Hardware

Fabric A Fabric B

Mezz Mezz

SAN A SAN B

UCS 2204 (2nd Gen FEX) — Sever to Fabric Pinning

2 links

4 links

Uplink: slots 1,2,3,4,5,6,7,8

Fabric Interconnect

UCS 2204 (2nd Gen FEX) — Sever to Fabric Pinning

4 links

Server slots channeled acrossall uplinks

Uplink 1: slots 1-8Uplink 2: slots 1-8Uplink 3: slots 1-8Uplink 4: slots 1-8

Fabric Interconnect

UCS 2208 (2nd Gen FEX) — Server to Fabric Pinning

2 links

4 links

Uplink: slots 1,2,3,4,5,6,7,8

Fabric Interconnect

UCS 2208 (2nd Gen FEX) — Server to Fabric PinningServer slots pinned to uplink

8 linksUplink 1: slot 1Uplink 2: slot 2Uplink 3: slot 3Uplink 4: slot 4Uplink 5: slot 5Uplink 6: slot 6Uplink 7: slot 7Uplink 8: slot 8

Fabric Interconnect

8 links Uplink 1: slots 1-8Uplink 2: slots 1-8Uplink 3: slots 1-8Uplink 4: slots 1-8Uplink 5: slots 1-8Uplink 6: slots 1-8Uplink 7: slots 1-8Uplink 8: slots 1-8

Fabric Interconnect

Server slots channeled acrossall uplinks

2nd Generation Fabric Interconnect

Fabric A Fabric B

Network Statistics in UCSM

UCS CLI

DMEUCSM AG blade

Events {xml}

HTTP{xml}

Polling/notification

XML API

CIM Object Translation

IBM Tivoli

Syslog

AG FI AG chassis

SNMPCIM-XML

HPNNM HPOM Misc. SNMP

Events{SNMPTraps}

Polls{SNMPGET}

UCSM Core

5 most recent stats history records are contained by the current stats instance.

Stats history are subclasses of the stats.

topSyssys

equipmentChassissys/chassis-1

equipmentChassisStatssys/chassis-1/stats

equipmentChassisStatsHistsys/chassis-1/stats/1

... equipmentChassisStatsHistsys/chassis-1/stats/5

Statistics are contained by monitored object

Statistics are instances of statsCurr

Statistics dn is often deducible.

UCSM Statistics

Stats Collection Policy

A Systemwide Set of Stats Collection Policies (One per Application Domain) Allow Configuration of Collection and Reporting Intervals.Domains: adapter, chassis, host, port, serverWhen a Collection Interval Is Changed, the reportingInterval Is Restarted collectionInterval: frequency that endpoints will send stats updates to DME reportingInterval: frequency that DME will report stats updates to external

collectors and update stats history

Network statistics are collected by UCSM from the NX-OS software in the Fabric Interconnects. These are counters that are available for networking components Because of NIV technology the Fabric has visibility to the Cloud (LAN/SAN

uplinks), the IOM and the server NIC.

Access Statistics through:– LAN or SAN tab (port-group) – Devices tab ( server ports, network uplink, storage ports, and mezz ports,)– Server Tab (vNIC)

FI to LAN Network Uplink (Cloud)

Fabric A Fabric B

Port Channel is the Aggregate of all interfaces in the channel Statistic for the channel are the sum of the statistics of the members Individual member statistics are also visible in the system Network usage is measured against TX Total bytes and RX Total bytes

Network Statistics in UCSMFI to Uplink Port Channel Statistics

Network Statistics in UCSMFI to Uplink Individual Port Statistics

Network Statistics in UCSMFI to SAN Port Statistics

Fabric A Fabric B

SAN A SAN B

A FC Port Channel is the Aggregate of all interfaces in the channel

Statistic for the channel are the sum of the statistics of the members

Individual member statistics are also visible in the system

FC usage is measured against Bytes RX and Bytes TX

FI to SAN Uplink (Cloud) Port Statistics

Network Statistics in UCSMFI to SAN Port Statistics

Network Statistics in UCSMFI to IOM (Internal LAN)

Fabric A Fabric B

Network Statistics in UCSMFI to IOM (Internal LAN)

UCS Networking OverviewServer to IOM

Network Statistics in UCSMServer to IOM vNIC Port Statistics

Using SNMP tools to Monitor Traffic

The network statistics in UCSM are pulled from the NX-OS statistics for the networking components.

It is possible to monitor the traffic for the networking components using an external SNMP collection tool such as MRTG.

The advantage is long term analysis, however most tools have no alerting capabilities.

Understanding Collection Policies

A collection Policy consist of a collection interval and a reporting interval The collection policy is set under the admin tab in Stats Management ->

Collection Policies -> Collection Policy name A unique policy can be set for, Adapters, Chassis, FEX, Port, Server, and Host* Not all policies involve network components

*Host is an unused policy in UCSM at present

The collection interval is how often the system will query a device for statistics.

The default collection interval is 60 seconds

The more frequent the interval the more granular the data. We will use 30 seconds.

The timing of the collection interval is important because it will be used in BW calculations for hotspot detection

Collection Interval

The reporting interval is internal to UCSM and determines how often UCSM will store data from the collection interval.

This data is stored in tables and the last 5 reporting intervals are available for inspection in the system

Reporting interval data is used to calculate minimum, maximum and average values shown in the statics view.

Reporting Interval

Select the Policy you want to change. Make selections and press the save changes button

Hotspot Detection

Hotspot Areas

Fabric Interconnects to LAN/SAN

FEX to Host

FEX to FI

Fabric A Fabric B

Mezz Mezz

FEX AFEX A FEX BFEX B

SAN A SAN B

There are three potential hotspot locations for UCS network connectivity.

1.) Send and Receive between Fabric Interconnect and LAN/SAN

2.) Send and Receive between FEX and Fabric Interconnect

3.) Send and Receive between Host and FEX

Hotspot Detection

To identify hot spots we will use Threshold policies in conjunction with collection policies to alert as we pass thresholds.

A threshold is calculated by measuring a statistic against a policy.

The policy measures change against a user defined normal value and turns on the alert between a users set high/low threshold and turns off the alert below the user set low threshold.

Threshold Policies

High - Up

Low - Up

Normal

High - Up

Low - Up

Normal

High - Up

Low - Up

Normal

Hotspot Detection

The statistic we use to calculate bandwidth is the delta in bytes.

This should be measured for both TX and RX

This delta is calculated in bytes changed over a period of time defined by the collection interval, for example 30 seconds

Calculating BW Threshold Limits for an Element

‟… if standardizing on one size was a good thing, standardizing on several sizes would be even better.”

• Radia Perlman• Interconnections: Bridges and Routers

Kilo, Mega, etc

What is a Kilobit?– “Once upon a time… Computer professionals noticed that 210 was very nearly equal to 1000

and started using the SI prefix “kilo” to mean 1024.– The rest of the world uses the term Kilo in the SI format to mean 1000.– The result is that no one seems to know what kilo and mega mean.

The IEEE standards board decided that the IEEE standards will use the conventional, internationally adopted, definitions of the SI prefixes. Therefore Mega will mean 1,000,000 and Kilo will mean 1,000.*

In Telecom and by extension networking Kilo means 1000 and NOT 1024

The SNMP reported speed for a 10Gig link (on any platform) will be reported by SNMP is 1,250,000,000 – Bytes (MaxBytes[10.29.148.6_436207616]: 1250000000)

*Sources: NIST and IEEE

IEEE Networking Standards for Prefix Values

Hotspot Detection

For Ethernet the BW of a single link is 10Gbps

First we determine our desired threshold for example 8Gbps

We need to calculate the expected change in bytes over the collection interval.

To calculate divide the desired BW by 8 bits per byte and then multiply by the collection interval by the time to get the expected delta in bytes for our collection period.

Example 8 Gbps over 30 seconds = 30,000,000,000 bytes– 8Gbps / 8bits per byte = 1,000,000,000 bytes per second – 1,000,000,000 bytes per second * 30 seconds = 30,000,000,000 bytes

Calculating BW Threshold Limits for an Element

Hotspot DetectionThreshold Calculations

Speed in Gbps Percentage of BW Conversion to BytesDelta expected over 30 second collection interval

10 100% 1,250,000,000 37,500,000,000

9 90% 1,125,000,000 33,750,000,000

8.5 85% 1,062,500,000 31,875,000,000

8 80% 1,000,000,000 30,000,000,000

7.5 75% 937,500,000 28,125,000,000

7 70% 875,000,000 26,250,000,000

6.5 65% 812,500,000 24,375,000,000

6 60% 750,000,000 22,500,000,000

5 50% 625,000,000 18,750,000,000

4 40% 500,000,000 15,000,000,000

3 30% 375,000,000 11,250,000,000

Hotspot DetectionThreshold Alerts

Hotspot Detection

Threshold policies can be configured for the following:– Internal LAN – IOM to FI– LAN Cloud – FI to Upstream Ethernet switches– SAN Cloud – FI to Upstream SAN switches– Server – Between the server NIC and the IOM

Threshold Policies Placement

Hotspot Detection

Navigate to admin->stats management and expand the fabric.

Select thr-policy-default and create a threshold class

Choose Ether Tx Stats from the stat class and click next.

Threshold Policies Internal LAN

Hotspot DetectionThreshold Policies Internal LAN

Hotspot Detection

Click the add button and select Ether Tx Stats Total Bytes Delta as the property type enter 0.0 as the normal value

Select the Alarm triggers you want to get and enter your values and click OK

Click finish to be returned to the policy.

Click the classes tab to see your policy.

Threshold Policies

Hotspot Detection

You will need to add another class for RX traffic.

Click the + bottom to the right and repeat the steps from the previous policy choosing Eter Rx Stats as the stats class this time

Click save changes once you have completed the steps

Threshold Policies

Hotspot DetectionThreshold Policies

Hotspot Detection

For Uplinks you can repeat this process for the LAN cloud.

For SAN use a single Stats class fcstats and create a definition for rx and tx stas under the same stats class

Threshold Policies

Hotspot Detection

For the vNIC port you will need to create a threshold policy to be used with a service profile.

Go the the appropriate organization level and select create threshold policy

Threshold Policies

Hotspot Detection

Give the policy an applicable name and description and press next

Choose the vnic stats class and create a single threshold with the rx bytes delta and the tx bytes delta.

Threshold Policies

Hotspot Detection

Apply the threshold policy to the Service profile of the servers you want to monitor

Threshold Policies

Hotspot Detection

When a server reaches a threshold you will receive an alert on UCSM.

This will exist while threshold is exceeded once it drops below the definition the alert will disappear

Alerts show as system faults as defined by the threshold policy

UCSM will send a TRAP when an alert is generated.

The TRAP will be sent regardless of what level of alert has been set.

Threshold Policies

Hotspot Detection

When a threshold policy has been crossed UCSM will send out an SNMP Trap alert

This assumes that UCSM has been configured for SNMP operation and the Trap Receiver has been defined

This feature was added in UCSM version 2.x.x

Hotspot Detection

UCSM has a limited amount of space to store statistics and can only provide the last 5 instances of collection intervals. It provides a good tool for monitoring thresholds, but does not provide a long term monitoring solution.

Tools like MRTG are great for Long Term analysis of BW usage, but may not provide the required alerting and usually require a great deal of customization to be usable

UCS Central version (TBD) will provide tools for both alerting as well as long term analysis of traffic within systems it manages.

Long Term Monitoring vs. Point in Time

Automating Threshold Policies

Creating threshold policies for the systems require time and effort. Isn’t there some way to push standard configurations to the system?

GoUCS is an automation tool available from the Cisco Developer Network (CDN) that allows users to build XML configuration scripts that can be sent to UCS.

PowerTool can also be used to automate this process

http://developer.cisco.com/

GoUCS and CDN

Engineering to Avoid Hotspots

Traffic Engineering

Class-A Class-B Class-C

vNIC-1

vNIC-2

vNIC-3

Blade-2, VIC-1vNIC-1

vNIC-2

vNIC-3

VIC with 3 vNICs

2 Fabric Extenders in chassis, each with 1 link to the FI.

2 FI, both with 1 connection to each FEX

Blade-1, VIC-1

vNICs can be pinned to a specific FI when created (with configurable failover to other switch)

Depending on requirements, vNICs could be pinned to one interconnect or distributed evenly

vNICs in System Class C pinned to one interconnectvNICs in System Class C pinned to one interconnect

Controlling Pinning in Profile

From the each server a fabric interconnect can be chosen to balance the traffic

Traffic Engineering

Class-A Class-B Class-C

vNIC-1

vNIC-2

vNIC-3

Blade-2, VIC-1vNIC-1

vNIC-2

vNIC-3

Blade-1, VIC-1

vNICs can be pinned to a specific FI when created (with configurable failover to other switch)

Depending on requirements, vNICs could be pinned to one interconnect or distributed evenly

vNICs in System Class C distributed across interconnectsvNICs in System Class C distributed across interconnects

QoS Architecture

No packet drops within the array– Largest buffers are on switch and

host memory, so congestion pushed to edges

– Priority Flow Control (PFC) used to ensure packet drops are at vNIC or Switch

All traffic in a CA system belongs to 1 of 6 System Classes

– Four are user configurable while the other two are for FCoE and standard Ethernet

QoS parameters can be configured at a per system class level, or a per vNIClevel.

Compute ChassisCompute Chassis

erx86 Computer x86 ComputerX

IIx8x8x8x8

X X X X X

AAGG GG

PPMM PP

SANLAN

FabricSwitchFabricSwitch

FabricExtender

FabricExtenderFabricExtender

Compute Blade(Half slot)

AdapterAdapter

Compute Blade(Full slot)

AdapterAdapterAdapterAdapter

System Buffering/Queuing

User Configuration

Globally for each System Class

COS value for packets in this class

Drop/No-drop behavior

Strict PriorityStrict Priority

Bandwidth/WeightBandwidth/Weight

Users configure QoS parameters at two levelsUsers configure QoS parameters at two levels

Class Name FC Bronze

COSValue 3 0

Drop/No-Drop No-Drop Drop

Strict Priority No No

Bandwidth/Weight 20% 30%

Example

User Configuration

For each vNIC (Egress properties)

System Class for traffic from this vNIC

Rate limit (Mbps)Rate limit (Mbps)

Burst Size (Kbytes)Burst Size (Kbytes)

Users configure QoS parameters at two levelsUsers configure QoS parameters at two levels

vNIC1 vNIC2 vNIC3

Class FC FC Bronze

Rate 4000 4000 5000

Burst 300 400 100

Example: Logical Server A

User Configuration – Example

Class Name FC Gold Ethernet BE

COS Value 3 1 0

Drop/No-Drop No-Drop Drop Drop

Strict Priority No No No

Bandwidth/Weight 1 (20%) 3 (60%) 1 (20%)

vNIC1 vNIC2 vNIC3

Class FC FC Eth. BE

Rate 4000 4000 5000

Burst 300 400 100

Logical Server A

Global System Class Definitions

vNIC1 vNIC2

Class Gold Eth. BE

Rate 600 4000

Burst 100 300

Logical Server B

FC TrafficHigh PriorityEthernet

Best EffortEthernet

QoS Tools

Priority Flow ControlPriority Flow Control

• Enables lossless Fabrics for each class of service

• PAUSE sent per virtual lane when buffers limit exceeded

Transmit Queues Ethernet Link Receive Buffers

EightVirtualLanes

OneOne OneOneTwoTwo TwoTwo

ThreeThree ThreeThreeFourFour FourFourFiveFive FiveFive

SevenSeven SevenSevenEightEight EightEight

SixSix SixSixSTOP PAUSE

COS based Bandwidth ManagementCOS based Bandwidth Management

• Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth

•802.1Qaz Enhanced Transmission

10 GE Link Realized Traffic Utilization

3G/s HPC Traffic3G/s

3G/sStorage Traffic3G/s

LAN Traffic4G/s

5G/s3G/s

t1 t2 t3

Offered Traffic

t1 t2 t3

3G/s 3G/s

3G/s 3G/s 3G/s

3G/s 4G/s 6G/s

Among the tools used are aggregate shapers at the vNICs (VIC Adapter), ETS, Policers at the switch for each vNIC.

QoS Configuration in UCSMEnable QoS Classes in UCSM

QoS Configuration in UCSMCreate a QoS Policy

Applying QoS to a PolicyApply Policy to Adapter

Summary

UCSM is designed for optimized traffic flow Stats Management and Threshold policies allow you to monitor traffic levels QoS and Traffic engineering tools allow you to manage potential bottlenecks in

Maximize your Cisco Live experience with your free Cisco Live 365 account. Download session PDFs, view sessions on-demand and participate in live activities throughout the year. Click the Enter Cisco Live 365 button in your Cisco Live portal to log in.

Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Cisco Daily Challenge

points for each session evaluation you complete. Complete your session evaluation

online now through either the mobile app or internet kiosk stations.

traffic monitoring and engineering for...

Documents

de ucs ucs/m ucs/h ucs900 - eurapo · 2014-03-12 · ucs...

brkcom-1005 ucs architecture

figura1 - ucs · title: figura1 author: ucs created date:...

cisco unified computing system (ucs) storage · pdf...

ucs central for multi ucs environmentsucs-4786

ucs fabric fundamentals - alcatron.net live 2013...

cisco ucs administration and...

multi-ucs management with ucs central - alcatron.net live...

brkcom-3001 (1)

troubleshooting the cisco ucs compute deployment live 2013...

cisco intel ucs launch · c-series rack-mount servers ucs...

여러분의데이터센터를위한 차세대 cisco ucs...

ucs administration and...

ucs c-series deployment options, best - alcatron.net live...

ucs networking deep dive - alcatron.net live 2015...

fotografias - ucs · 2015. 10. 14. · fotografias...

ucs management deep dive - alcatron.net live 2014...

using advanced features on cisco...

zggh]hph>new.groteck.ru/images/catalog/40429/9deec18637ee6ecbb5a6ae0a1… ·...

cisco ucs to brocade connectivity guide...cisco ucs, and...