the aruba tech support top 10: wlan design, configuration and troubleshooting tips

#ATM15 |

The Aruba Tech Support Top 10 TipsTarun George & Gowri Amujuri

March 2015

@ArubaNetworks

CONFIDENTIAL © Copyright 2015. Aruba Networks, Inc. All rights reserved2#ATM15 |

WLAN Design, Configuration and Troubleshooting

Tips by TAC

@ArubaNetworks

3 CONFIDENTIAL © Copyright 2015. Aruba Networks, Inc. All rights reserved#ATM15 |

• Segmental Troubleshooting

• AP Stability and System profile Optimization

• Optimize load on processes

• Datapath Debugging

• Deployment Tips

Aruba OS

@ArubaNetworks


Transition Content

#1

Segmental Troubleshooting

@ArubaNetworks


Transition Content


Segmental Troubleshooting gains.

# Faster root cause analysis.

# One time Data Collection

# Bring focus on the smallest segment in the network within our control.

@ArubaNetworks


Transition Content


Where do we start if we are unsure of the exact cause of the current issue being faced?

@ArubaNetworks


Transition Content


Usershow tech-support user mac <Mac Address>

tar logs user mac <User Mac > tech-support

User Debugging

Logging Level debugging user-debug <Mac Address>

@ArubaNetworks


Transition Content


AP

show ap tech-support ap-name <Name of AP>

show ap debug counters

show ap bss-table ap-name

show ap debug system-status ap-name

@ArubaNetworks


Transition Content


Controllershow tech-support

tar log tech-support

Outside world

Debugging for Specific process/Sub-cat.. (Explained)

Pcap

show interface gigabitethernet <slot/module/port>

Network Diagram

Note: Show tech-support <filename> Store output in file.

@ArubaNetworks


Transition Content

Segmental Troubleshooting.. Processes

show process monitor statistics

Process Monitor Statistics

Name State Restarts Allowed Restarts Timeout Value Timeout Chances Time Started

/mswitch/bin/dbstart PROCESS_RUNNING 8 0 240 3 Sat Feb 28 21:31:55 2015

/mswitch/bin/packet_filter PROCESS_RUNNING - 0 240 3 Sat Feb 28 21:31:56 2015

Mdns , httpd_wrap , Authmgr ,STM , WMS , cfgm , dhcp

@ArubaNetworks


#2

AP Stability and System Profile Optimization

@ArubaNetworks



AP System Status.

Campus and Remote APs have similar challenges to stay connected to the controller.

Health Check of the AP is vital, since it can trigger client and controller anomalies.

Show AP debug system-status ap-name <Name of AP>

@ArubaNetworks



• Reboot Information

DHCP/Controller/Keep Alive miss

• Rebootstrap Information

Date Time Reason (Latest 10)

LMS Change/Heartbeat Miss

• HA Failover Information

Date Time Reason (Latest 10)

@ArubaNetworks



• Recent Control Messages from AP to Controller

Date Time Message Description

Sun Mar 1 12:29:49 2015(164 secs ago): SENT REQ type=KEEPALIVE len=45 peer=10.163.196.72 seq_num=4567 num_attempts=1 rtt=0 secs

• Rebootstrap LMS

• Crash Information

@ArubaNetworks



• CPU and Memory UsageTimestamp CPU Util(%) Memory Util(%)

2015-03-01 12:32:27 2 24

• Peak CPU Util in the last one hourTimestamp CPU Util(%) Memory Util(%)

2015-03-01 12:19:25 3 24

@ArubaNetworks



Heartbeat Stats of Serving ControllerHeartbeats Sent Sent Seqnum Heartbeats Received Rcvd Seqnum MTUs sent Misc sent Measurement Duration

2690183 25824 2667575 25824 22607 0 since last rebootstrap

2690193 n/a 2667575 n/a 22607 0 total since bootup

Interface countersInterface Rx_pkts Rx_errors Rx drops Tx_pkts Tx_errors Tx_drops Resets

wifi0 3209433 16822381 2230363 236918 61 0 0

wifi1 4096977 2070224 4095468 2242763 58 0 11

@ArubaNetworks



MTU DiscoveryProbes Responses Last Sent Last Rcvd

45214 22607 2712890 2712890

Switch MTU, 1500

Ethernet bondingSlaveId Name Link State #LinkFails Ethernet Duplex/Speed Settings

Autoneg Speed (Mbps) Duplex Iface

0 eth0 UP ACTIVE 0 on 1000 Full eth0

eth1 DOWN STANDBY 0 on 10 Half eth1

@ArubaNetworks



Controller InformationItem Value

Primary LMS 10.163.196.72

Backup LMS 10.163.196.71

AP to Active Controller Message InformationItem Value

AP state REGISTERED

Power StatusOperational State : Unknown

Current HW State POE-AT: No restrictions

@ArubaNetworks



MTU DiscoveryProbes Responses Last Sent Last Rcvd

45214 22607 2712890 2712890

Switch MTU, 1500

Ethernet bondingSlaveId Name Link State #LinkFails Ethernet Duplex/Speed Settings

Autoneg Speed (Mbps) Duplex Iface

0 eth0 UP ACTIVE 0 on 1000 Full eth0

eth1 DOWN STANDBY 0 on 10 Half eth1

@ArubaNetworks


AP Stability Optimizations

@ArubaNetworks


AP Stability Optimizations

Heartbeat DSCP: Assign a DSCP value to AP heartbeats to prioritize heartbeats traveling over low-speed links. The

supported range is 0-63, and the default value is 0.

Bootstrap threshold: Number of consecutive missed heartbeats on a GRE tunnel (heartbeats are sent once per second on each tunnel) before an AP rebootstraps. On the controller, the GRE tunnel timeout is 1.5 x bootstrap-threshold; the tunnel is torn down after this number of seconds of inactivity on the tunnel.

SAP MTU: Maximum Transmission Unit, in bytes, on the wired link for the AP.

Spanning Tree: Select this checkbox to enable the Spanning Tree protocol.

@ArubaNetworks


# 3

Optimize load on processes

@ArubaNetworks


HTTPD

Stress on the webserver can be because of number of sessions in the initial role for guest access. This is either because of large certificate (Key Length 2048 or 4096) that are used by the server or a large number of devices (phones/Tablets with APPs) that generate HTTP/HTTPS sessions and get re-directed to the web-server.

show web-server profile

Web Server Configuration

Parameter Value

SSL/TLS Protocol Config tlsv1

Captive Portal Certificate GUEST-AUTH

User session timeout <30-3600> (seconds) 3600

Maximum supported concurrent clients <25-320> 75

Enable WebUI access on HTTPS port (443) true

Enable bypass captive portal landing page false

@ArubaNetworks


HTTPD

show web-server statistics

Web Server Statistics:

Current Request Rate: 1 Req/Sec

Current Traffic Rate: 1 KB/Sec

Busy Connection Slots: 7

Available Connection Slots: 68

Total Requests Since Up Time: 284

Total Traffic Since Up Time: 1122 KB Avg.

Request Rate Since Up Time: 1 Req/Sec Avg.

Traffic Rate Since Up Time: 6144 Bytes/Sec

Server Scoreboard: _____________KKKKKK_W_____________Scoreboard Key: _ - Waiting for Connection, s -

Starting up R - Reading Request, W - Sending Reply K - Keepalive, D - DNS Lookup C -Closing connection, L - Logging G - Gracefully finishing, I - Idle cleanup of worker . - Open slot with no current process

@ArubaNetworks


STM and WMS

STM

Station Management is responsible for all AP information and Station information. This process can be over run if there is,

# Aggressive polling from Airwave/SNMP servers for Wlan tables.

# Network wide AP reboot and bootstraps

# AP debug scripts run from the controller.

WMS

IDS/IPS events and frequent AP bootstraps could lead to WMS being busy. WMS is actively looking for RF information of WiFi devices(Rogue/Valid/Interfering).

@ArubaNetworks


Mitigation

Use AMON

WMS Offload to Airwave.

Reduce SNMP polling or increase the polling period

Disable WMS functionality if you do not require IDS/IPS functionality.

@ArubaNetworks


# 4

Datapath Debugging

@ArubaNetworks


Datapath Monitoring

Show datapath utilization

show datapath utilization

Datapath Network Processor Utilization

| Cpu utilization during past |

Cpu | 1 Sec 4 Secs 64 Secs |

10 | 99% | 99% | 99% |

11 | 0% | 0% | 0% |

12 | 0% | 0% | 0% |

13 | 0% | 0% | 0% |

14 | 0% | 0% | 0% |

15 | 0% | 0% | 0% |

16 | 0% | 0% | 0% |

show datapath frame 10

|SUM/| | | |

|CPU | Addr | Description Value |

+----+------+-----------------------------------------------------+

| 10 | [00] | Allocated Frames 1040|

| 10 | [01] | Max Allocated Frames 2208 |

| 10 | [03] | Unknown Unicast 147074970|

| 10 | [34] | Flood Frames 1506164167|

+----+------+-----------------------------------------------------+

| 10 | [00] | Rx Frames 635394472|

| 10 | [01] | Rx Bytes 1864525959|

| 10 | [02] | Tx Frames 1240985989|

+----+------+-----------------------------------------------------+

@ArubaNetworks


Datapath Bandwidth Management

show datapath bwm

Datapath Bandwidth Management Table Entries

Type Id Bits/sec Policed Bytes Bytes Flags CPU Status

---- ---- --------- ---------- ------- ----------- ------- ------- ------

0 1 20000000 0 78125 0/0 9 ALLOCATED

0 2 4000000 0 15625 0/0 9 ALLOCATED

0 3 160000000 0 624890 0/0 9 ALLOCATED

0 4 4000000 0 15625 0/0 9 ALLOCATED

0 5 2000128 0 7813 0/0 9 ALLOCATED

0 6 2000128 0 7813 0/0 9 ALLOCATED

0 7 2000128 0 7813 0/0 9 ALLOCATED

Firewall:

Rate limit CP untrusted ucast traffic Enabled 20 Mbps

Rate limit CP untrusted mcast traffic Enabled 4 Mbps

Rate limit CP trusted ucast traffic Enabled 160 Mbps

Rate limit CP trusted mcast traffic Enabled 4 Mbps

Rate limit CP route traffic Enabled 2 Mbps

Rate limit CP session mirror traffic Enabled 2 Mbps

Rate limit CP auth process traffic Enabled 2 Mbps

@ArubaNetworks


CP and DP Packet Capture

• Wifi -Clientpacket-capture datapath wifi-client aa:aa:aa:aa:aa:aa all

• VIA client/RAPpacket-capture datapath ipsec <peer-ip>

• Generic traffic to controllerpacket-capture controlpath tcp/udp 4343

@ArubaNetworks


# 5

@ArubaNetworks

Deployment Tips

Missing optimizations


Deployment Tips

Honey Comb Pattern

Wireless

Local Probe Threshold = 25

Transmit Power of AP 5Ghz Min Tx – 12 Max Tx – 15

2.4 Ghz Min Tx – 6 Max Tx – 9

Avoid Asymmetric RF

The difference between minimum and maximum Tx power on the same radio should not be more than 6dbm

DMO Enable

Basic and Beacon rate

802.11a 5Ghz – 24

802.11g 2.4Ghz – 12

80 Mhz Channel bonding - DFS Channels

@ArubaNetworks


Deployment Tips.. Contd

GRE Stripping IP - VRRP for LMS and Stripping IP

Jumbo Frames - Enabled

802.3at

Airgroup

Dot1x

OKC

Validate PMK ID

802.11r/k/v

EAPOL Rate Optimization

@ArubaNetworks


Deployment Tips… ASE

https://ase.arubanetworks.com/

@ArubaNetworks

https://ase.arubanetworks.com/


Deployment Tips..

ASE for troubleshooting

@ArubaNetworks

36#ATM15 |

Network ServicesAirWaveClearPass

@ArubaNetworks


Transition Content

# 6

ClearPass Platform: System Cleanup Options

@ArubaNetworks



• Free disk space threshold is a config in Cluster Wide Service Parameter. Default 30%

• A system cron job runs every hour and checks the disk utilization. If the free space falls below the configured threshold, an alert is logged into the system. NOW in addition, the following aggressive cron cleans up anything more than 1 day old in version 6.5 of CPPM

• Log database records

• Core files

• System load monitor files

• Application and system log files

• Auto and manual backup files

• Stored reports

• Expired guest accounts

• Audit records



We also introduced some new CLI commands

– Check on disk-space and memory usage - “show sysinfo”…

– system cleanup [# of days to retain] **This is an on-demand task



• Same command function also exist in the GUI

– Remember this is an on-demand task


# 7

@ArubaNetworks

ClearPass Platform : Graphite


• Graphite is a new reporting tool to compliment Insight in CPPM from 6.3 version.

• Graphite runs on every node irrespective of standalone or cluster and statistics can be viewedfrom any node.

• Performance monitoring Display is disabled by default and should be enabled manually and setaccess permission levels accordingly.

• To access Graphite data, use the URL https://<CPPM IP Address>/graphite



• Make sure Performance monitoring is enabled from GUI



• Setting up access to Graphite from CPPM UI



• We can allow or deny any networks to access Graphite for a node or cluster.

• Make sure stats collection is set true True under Service parameters.



Transition Content

# 8

ClearPass : Upgrade Utility Tool

@ArubaNetworks



• The Cluster Upgrade Tool is a simple user interface that automates the upgrade procedurefor a ClearPass cluster.

• When the upgrade is initiated, no manual actions are required until all selected nodes havebeen upgraded.

• The Upgrade Tool is not available while the publisher is rebooted and migrating theConfiguration Database.

• The Upgrade Tool will not detect nodes that were upgraded manually without the tool.

• If a configured standby publisher node was manually upgraded without the tool, the UpgradeTool will not restore the state of the standby publisher configuration.



• The Cluster Upgrade Tool is released as a patch update. It can be downloaded and installedeither through Policy Manager’s Software Updates portal, or from the Aruba Support portal.



• Log in to Policy Manager on the publisher node and go to Administration > Agents and Software >Updates > Software Updates.

• When the installation is complete, the Admin service will be restarted. You do not need to reboot.



• Before you begin the upgrade, the upgrade image must be present on the publisher node of the cluster.

• Download the upgrade image to the publisher under Software updates



• To monitor the progress of the other nodes in the cluster, wait until the database migration is complete and then log in to the tool again.

• Change the url to https://CPPM IP Address/upgrade

• We should see all the subscribers status that are in sync.

• The list of subscribers will be present and subscriber upgrades will go in parallel.

• ‘Start Upgrade’ to start the upgrade on the servers.



• Check the logs for each node by ‘View Logs’ next to each node and we can see the progress of patches and upgrades from publisher.


ClearPass : Upgrade Utility tool


# 9

@ArubaNetworks

AirWave – VisualRF Performance Tips


• Sometimes we see VisualRF takes long time to show up new AP’s to deploy on the floor plans.

• We can manually force VisualRF to poll the AP’s to get new AP or existing AP’s updated details.

• NO need to restart VisualRF to show up new AP’s.

• Change the url in AMP to https://<Airwave IP Address>/visualrf/poll_aps_now.xml



• In 8.x moved to HTML5 for VisualRF UI for faster UI interaction with backend.

• Whilst we are improving the features on the new UI, there are some features which were present in flash and not in HTML5.

• In VisualRF > Setup page, we can switch between HTML5 and flash so that we can take advantage of options present in both version

• Switching between HTML5 and Flash version is easy with below URL without refreshing VisualRF.

• Change the URL to

https://Airwave IP Address/site?campus_id=6c56c239-bfba-4d19-aeca-8ec5af68b725

from

https://Airwave IP Address/vrf?campus_id=6c56c239-bfba-4d19-aeca-8ec5af68b725



AirWave – VisualRF Performance tips


• We can change ‘vrf’ to ‘site’ on any page for VisualRF URL’s to switch to flash mode from HTML5 mode.

At times we see Heat maps not showing/updating properly in VisualRF

• We can resize the floor plan to same size so that the grid calculation happens and heat maps will be re-drawn.

• No need to restart VisualRF for heat maps to update.



• Unlock the floor plan and go to properties and ‘Measure’

Airwave – VisualRF Performance Tips


• Select the distance, click ‘OK’ and ‘Save’ without changing the distance, this will trigger floor plan to recalculate the heatmpas.

Airwave – VisualRF Performance Tips


Transition Content

# 10

Airwave – Tips for Data Retention Settings

@ArubaNetworks



• Data in AirWave is primarily stored in 2 formats:

• Postgres - an open source, relational SQL database. Usually, when you see data in tables, that data is stored in Postgres.

• RRD Files - used for storing data that's displayed in time-sequence graphs (i.e, client count over the last year, bandwidth used over the last month). There can be many thousands of RRD files on a single AirWave server. One benefit of RRD is that its files have a fixed size. As data is inserted to an RRD file (like by an AirWave monitoring process), it does not grow. A downside of this is that the file starts using storage space as soon as it is created.



• We can set data retention settings under AMP Setup > General page under the section ‘Historical Data Retention’.

• Client Association and VPN Session History. This setting has a bearing on how much history we can show in the association history on the client historical table and how much data can be included in the user session data.

• Its recommend to keep high because the data is useful


Airwave – Tips for Data Retention settings

• Inactive Client and VPN User Data. This setting determines how long we keep the information on every client that has ever connected to the network.

• This impacts how long we keep RRD files. Keeping it low can save disk space.



• Client data retention Interval : This influences how much historical data you can see for each client in the graphs, for example the signal quality, usage graphs on the client detail/diagnostic page.

• It's very important to keep this low, like in the 14-31 days range.

• This is especially important in public wi-fi deployments that will have lots of unique users.

• This setting controls what size RRD files are created to store per-user historical signal, usage, goodput, health and other metrics.

• Keeping it low doesn't impact device, group and folder-level monitoring, and it doesn't have any negative impact on reports. It only impacts the graphs on the client Detail and Diagnostic pages.



• By default Rogues are kept forever which will impact the overall system performance and for RAPIDS page load.

• It also impacts VisualRF for Rogue calculation if it has thousands of Rogue devices.

• This is especially important in public wi-fi deployments that have open SSID and lot of nearby devices are detected as Rogues

• Setting the value to low as 14 days will greatly help.



• Airwave by default has 20+ Reports which runs daily.

• Keep Reports that are needed for the environment and delete/disable the default reports.

• Report retention setting can be costly in high dense environments especially for disk space.

• This increase the nightly backup file size, nightly maintenance time and report generation time.

• Exporting the reports via .csv or .pdf or emailing them is a good option.

• Keeping the retention value will have the pickled client tables not to grow huge in size and makes report generation faster.


Questions

@ArubaNetworks

THANK YOU

75#ATM15 | @ArubaNetworks

the aruba tech support top 10: wlan design, configuration and troubleshooting tips

Technology

aruba networks

networks mobility

aruba tech support

ap techsupport ap

troubleshooting tips

atm15 wlan design

techsupport store output

filter process