Network Monitoring Session
NSRC-GARNET-KENET September 14th 2010
Kevin G. ChegeKENET
Main goal – To install and run Nagios, Cacti, iPerf and Smokeping on our Ubuntu PCs
To understand network monitoring and the various terminologies
Understand linux and networking basics Understand monitoring tools Mixture of labs and slides but more hands on –
70% especially the afternoon session
Some Guidelines
We will use basic username and password settings Do NOT do this in the real world on production
servers! We will use mostly default configurations and basic
configs to get the tools running By the end of the day, you should be able to
configure the tools with basic configs but there is a lot more that can be done,
Google is your friend! Method to install found on most howtos on the net
What is Linux/Unix OS and where did it come from
UNIX is an operating system which was first developed in the 1960s
It is a stable, multi-user, multi-tasking system for servers, desktops and laptops
GNU Linux is a free Unix-type operating system originally created by Linus Torvalds
Source code for Linux is freely available to everyone GNU Project (1983) began working on a free
operating system called GNU which in 1992 was completed by combining it with the Linux kernel hence name GNU LINUX
History Diagram
Types of Linux/Unix whats the difference
Unix and Linux are similar but not the same Unix based OS remained true to the original
Unix Linux based OS are those originating from
GNU/Linux of 1992 and have the same kernel which is continually developed and updated
Unix OS - *BSD HP/UX AIX Solaris Linux – Debian RedHat Ubuntu Suse
However, they all have....
Kernel - the hub/heart of the operating system which allocates time and memory to programs
Shell – an interface to allow human interaction with the kernel, we can input commands to be processed
All use / (not like Windows \) Many commands on Linux and Unix are similar All commands are lower case and case sensitive!
Like for like
Task Proprietary Open Source
Operating System Ms Windows (Work Station and Server), Novell Netware, MacOS, Sun Solaris
Linux (Redhat, Slackware, Mandrake, Gentoo, SuSe), BSD (FreeBSD, OpenBSD, NetBSD)
Like for like
Task Proprietary Open Source
File Sharing/ Domain Controller
Microsoft Windows 2003 Server
Samba running on Linux or BSD
E-mail Server Microsoft Exchange Server, GroupWise, MDaemon
Qmail, Exim, Sendmail, Postfix
DNS Server Microsoft DNS Server, BIND, djbdns
Web Server Microsoft IIS Apache
Database Management System (DBMS)
Microsoft SQL Server, Oracle
My SQL, PostgreSQL
Task Proprietary Open Source
Office Suite MS Office, Lotus Notes/Domino, WordPerfect Suite, Novell Office Suite
OpenOffice, KOffice, Abiword, Gnumeric
Financial/Accounting Systems Quickbooks, Blackbaud, Sage, Peachtree, Pastel
GNUCash, SQL-Ledger, Appgen
Web Design Front Page, Dreamweaver
OpenOffice Web, Bluefish, Mozilla, Quanta Plus
Grapics/Desktop Publishing
Photoshop, Corel Draw, Fireworks Indesign, Quark
The Gimp, Scribus
Scripting/Development Languages
VbScript, ColdFusion PHP/Perl/Python
Project Management
Visio, MS Project
MrProject, KOrganizer
E-mail Client Ms Outlook, Outlook Express, CCMail,
Evolution, Kmail, Balsa, Ximian
Web Browser Internet Explorer Netscape, Mozilla, Konqueror, Opera
Why are we using Ubuntu
Ubuntu is based on GNU/Linux Easy to use – very user friendly Supported Everything just works! Very suitable for the desktop Very good for users fresh from Windows
environment Well documented
Some must know tips on Ubuntu
The Shell
Click Applications Go to accessories Click on Terminal
Directory Structure
Basic commands
Find out where you are on the shell, type 'pwd' in the shell
Create a directory in your home folder – type 'mkdir myfolder' in the shell
Create a file in your folder – type 'touch myfile' in the shell
List the contents in your directory – type 'ls' in the shell. Get more details with 'ls -l'
Add some content to the file myfile – 'echo “Winneba is near the beach” > myfile'
Display the contents of my file 'less myfile'
Transfer a file
We want to copy a file from the instructor's pc to your pc. Type the following in your shell:
sudo scp [email protected]:/home/garnet/90-apt-proxy.conf /etc/apt/apt.conf.d/
The root user, rights and sudo
Root user 'root' is the superuser with full rights To become root – 'sudo su' You know you are root when you see '#' at the
shell You do not always need to be root – you can
use sudo to temporarily use root rights A safer option We will mostly use sudo
Know your rights!
Rights are rwx R = read or 4 W = write or 2 X = execute or 1 Do the following on your shell:
'ls -l' and lets discuss the rights
Lets install vim
Vim is a shell based text editor Very powerful Improvement of the original vi. Vim stands from
Vi Improved We will use it to edit text files on the shell Lets install it! sudo apt-get install vim
Network Monitoring and Management
What is Network Monitoring? It is the use of network analysis tools to accurately
determine traffic flows and network utilization in a LAN or WAN network
The idea is to get an accurate picture of what is going on in the network
Why should I monitor the network?
Justification of network budget and resources
Detection of network intruders
Detection of network viruses
Troubleshooting of network problems – e.g. device failures
Optimization of network performance
Capacity planning
Ensuring proper use of a network
How to monitor the network
This is done by using a variety of monitoring programs available on the Internet.
The program you select should be:• Affordable
• Lightweight
• Able to display graphically
• Versatile
• Supported
• Data retentive
• User friendly and feature full for example able to be viewed over the web
Tools: open source vs proprietary
There are excellent open source and free tools Similarly, there are excellent proprietary tools Select the one appropriate to your needs Some of free ones also run on Windows if you
must maintain a Windows environment There are some proprietary (paid for) ones that
have a free or a trial version that has all the features of the full program but maybe only support one device
Can you survive with free ones?
Yes, there are some powerful well supported and documented tools out there
Unless you are running a huge production NoC, you do not need to purchase monitoring software
If you need an enterprise solution, there are free tools for this as well eg Zenoss, OpenNMS and Zabbix etc
Network Definitions and Metrics
Common measurements
bps – bits per second (small b) Remember 1 byte = 8 bits Bytes not used to measure transmission speed Kbps or kbps – kilo bits per second (small b) Mbps or mbps – mega bits per second (small b) Pps – Packets per second RTT – Round trip time Jitter – Variation on the Round Trip Time
Channel capacity and utilization
The maximum number of bits that can be transmitted for a unit of time (eg: bits per second)
Depends on: Bandwidth of physical medium Channel encoding and compression Efficiency of algorithms in use to access medium
Effective Channel capacity
Always a fraction of the nominal channel capacity
Dependent on: Additional overheads on each layer Device limitations on both ends Other factors
95th Percentile
The smallest value that is larger than 95% of the values in a given sample
The 95th percentile is a widely used mathematical calculation to evaluate regular and sustained utilization of a network pipe
The 95th percentile is a good value to use to show the bandwidth that is actually used at least 95% of the
Gives you an idea of the standard, sustained channel utilization
When do you upgrade your bandwidth?
The use of the 95th percentile will help you answer this
If a graph constantly indicates that network traffic is at peak for more than 95% of the time that is an indicator
that it is time to upgrade your networking capacity. The 95th percentile is a widely used mathematical
calculation to evaluate regular and sustained utilization of a network pipe
The 95th percentile is a good value to use to show the bandwidth that is actually used at least 95% of the
time.
Sample graph
945.40 kbpsOut:268.47 kbpsIn: 95th Percentile for last 24 hours:
45.34 GbitsOut:9.62 GbitsIn: Total over last 24 hours:
Packet Loss
Occurs when packets traversing a network fail to reach their destination
Can be caused by congestion, distance between transmitter and receiver, faulty medium etc
Usually expressed in percentage, ie if 100 packets are sent and only 90% make it correctly, then there is a 10% packet loss
Packet loss may means that communication between links is not optimal
Packet loss cont'd
However, not all packet loss means there is a problem
Some routers discard or give icmp packets less priority at peak times
Some firewalls just drop icmp in some cases
Break!
iPerf
Tool that measures throughput between two network points
Very useful in determining effective channel capacity
Can be run over any network Needs a client and a server where the client
connects to the server and the throughput between the two determined
Iperf lab
Iperf should be installed on your Ubuntu OS by default if not do 'sudo apt-get install iperf' at the shell
Find out – 'which iperf' Connect to instructor's iperf server:
iperf -c x.y.x.a Lets test iPerf on a wireless interface
SNMP
SNMP Concepts
SNMP – Simple Network Management Protocol Industry standard, hundreds of tools exist to exploit it Present on any decent network equipment
Query – response based: GET / SET GET is mostly used for monitoring Tree hierarchy Query for ”Object Identifiers” (OIDs) Concept of MIBs (Management Information Base) Standard and vendor-specific (Enterprise)
SNMP Version
UDP protocol, port 161 Different versions
- V1 (1988) – RFC1155, RFC1156, RFC1157Original specification
- v2 – RFC1901 ... RFC1908 + RFC2578 Extends v1, new data types, better retrieval methods
(GETBULK) Used is version v2c (without security model)
- v3 – RFC3411 ... RFC3418 (w/security) Typically we use SNMPv2 (v2c)
SNMP read only and the community name
To get SNMP going on a machine to be monitored, you need to set an SNMP community name
An SNMP community is the group that devices and management stations running SNMP belong to
It helps define where information is sent. A device or agent can have several communities
Typically, most devices come with an SNMP community name public
This can and should be changed
How SNMP Works
Terminology: Manager (the monitoring ”client”) Agent (running on the equipment/server)
GetRequest - The manager requests information from the agent. GetNextRequest - This is used to get information that is
contained by an array. GetResponse - Used by the agent to satisfy a request sent by
the manager. SetRequest - The manager changes the value of an agent's
parameter. Trap - A command or message used by the agent to inform the
manager of a certain event.
SNMP Labs
Configuring SNMP on a switch Installing SNMP on our Ubuntu machines sudo apt-get install snmpd SNMP Walk
Firewalls
Whats a firewall
System or network designed to restrict access to certain devices or applications
A security device frequently used to prevent unauthorized
Internet users from accessing private networks connected to the Internet
There are hardware based firewalls and software based firewalls
Types of firewalls
Packet filters – inspect traffic coming in and make decisions whether to allow or drop connections (network layer firewall)
Proxy servers – intercept traffic and apply rules to traffic. Hide/protect identity of LAN from WAN (application layer firewall)
Linux – ipTables and Squid Unix – PF, IPFW, IPF and Squid Vendor – Cisco ASA, Sonicwall
ACLs
Access Control Lists Available in Cisco IOS Record that identifies and manages traffic. After
identifying that traffic, an administrator can specify various events that can happen to that traffic.
Two types of IP ACLs: standard and extended Standard IP ACLs: 1 to 99 Extended IP ACLs: 100 to 199
ACLs cont'd
You can use an ACLs to match a certain type of traffic and then configure the router to perform specific rules on the identified traffic
Example Access-list 1 permit any Access-list 1 deny host 1.2.3.4
Can be used to create elaborate firewall rules Very useful especially when controlling access
to the router or SNMP
Sample Cisco config for SNMP
snmp-server community public RO 10 access-list 10 permit 192.168.16.12 access-list 10 deny any
Ping and ICMP
Ping is a utility used to test the reachability of a host
Also used to measure RTT Ping operates by sending Internet Control
Message Protocol (ICMP) echo request packets to the target host and waiting for an ICMP response
authored by Mike Muuss in December 1983 Utility present on most networking devices
ICMP Types
From IANA http://www.iana.org/assignments/icmp-parameters: Type Name Reference ---- ------------------------- --------- 0 Echo Reply [RFC792] 1 Unassigned [JBP] 2 Unassigned [JBP] 3 Destination Unreachable [RFC792] 4 Source Quench [RFC792] 5 Redirect [RFC792] 6 Alternate Host Address [JBP] 7 Unassigned [JBP] 8 Echo [RFC792] 9 Router Advertisement [RFC1256]
What to allow and when
To send ping requests, allow type 8 OUT and type 0 IN.
To respond to ping requests, allow type 8 IN and type 0 OUT.
To request a traceroute, allow type 11 IN. To respond to a traceroute, allow type 11 OUT.
Allow ICMP?
Yes, many monitoring tools require a response to an echo request
You can configure your network to allow the ICMP types that you need and discard the ones you dont
You can also throttle ICMP if need be ie limit ICMP response to no more than 64Kbps for example
How to configure this ranges per device and operating system
The BIG THREE
Cacti, Nagios, Smokeping
Cacti – Measures link utilization Can also measure:
Errors Disk Usage CPU and Memory usage Many others
Web based gui, many features Nagios – Measures up times
Also measures Latency Availability Can give warnings/alerts
Web based Gui
Smokeping
Measures latency Also measures
Uptime Loss RTT
Sends alerts Has a web interface Directory - /etc/smokeping
Another nice tool: BandwidthD
BandwidthD tracks usage of TCP/IP network subnets and builds html files with graphs to display utilization
Easy to use Works on all OSes even Windows Displays top 20 talkers on the network And what they are talking about
LAB TIME!