Nagios at Funet

Teemu Kiviniemi, CSC/Funet

6th June 2012

6th TF-NOC meeting

Dublin, Ireland

Introduction

Funet uses Nagios extensively for

monitoring.

– network

– servers

– services

Two Nagios monitoring servers

– Over 900 monitored hosts

– Over 10000 monitored services

2

Nagios at Funet NOC

NOC follows the (combined) hostgroup

and servicegroup summaries

– Traditional and iPad versions are available.

NOC receives SMS and/or e-mail alerts

about critical services.

NOC opens a ticket about each problem.

Problems are acknowledged in Nagios

with the ticket number.

Nagios scheduled downtime is set before

maintenance. 3

NOC monitoring levels

We have four different monitoring urgency

levels for our services.

Monitoring levels have different reaction

time requirements:

– 30 minutes, 4 hours, NBD, best effort

Also the operative processes and the

documentation available to NOC must be

better in services at higher monitoring

levels.

4

NOC monitoring levels (continued)

Monitoring levels have different notification

options.

Services at the highest monitoring level

trigger SMS alerts to NOC immediately.

No e-mail or SMS alerts are sent about

best effort services.

Nagios host and service escalations are

defined to escalate longer service

disruptions to managers.

5

Nagios configuration management

Nagios configuration is split to several

directories and files.

Some configuration is identical between

the two monitoring servers.

Configuration files are in Subversion VCS.

Service administrators configure service

checks mostly on their own, following the

agreed guidelines.

6

Automatically generated

configuration

Large parts of Nagios configuration are

generated automatically.

– Linux servers, routers, DWDM, switches, DNS

zones.

Configuration is generated with Perl

scripts, and Nagios is updated

automatically.

Linux server administrators can customize

some aspects of the generated

configuration. 7

Custom check plugins

We have written a lot of custom check

plugins for our monitoring needs.

A total of 85 custom Nagios check plugins

are enabled in our current configuration.

Examples:

– BGP route status and other router/switch

SNMP checks

– IPv6 transition mechanisms

– DNS zone SOA reachability

– RRD statistics 8

Reporting

We plot Nagios performance data using

pnp4nagios.

For all other reporting we use Nagios-

Surfer – a tool developed at Funet.

9

How Nagios-Surfer works

10

Nagios configuration overview

reports

Generated by Nagios-Surfer for all hosts,

services, contacts, and groups.

Reports contain information about

– Service checks - What is monitored and how?

– Notifications - Who received notifications and

when?

– Configuration differences – What differences

are there between the monitoring configuration

of hosts or services in the same group.

11

Nagios configuration overview

reports

12

Nagios availability reports

Nagios-Surfer generates availability

reports of all hosts, services, contacts and

groups.

Availability reports are pregenerated.

– Unlike Nagios avail.cgi which reads through

the event log each time a report is requested.

– We get 1.5GB of event log per month.

Availability numbers are reported per-

month.

13

Nagios availability reports

14

Nagios event log reports

Nagios-Surfer generates monthly event log

summaries of all hosts and services.

– Redundant information, such as duplicate and

subsequent OK lines are removed.

Each break contains a link to detailed

information about the break.

Event logs can be accessed easily through

the availability reports.

15

Nagios event log reports

16

Nagios and quality assurance

We have internal quality assurance

processes that oversee that services meet

the set reliability requirements.

Service administrators investigate new

service breaks and save the information to

Nagios-Surfer.

– A quality assurance process can use the data

to concentrate on the most relevant issues.

17

Archiving information about breaks

Information about the causes of all breaks

is archived with Nagios-Surfer.

Investigating old issues becomes easier,

as the breaks of possible service

dependencies are visible.

Makes it easier to notice patterns.

18

Gathering detailed information about

Nagios breaks

Nagios-Surfer sends break clarification

requests to administrators by e-mail.

Administrators can categorize and

describe breaks. The information is saved

to Nagios-Surfer database for later use.

If a break is categorized as scheduled

downtime, the change will be reflected in

the availability reports.

– If a break happens during Nagios scheduled

downtime, the break is automatically

categorized as scheduled downtime. 19

Gathering detailed information about

Nagios breaks

20

Providing availability reports to end-

user organizations

An organization connected to Funet will be

able to see the availability history of all

used services at a glance.

– IP connections

– Light paths

– … and more?

Availability data is provided by Nagios-

Surfer.

Work in progress

21

Some other useful tools

A tool for scheduling Nagios downtime

according to predefined templates.

– Server X is rebooted – affects also services Y

and Z.

– Scheduled downtime is set for all affected

services.

A tool which combines several Nagios

service groups into one large service

group.

22

Performance

Our primary monitoring server is a quad-

core Xeon with 12GB of RAM and Ubuntu

10.04 LTS.

Nagios keeps up with the monitoring

schedule.

Occasionally we have seen bad

interactivity on the server, caused by

massive disk I/O.

– Especially when writing the state retention file

– Nagios status files and object cache are now

stored on tmpfs. 23

Things to improve

Our high resolution end-user site ping

monitoring is done outside Nagios.

Nagios polls the status of end-user sites

periodically, from the extenal monitoring

system.

New problems are seen by Nagios only

after the next service check.

– It would be better to push state changes to

Nagios immediately.

NOC would not have to look at two different

monitoring screens. 24

Things to improve (continued)

We have still some legacy monitoring that

is done with custom-made scripts.

We would like to integrate all our

monitoring to Nagios.

– We could use the same reporting for all our

monitored services.

– We could have a single NOC monitoring

screen.

25

Conclusions

Nagios suits us well.

Nagios is easy to customize.

– Has allowed us to modify and build on the

available features.

On the other hand, switching away from Nagios

would be a lot of work now.

26