The Journey Toward 24/7 IT Monitoring

University of North Carolina at Greensboro

Design and Build of Network Operations Center

Copyright Thomas M. Sheriff, 2004. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

The 2004 Enrollment

• 11,497 undergraduate students.

• 3,217 graduate students.

• 14, 714 total headcount.

• Largest freshman class (2,158)

• Residence halls at capacity! (approx. 3,800)

• Approx. 2500 employees (Faculty/Staff)

The Data Network

• End to end Cisco network (IP only)

• 700 network switches

• Approx. 25,000 ports.

• 50 buildings connected to the campus network

The Server Farm

• Two Machine rooms– 9 VMS servers– 63 UNIX servers– 30 Netware servers– 71 Windows servers

The Enterprise Applications

• Banner Student, Alumni, Financial Aid

• SCT FRS and HRS

• Lotus Domino

• Blackboard

• Pipeline

• Novell file and print services

• Windows based web services

The Motivation

• Be ahead of the client– Clients often notified IT of some outage

• Collect monitoring knowledge in single location– Monitoring being carried out in isolated IT

enclaves

• Clearing house for enterprise events– Response to events differ throughout IT

The Goal

• Improve customer service and communication

• Encourage a team approach to event management

• Delivery of concise information to management

The Method

• Develop a project within the IT Project Management model

• Three phase approach

The 3 Phase Approach

• Specifications– Define the physical characteristics of the NOC

• Design– Draw upon Physical Plant and Vendors for

design development

• Build– Facilitate renovation– Facilitate infrastructure installation– Bring facility to operational state

The Process

• Weekly meetings with project team– Developed NOC specifications document– Developed monitoring priorities

• Planning sessions with Physical Plant

• Visits from vendors

• Visits to working NOC’s

The Obstacles

• Other projects gaining priority

• Lack of funds

• Staffing challenges

• Marginally acceptable physical space

• Renovation costs

The Progress

• Improved customer service

• Improved service response time

• Centralized IT troubleshooting location

• Monitoring with low cost/no cost tools

• Monitoring centralized

• Low cost functional NOC

The Hardware

• 18 Dell SX280• 2 Belkin KVM switches• 2 Samsung giant monitors• 8 Dell GX models• 3 Sun workstations• 6 EDP console units

– Rack cubes– Phone mounts

• 10 Cisco switches– All VLAN’s present– Out of band networks

The Software

• What’s Up Gold– 140 servers monitored

• ICMP (ping), DNS, NTP, HTTP, FTP, TELNET• Custom services defined as needed

– 700 switches• Maps automatically generated and imported via XML

– MAN and WAN routers monitored

• MRTG – trend analysis (daily/weekly/monthly)– Critical traffic paths– Router CPU usage– Domino connection counts– Environmental conditions

• APC Enterprise - Switch closet UPS monitoring

The Lessons Learned

• A NOC can be built without breaking the budget

• Adequate staffing resources should be identified in advance

• Pooling monitoring tools requires tact

• A completed NOC will take years

The Road Ahead

• Design of NOC for renovated building

• Incorporating enterprise monitoring tools

• Providing enterprise management tools

• CA or BMC enterprise tools?

The following slides contain photographs of the Network Operations

Center at UNCG.

The following slides are screen images of MRTG and What’s Up Gold as used at the UNCG NOC.