syslog design, methodology and best...
TRANSCRIPT
BRKNMS-2031
Syslog Design, Methodology and Best Practices
Follow us on Twitter for real time updates of the event:
@ciscoliveeurope, #CLEUR
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 2
Housekeeping
We value your feedback- don't forget to complete your online session evaluations after each session & the Overall Conference Evaluation which will be available online from Thursday
Visit the World of Solutions and Meet the Engineer
Visit the Cisco Store to purchase your recommended readings
Please switch off your mobile phones
After the event don’t forget to visit Cisco Live Virtual: www.ciscolivevirtual.com
Follow us on Twitter for real time updates of the event: @ciscoliveeurope, #CLEUR
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 3
Session Abstract
―This session will help define the design and methodology for
implementing a robust syslog solution using open source tools on
Linux platforms.
It provides leading practices for deployment of a set of tools and
applications to support effective collection, storage, and analysis of
syslog messages.
This session provides examples using messages from Cisco IOS
Software, but is applicable to all other syslog message types and
general event management.‖
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 4
Joined Cisco in 2005 as a Network Consulting Engineer
Background was in routing and switching for 8 years
Moved to Network Management/OSS automation about 11 years ago
Frequent speaker at Networkers
Author of several Cisco.com whitepapers on network management architectures and large-scale syslog deployment
Meet the Engineer—Clayton Dukes
Self Portrait
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 5
Why Syslog?
Syslog Basics
The Syslog Message
Relevant IOS commands
Syslog vs. SNMP
Management Techniques/Methodologies
Syslog Analysis
Syslog Architectures
Analysis Tools
Implementation Walk-Through (Using Open Source Tools)
Topics
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 6
Why Syslog?
Syslog Basics
The Syslog Message
Relevant IOS commands
Syslog vs. SNMP
Management Techniques/Methodologies
Syslog Analysis
Syslog Architectures
Analysis Tools
Implementation Walk-Through (Using Open Source Tools)
Topics
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 7
―Cat6500 IOS 12.2(18)SXF contains about 90 SNMP traps, but has over 6000 syslog event messages.‖
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 8
Why Syslog?
Proactive Syslog management benefits both operations personnel and the company as a whole from a cost savings perspective
Successful event management provides:
Reduced downtime through operational effectiveness
Improved Incident Management through real-time detection and self-remediation
Reduced volume of incidents through proper problem management
Reduced severity of business interruptions
Proactive measures to reduce the need for post-mortem troubleshooting
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 9
Why Syslog?
Ignoring Syslog Doesn’t Mean Your Network Isn’t Failing or Degraded
%CDP-4-DUPLEXMISMATCH:
Duplex Full/Half between connections
%ENVMON-3-FAN_FAILED:
Fan failure – may cause overheating
%ENVMON-3-OVER_HEAT:
Device temperature is > 60C (140F)
%PQUICC-5-COLL:
Excessive collisions - broken or unterminated Ethernet cable
%SYS-3-CPUHOG:
The indicated process has run for too long a period of time without relinquishing the processor
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 10
Downtime = Lost Revenue
Reducing downtime through proactive problem management reduces operational cost
0.09
0.09
0.1
1.2
2.6
4.5
0 2 4 6
Transportation
Retail
E-Commerce
Media
Banking
Brokerage
Industry Cost of Downtime
Revenue LossPer Hour (inMillions ofDollars)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 12
The Syslog Protocol
Syslog is a client/server protocol
-The syslog sender sends a small (less than 1KB) text message to the syslog receiver. The receiver is commonly called "syslogd", "syslog daemon" or "syslog server". Syslog messages (RFC 3164) can be sent via UDP (514) and/or TCP*. The data is typically sent in clear text.
Originally developed in the 1980s by Eric Allman as part of the Sendmail project, syslog is now standardized within the syslog working group of the IETF
Syslog is supported by a wide variety of devices and receivers across multiple platforms. Because of this, it can be used to integrate log data from disparate systems into a central repository for real-time and historical analysis.
* TCP Support Is available with some syslog daemons, such as syslog-ng or rsyslog as well as Cisco IOS Software Releases after 12.4(11)T, 12.2(33)SRB, 12.2(33)SB, and Cisco IOS XE Release 2.1 12.2(33)SXI.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 13
The Syslog Message
Every syslog message should contain five distinct fields with the following information:
Facility
Severity
Hostname
Timestamp
Message
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 14
Syslog Message Facility
Syslog messages are broadly categorized on the basis of the sources that generate them such as OS, process or application and are represented in integers ranging from 0-23, Cisco devices use the local facility ranges 16-23 (local0 – local7)
By default, Cisco IOS devices, CatOS switches, and VPN 3000 Concentrators use facility local7 while Cisco Firewalls use local4
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 15
Network Devices
Should Log Levels 0-6
Level 7 Should Be
Used for Console
Troubleshooting
Syslog Message—Severity
The log source (such as a router) that generates the syslog message also specifies the severity of the message using single-digit integers 0–7
*Jun 28 08:50:47.359 EDT: %SYS-0-SYS_LCPERR0:Module 6: Linecard received
system exception: Module needs troubleshooting or TAC assistance
0 - Emergency: System is unusable.
Leading
Practice
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 16
Syslog Message—Severity
The log source (such as a router) that generates the syslog message also specifies the severity of the message using single-digit integers 0–7
*Jun 28 08:50:47.359 EDT: %RTD-1-LINK_FLAP: FastEthernet0/1 link down/up 5 times
per min
1 - Alert: Action must be taken immediately.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 17
Syslog Message—Severity
The log source (such as a router) that generates the syslog message also specifies the severity of the message using single-digit integers 0–7
*Jun 28 08:50:47.359 EDT: %SYS-2-MALLOCFAIL: Memory allocation of 27 bytes
failed from 0x3560638, alignment 0
2 - Critical: Critical conditions.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 18
Syslog Message—Severity
The log source (such as a router) that generates the syslog message also specifies the severity of the message using single-digit integers 0–7
3 - Error: Error conditions.
*Jun 28 08:50:47.359 EDT: %SYS-3-MOD_PWRFAIL:Module 1 failed to power up
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 19
Syslog Message—Severity
The log source (such as a router) that generates the syslog message also specifies the severity of the message using single-digit integers 0–7
*Jun 28 08:50:47.359 EDT: %C4K_EBM-4-HOSTFLAPPING: Host CE:E9:F7:81:33:19
in vlan 101 is flapping between port Po1 and port Gi7/6
4 - Warning: Warning conditions.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 20
Syslog Message—Severity
The log source (such as a router) that generates the syslog message also specifies the severity of the message using single-digit integers 0–7
5 - Notice: Normal but significant condition.
*Jun 28 08:50:47.359 EDT: %SYS-5-CONFIG_I: Configured from console by Skeeter
McGillicutty (10.10.86.123)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 21
Syslog Message—Severity
The log source (such as a router) that generates the syslog message also specifies the severity of the message using single-digit integers 0–7
6 - Informational: Informational messages.
*Jun 28 08:50:47.359 EDT: %STANDBY-6-STATECHANGE: Vlan42 Group 42 state
Standby -> Active
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 22
Syslog Message—Severity
The log source (such as a router) that generates the syslog message also specifies the severity of the message using single-digit integers 0–7
7 - Debug: Debug-level messages.
*Jun 28 08:50:47.359 EDT: %DOT11-7-AUTH_FAILED: Station 0000.1111.0c81
Authentication failed
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 23
Leading
Practice
Syslog Message—Severity
The log source or facility (such as a router) that generates the syslog message also specifies the severity of the message using single-digit integers 0–7
0 - Emergency: System Is Unusable
1 - Alert: Action Must Be Taken Immediately
2 - Critical: Critical Conditions
3 - Error: Error Conditions
4 - Warning: Warning Conditions
5 - Notice: Normal But Significant Condition
6 - Informational: Informational Messages
7 - Debug: Debug-Level Messages
Network Devices
Should Log Levels 0-6
Level 7 Should Be
Used for Console
Troubleshooting
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 24
Syslog Message—Hostname
The hostname field consists of the host name (as configured on the host itself) or the IP address
In devices such as routers or firewalls, which use multiple interfaces, syslog uses the IP address of the interface from which the message is transmitted (unless otherwise configured using the ―logging source‖ command
Note: Don’t get confused by ―host name‖ and ―hostname‖. ―Hostname‖ Is typically associated with a
DNS lookup. If the syslog message contains a ―host name‖, it may be (and often is) different than the
actual DNS hostname of the device.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 25
Syslog Message—Timestamp
The local time, in MMM DD HH:MM:SS format, of the device when the message was generated
The * and . characters preceding a syslog
message are indicators of a problem with
NTP.
* Means that time is not authoritative: the
software clock is not in sync or has never
been set.
. Means that time is authoritative, but NTP is
not synchronized: the software clock was in
sync, but has since lost contact with all
configured NTP servers
For the Timestamp
Information to Be
Accurate, It Is Good
Administrative Practice
to Configure All the
Devices to Use the
Network Time Protocol
(NTP)
Leading
Practice
*Jun 28 08:50:47.359 EDT: %SYS-5-CONFIG_I:
Configured from console by Skeeter McGillicutty
(10.10.86.123)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 26
Syslog Message—Message Text
This is the text of the syslog message, along with some additional information about the process that generated it
Messages generated by Cisco IOS devices begin with a percent sign (%) and use the following format:
- %FACILITY-SEVERITY-MNEMONIC: Message-text
The mnemonic is a device-specific code that uniquely identifies
the message such as “up”, “down”, “changed”, “config”, etc.
*Sep 16 08:50:47.359 EDT: %SYS-5-CONFIG_I: Configured from console by vty0 (172.18.86.123)
Note: The ―Facility‖ in Cisco Mnemonics are not the same as the IETF definition of ―facility‖ (such as
local7). Cisco Facility Mnemonics are a free-form method of identifying the source message type such
as SYS, IP, LDP, L2, MEM, FILESYS, DOT11, LINEPROTO, etc. (the list is very large)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 27
Relevant IOS Commands
service timestamps debug datetime localtime show-timezone
service timestamps log datetime localtime show-timezone
clock timezone GMT 0
!
logging source-interface Loopback0
logging buffered 65536
logging host 192.168.100.20
logging host 192.168.100.21
logging host 192.168.100.22
logging trap informational
!
ntp server 143.232.55.5
ntp server 204.34.198.40
ntp peer 192.168.100.2
ntp peer 192.168.100.3
ntp update-calendar
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 28
Configuration Command Detail—Time
service timestamps debug datetime localtime show-timezone
service timestamps log datetime localtime show-timezone
clock timezone GMT 0
Time stamps can be added to either debugging or
logging messages independently
Datetime
Adds time stamps in the format MMM DD HH:MM:SS, indicating the date and time according to the system clock
Uptime
Adds time stamps in the format HHHH:MM:SS, indicating the time since the system was rebooted
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 29
logging source-interface loopback0
The ―logging source-interface‖ command instructs the system to generate logging to the remote system from this source interface
Ensures that all messages appear to come from the same IP and makes it easier to track in the destination syslog receiver
Allows you to create a DNS entry for that source interface
Configuration Command Detail—Logging
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 30
Logging (Cont.)
logging buffered 65536
Used to reserve a memory buffer for logging to the console of the device
Since today’s devices have plenty of memory, feel free to set this number higher than the old 16k buffer, but be aware that there is a point of diminishing returns
The typical recommendation is to have 256k buffers on core devices and 64k elsewhere
Note: Console refers to the output of the screen when attached to the device either by serial
or via telnet/ssh using the ―Terminal Monitor‖ command.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 31
Logging (Cont.)
logging host <ip address 1>
logging host <ip address 2>
logging host <ip address 3>
Sets the remote syslog daemon to send messages to
Use a maximum of four syslog servers
The syslog server can then be configured to forward or ―fork‖ messages to other Network Management Systems
Devices Should Be
Configured with a
Maximum of 3-4
Destination Servers
Leading
Practice
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 32
Logging (Cont.)
logging trap informational
Sets the syslog server logging level (emerg through debug)
Note: the term ―Trap‖ here has nothing to do with SNMP ―Traps‖ - it is simply a statement telling the
device to log the specified severity levels.
It’s NOT a Trap!
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 33
Configuration Command Detail—NTP
ntp server <ip address 1>
ntp server <ip address 2>
ntp peer <ip address 3>
ntp peer <ip address 4>
ntp update-calendar
The ―ntp update-calendar‖ command is used to synchronize the time of the internal clock with the clock of the NTP reference server
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 34
NTP Recommendations
Use a minimum of two reference clocks (GPS and Internet derived are popular)
―Peer‖ time between the reference clocks
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 35
Additional/Useful Logging Statements
logging count
Enables the error log count capability
http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_logging_count_ps6350_TSD_Prod
ucts_Configuration_Guide_Chapter.html#wp1025043
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 36
Logging Count – Sample Output
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 37
Syslog vs. SNMP Notifications
Can’t I just turn on SNMP traps?
- The simple answer is: no. In general there are significantly more syslog messages available within IOS as compared to SNMP Trap messages
- Cat6500 IOS 12.2(18)SXF contains about 90 SNMP traps, but has over 6000 syslog event messages
If You Had to Pick SNMP Traps or Syslog, Go with Syslog;
However, a Truly Robust and Full-Featured Event Management
Solution Would Take Advantage of All Fault Indicators
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 39
Traditional Syslog Management
File-based storage
-Traditionally, syslog daemons would store all incoming messages to one or more files for later parsing. This led to a very reactive use of syslog for after-the-fact troubleshooting and could not scale beyond very few devices
grep and tail
-Great tools in their own right, but hardly useful for scraping through gigabytes of log data…better tools are necessary
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 40
A Better Way
Store all incoming messages in a database
Provides speed and scalability
Capable of storing thousands of messages per second
Allows for trending and metrics
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 41
How?
Syslog-ng
-―An open source implementation of the syslog protocol for UNIX and UNIX-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP for transport‖
syslog-ng is a unix/linux daemon—it listens on a specified port for incoming data and forwards the information to a specified destination
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 42
How?
Rsyslog
An alternative daemon that works on the same principals as syslog-ng such as the ability to multiplex messages, use filters, pipe to programs, etc.
Not quite as ―mature‖ as syslog-ng, but it does have a lot of community support and many of syslog-ng’s ―pro‖ features for free.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 43
Now We’ve Introduced a New Problem!
How do we manage such a large amount of data?
How do we detect errors from a single device?
New processes need to be developed to detect device errors, degradation, change notifications, etc.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 44
Syslog Analysis
Database metrics
Collect metrics to show fault indicators and performance degradation
Top hosts/messages/severities, etc.
Messages per second/minute/day, etc.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 45
Syslog Analysis (Cont.)
Database metrics
Integrate syslog data with performance managers to trigger baseline thresholds
Example: Collecting the number of average messages per second a single device generates and alerting on variations outside the derived baseline
Integrate with Inventory systems!
If a device is talking to you, there’s a good chance it exists
Example: new devices being added to the network will have to wait until the next polling cycle by discovery systems, but if syslog is turned on in that device, your syslog manager will pick it up almost immediately.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 47
Syslog Architectures
Syslog-ng
Design Guidelines
Single Server Deployment
Multi-Server Deployment
Server Sizing
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 48
Syslog-ng
This design is based on the use of syslog-ng
Syslog-ng allows for collection and ―forking‖ of syslog messages to many hosts
This allows log data to be collected and distributed in a much more robust fashion
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 49
Traditional Logging Architecture
Traditional logging collection requires that many logging destinations be stored in each device
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 50
Syslog-ng Logging Architecture
Syslog-ng collectors allow for only a few logging hosts to be configured in your devices but then replicates these messages to end hosts
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 51
Design Guidelines
Collection stations
Design your syslog architecture in a distributed, hierarchical fashion
Syslog collectors should lie as close to their networks as possible
Some filtering may be done at the collection level to weed out unnecessary log data
These collectors should forward filtered messages to a centralized server/database for further filtering and processing
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 52
Design Guidelines (Cont.)
Device logging levels
Devices should be set to log all messages 0–6 for normal operation (and possibly 0–7 for debugging—although, if you are debugging, you're probably doing so on the console of the device and may not need to send level 7 to the collectors)
Network Time
It is important that you enable NTP throughout the architecture to ensure proper timestamps, not just for logging
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 53
Design Guidelines (Cont.)
Syslog Event Manager
Deploy a performance management tool such as Cacti to establish a baseline of your logs
Assign people (or groups) to monitor daily Top X events/hosts/messages, etc. and remediate common problems such as fan failures, duplex mismatch, redundant power fails, etc.
Log Rotation and Retention
Establish a log retention and rotation policy
Include logs and log archives in a standard backup process
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 54
Single Server Deployment
Can handle 100-200 million messages per day
Dependant on server CPU(s), disk(s) and memory
One million logs = ~350M of DB disk space a day, so size accordingly!
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 55
Multi-Server Deployment
Assured delivery via TCP
Can handle large amounts of messages
Requires high end (master) server
Distributed collectors can be small servers used to filter and forward
TCP
TCP
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 56
Server Sizing
The following is provided to help you decide which deployment scenario explained above is right for your organization
Please be aware that this is only a rough estimation
These calculations are based on data collected from Cisco’s internal IT management over a one week period with no guarantee of accuracy
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 57
Server Sizing
The Calculations for Each Field Are:
Approximate Messages Per Week = (Device Count * multiplier)
Approximate Messages Per Day = (Msgs Per Week/7)
Approximate Messages Per Minute = (Msgs Per Day/1440)
Approximate Messages Per Second = (Msgs Per Min/60)
Approximate storage capacity (in Mb) needed per day = (Msgs Per Day/1024/4)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 58
Device Type Multiplier
Router ACLs 4715
AIRONET 75
LANSWITCH 279
FIREWALL 533818
ROUTER 3238
VPN 1818
Reminder: These multipliers were either calculated using triangulated spy satellites and super-secret
algorithms OR they were a best guess based on past experience and industry averages - as noted
before, they should only be used as a rough guideline.
Device Multipliers
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 59
Server Sizing Sample
Device Type
Device
Count
Average
Msgs/Week Per Day Per Min
Per
Second
DB Size
(MB/Day)
Router ACLs 100 471,500.00 67,357.14 46.78 0.78 16.44
AIRONET 100 7,500.00 1,071.43 0.74 0.01 0.26
LANSWITCH 100 27,900.00 3,985.71 2.77 0.05 0.97
FIREWALL 100 53,381,800.00 7,625,971.43 5,295.81 88.26 1,861.81
ROUTER 100 323,800.00 46,257.14 32.12 0.54 11.29
VPN 100 181,800.00 25,971.43 18.04 0.30 6.34
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 60
High Level Starter Design
Syslog Protocol
Syslog Receiver
Syslog
Reporter
Sample Daily Reports
Top 10 Hosts
Top 10 Mnemonics
Top 10 Severities
Top 10 Programs
These are low hanging fruits, take advantage of them!
Sample Daily Activities
Identify Hardware related/Restart/Reboot events
Identify configuration changes (and forward to compliance manager)
Identify SNMP Authentication Failures
Identify large numbers of failed logins
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 61
High Level Advanced Design
Syslog Protocol
Syslog Receiver
Syslog
Reporter
Inventory Mgmt
NCCM
(Compliance)
Device
Synchronization
Filtered Change
Notifications
MoM/Fault
Mgmt
Event
Correlation
Fault
Notification
Performance
Mgmt
Event
Deduplication DB Poller
Incident/Ticketing
Baseline
Threshold
Alerts
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 62
Always filter unwanted messages versus wanted
Allows for proper metric trending
- You may not care that a VPN session has terminated, but do you care that 1000 of them are terminating every minute?
Saves you the embarrassment of having to explain to upper management why you MISSED a message that caused an outage.
Filtering Events
My Organization Has a LOT of Events.
Can You Please Tell Me What to Look For?
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 63
Design an architecture to build filters based on actionable events
Filtering Events
But… My Organization Has a LOT of Events.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 64
Actionable Events
When an Event Is Received, Two Immediate Questions Need to Be Asked:
Have we seen the event before?
Is an action required for the event?
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 65
Advanced Design – Actionable Events
Syslog Protocol
Event Initial Analysis Start
Syslog
(Event)
Database
Query
Response
Yes
Actionable
Event?
No
No Action: Leave Event in DB for
Later Forensics and Retention
Yes
Immediate
Action
Required?
Open
Incident
Open
Problem
Yes
No
Known
Event?
Periodically Move Old Events to Offline
(Disk) Storage
No
Mark as Known,
Determine Action
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 67
Syslog-ng Store Box (SSB)
LogLogic
Splunk
LogZilla
CiscoWorks LMS
Analysis Tools
#syslogtools
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 68
Syslog-ng Store Box
Turnkey solution for deploying syslog receivers using the pro version of syslog-ng
High-capacity log server with high-availability support
Able to collect logs from many different platforms
Made by the authors of syslog-ng
http://www.balabit.com/network-security/syslog-ng/log-server-appliance/
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 69
LogLogic http://loglogic.com
Commercial Solution
Capable of handling large amounts of data (70-100k mps)
Highly Scalable
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 70
Splunk http://splunk.com
Searches and navigates IT data (not just Cisco devices) from applications, servers and network devices in real-time
Free version available (limited to 500MB of storage)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 71
LogZilla http://www.logzilla.pro
Very fast, easy to use interface
Scalable to 20k EPS
Provides message de-duplication
Provides Cisco Mnemonics Tracking
@logzilla
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 72
CiscoWorks LMS (RME Component) http://www.cisco.com/go/lms
Supports filtering of unwanted syslog messages
Can trigger user-defined scripts in response to specific syslog messages
Provides reports to quickly view syslog events by severity, device, or message
Note: Some message filters are enabled by default, including: Link Up/Down, ASA, DEBUG, and IOS
Firewall Audit Trail messages. Large amounts of messages may lock up a server, so plan wisely!
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 74
Goals
Improve confidence of log integrity in the event of a security compromise.
How can we be certain that we’ll receive the event?
Provide mechanism to swiftly analyze logs on all systems.
Utilize daily reports to find ―top talkers‖
Provide mechanism to have instant reports on log-on activity on all systems, and any other ad-hoc reporting required.
Information Provided by Andrew Baughan - Cambridge IT Manager
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 75
Goals (Cont.)
Solution must be robust, and not introduce a significant overhead on target systems.
If possible, provide a mechanism to store all system logs for fault analysis and baseline statistical analysis for host data.
Provide a system capable of handling 100 Million events per day
Information Provided by Andrew Baughan - Cambridge IT Manager
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 76
Solution Selection
The two technologies chosen for this solution were syslog-ng and LogZilla
Because syslog-ng writes the log data to both the client (localhost) and the LogZilla server simultaneously, everything reaches the LogZilla server and cannot be simply deleted or modified on the local host.
Information Provided by Andrew Baughan - Cambridge IT Manager
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 77
Solution Selections (Cont.)
Since all logs are held in a central database, with a feature rich user interface, patterns of events across a large number of systems can be quickly identified and acted upon once the "signature" of a compromise is known.
It is easy to write scheduled jobs on the data stored within the MySQL database used as the back-end to LogZilla.
Open source makes it easy to customize to our needs if we ever need to.
Information Provided by Andrew Baughan - Cambridge IT Manager
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 78
Solution Selections (Cont.)
The syslog-ng software does not add significant overhead to the target systems.
Higher priority events can be filtered out from the lower priority background events for improved reliability during high traffic periods on the log server.
An increase to the allocated storage allowed a longer retention period, with all events being filtered to the log host.
Information Provided by Andrew Baughan - Cambridge IT Manager
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 79
Server Hardware
1 x Dell PowerEdge R410
- 2 x Intel Xeon E5504 @ 2 GHz
- 4 x 4GB DIMMs
- 1 x SAS 6/iR, internal raid card for Hotplug drives
- 2 x 160GB SATA 7.2k 3.5 inch Hotplug HDD (mirrored system disk)
- 1 x PERC H800 RAID Adapter for External JBOD, 512MB, PCIe
- 1 x iDRAC6 Enterprise
- 1 x 16X DVD+/-RW ROM Drive SATA
- 1 x Redundant Power Supply (2 PSU) 500W
- 1 x Sliding Ready Rack Rails
Information Provided by Andrew Baughan - Cambridge IT Manager
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 80
Storage Hardware
1 x PV MD1000 SAS Chassis
- 9 x 500GB NearLine SAS 6Gbps 7.2k 3.5" HD
- 1 x Rapid Rack Rails
- 1 x 2M External SAS Connector Cable
- 1 x PV MD1000 Bezel
Information Provided by Andrew Baughan - Cambridge IT Manager
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 81
Hardware Notes
Two internal disks are in hardware mirror (R1), ext3 file-system.
The external disks are in a hardware RAID6 configuration with one hot spare, ext3 file-system.
The memory was a bit low for our needs, so it will most likely get doubled fairly soon, otherwise no substantial bottlenecks that a bit of tweaking cannot fix (I/O caching for example has been tweaked).
Information Provided by Andrew Baughan - Cambridge IT Manager
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 82
Comments
Primary concern for large deployments should be DISK I/O and Memory.
When available use SSD disks (at least for write caching – google ―Cachecade Pro‖)
You can never have too much memory. Use a min of 32GB, but 96GB or even 128GB would be better.
-This determines how long it takes for SQL to do table sorting. The more memory you have, the less it swaps to disk.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 83
Implementation
Syslog Collector (using syslog-ng)
Search and Reporting Tool (using LogZilla)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 84
Implementation
Syslog Collector (using syslog-ng)
Search and Reporting Tool (using LogZilla)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 85
Hardware
My laptop and VMWare
Realistically, start with a dual or quad-core box with 4-8G ram and work up from there unless you expect a large amount of logs (> 15mil/day)
Software installed:
Ubuntu v10.04 Server (64bit is a must!) – Why?
Basic server with a LAMP stack
Updated to latest patches
Syslog-ng
Syslog Collector Server Environment
LAMP = Linux, Apache, PHP, MySQL
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 86
Remove ubuntu minimal?
Yes – Don’t Worry!
Use sudo If You’re Not Logged in as Root
(―sudo aptitude…‖
Syslog-ng is available in the apt repositories:
aptitude install syslog-ng
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libevtlog0
The following packages will be REMOVED:
rsyslog ubuntu-minimal
The following NEW packages will be installed:
libevtlog0 syslog-ng
Do you want to continue [Y/n]?
root@log#
Setting up syslog-ng (2.0.9-4.1) ...
* Starting system logging syslog-ng [ OK ]
Installing Syslog-ng
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 87
Installing Syslog-ng
That was difficult, wasn’t it?
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 88
Configuring Syslog-ng
The syslog-ng configuration file is typically stored in /etc/syslog-ng/syslog-ng.conf
There are five ―steps‖ to building a syslog-ng configuration
Main configuration options
Source definitions
Filter definitions
Destination definitions
Statement to apply the defined source, filter and destination
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 89
Main configuration options
Source definitions
Filter definitions
Destination definitions
Statement to apply the defined source, filter, and destination
Configuring Syslog-ng
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 90
Sets ―global‖ options that apply to everything, for example:
use_dns(yes);
use_fqdn(yes);
keep_hostname(yes);
chain_hostnames(no);
Main Configuration Sample
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 91
Sample ―Main‖ Configuration
options {
# buffer just a little for performance
# sync(1); (older versions use this instead of flush_lines)
flush_lines(1);
# memory is cheap, buffer messages unable to write (like to pipe)
log_fifo_size(16384);
# The time to wait before a dead connection is reestablished (seconds)
time_reopen(60);
#Use DNS so that our good names are used, not hostnames
use_dns(yes);
dns_cache(yes);
#Use the whole DNS name
use_fqdn(yes);
# Keep the hostname of the source device when forwarding to other NMS’s
keep_hostname(yes);
# Chain hostnames together so that the end NMS sees all hosts
chain_hostnames(no);
};
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 92
Main configuration options
Source definitions
Filter definitions
Destination definitions
Statement to apply the defined source, filter, and destination
Configuring Syslog-ng
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 93
Defines sources of information to receive messages from
source s_all {
internal();
unix-stream("/dev/log");
udp();
};
―s_all‖ Can Be Named Whatever You Want, Just Be Sure
to Use It Consistently
Syslog-ng Sources
Other UDP/TCP options are available, such as:
udp( ip(127.0.0.1) port(514) );
tcp( ip(0.0.0.0) port(5000) );
Only Allow UDP Messages from Localhost
Allow TCP Messages from All Hosts on Port
5000.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 94
Main configuration options
Source definitions
Filter definitions
Destination definitions
Statement to apply the defined source, filter, and destination
Configuring Syslog-ng
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 95
Defines a filter to be applied
Types of filters are:
Facility—Match on a facility code (kern, local7, etc.)
Level—Match on a level code (error, notice, emerg, etc.)
Program—Match messages by using a regular expression against the program field
Host—Match messages by using a regular expression against the host field
host("^cam(1|2|3|4|5)\.somehost\.tld$"); };
Match—Match a regular expression to the message itself
Filter—Call another filter rule and evaluate its value
Netmask—Determine if the sender’s IP is in the specified IP subnet
Syslog-ng Filters
―Level‖ Is Synonymous with ―Priority‖ or ―Severity‖
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 96
Main configuration options
Source definitions
Filter definitions
Destination definitions
Statement to apply the defined source, filter, and destination
Configuring Syslog-ng
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 97
Destinations Define Where to ―Fork‖ Messages to, Such as:
Files
Programs
Remote Hosts
PIPE/FIFO
Databases
Syslog-ng Destinations
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 98
Sample Destination Definitions
File
destination df_syslog { file("/var/log/syslog"); };
destination df_disk { file("/var/log/HOSTS/$YEAR/$MONTH/$DAY/$HOST" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); };
Program
destination d_mydest{ program("/var/www/logzilla/scripts/db_insert.pl" template("$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n") template_escape(yes));};
Remote Host
destination d_othersyslogbox { udp("170.19.86.100" port (514));};
A Destination Can Be Anything, Just Be Sure to Use the Same Name When Applying It
Later On
What Happens Here If You Do Not Have NTP Properly Configured on
Your Devices?
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 99
Syslog-ng Destinations (Cont.)
PIPE (or FIFO)
- destination d_mysql {
- pipe("/tmp/mysql.pipe" template(
- "INSERT INTO logs (host, facility, level, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$LEVEL', '$S_YEAR-$S_MONTH-$S_DAY $S_HOUR:$S_MIN:$S_SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes));
- };
Note: for direct DB inserts, it’s better to use the new SQL insert function built into syslog-ng 3.x, but
ONLY when you don’t use a pre-processor like PERL (for event correlation, deduplication, etc.)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 100
Syslog-ng Direct Database Inserts
The following example is for MySQL, but other types may be used such as sqllite, pgsql, mssql and oracle.
@version: 3.0
destination d_mysql {
sql(type(mysql)
host("localhost") username("syslog") password("syslog")
database("syslog")
table("logs")
columns("host", "facility", "level", "datetime", "program", "msg")
values("$HOST_FROM", "$FACILITY", "$LEVEL", "$YEAR-$MONTH-
$DAY $HOUR:$MIN:$SEC", "$PROGRAM", "$MSG―)
indexes("host", "facility", ―level", "datetime", "program"));
};
@ at the Beginning, syslog-ng 3.02 and Up Complains If This
Is Not Present
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 101
Using Direct Database Inserts
Plan on implementing this in a large scale environment?
If you have more than 1000 events per second, guess what happens?
- (MySQL will bottleneck causing dropped events)
Instead, use a ―program‖ call to Perl to process the incoming data and use bulk insert methods. This will handle up to ~35k mps (maybe more, but that’s what I’ve tested it to).
destination d_mydest { program("/var/www/myprog/scripts/myscript.pl‖
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 102
Main configuration options
Source definitions
Filter definitions
Destination definitions
Statement to apply the defined source, filter, and destination
Configuring Syslog-ng
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 103
Once you have a defined source, filter (optional) and destination, you must ―apply‖ them in a statement:
- log {
- source(my_source);
- filter(my_filter);
- destination(my_dest);
- };
Apply the Definitions
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 104
- source s_all {
- internal();
- unix-stream("/dev/log");
- file("/proc/kmsg" log_prefix("kernel: "));
- udp();
- tcp(port(2000));};
- filter my_filter {
- host("^router (1|2|3|4|5)\.cisco\.com$");};
Source
Filter
(Optional)
Final Sample Configuration
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 105
- destination my_dest {
- file("/var/log/logzilla/syslog.log"
- template("$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n"));};
- destination my_dest_hosts {
- udp("1.1.1.1" port (514));
- tcp("2.2.2.2" port (2001));};
- log { source(my_source); filter(my_filter);
- destination(my_dest);
- destination(my_dest_hosts);};
2
Destinations
Apply
Final Sample Configuration
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 106
Syslog-ng: Getting Help
Website
http://www.balabit.com/network-security/syslog-ng/
Documentation
http://www.balabit.com/support/documentation/
Mailing list
https://lists.balabit.hu/mailman/listinfo/syslog-ng
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 107
Implementation
Syslog Collector (using syslog-ng)
Search and Reporting Tool (using LogZilla)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 108
Installing LogZilla
Download LogZilla from http://www.logzilla.pro/downloads
- Ready2Run VMWare Server and ESX versions available
SVN option highly recommended!
cd /www root@log#
Extract (non SVN-based install):
tar xzvf logzilla_x.x.x.tgz root@log#
Use Subversion for Instant Software Updates!
http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.2#Subversion_Install
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 109
Installing LogZilla (Cont.)
Install:
cd /var/www/logzilla/scripts root@log#
./install.pl root@log#
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 110
Installing LogZilla (Cont.)
========================================
LogZilla Installation
========================================
Press Enter to Accept the
Defaults for These Unless You
Have Some Special
Requirements.
Enter the MySQL root username[root]:
Enter the password for root [mysql]:
Database to install to [syslog]:
Enter the name of the MySQL server [localhost]:
Enter the port of the MySQL server [3306]:
Enter the name to create as the owner of the logs database [syslogadmin]:
Enter the password for the syslogadmin user [syslogadmin]:
Enter the name to create as the WEBSITE owner [admin]:
Enter the password for skeeter [skeeter]:
Enter your email address [[email protected]]:
Enter a name for your website [The home of LogZilla]:
Enter the base url for your site (include trailing slash) [/logs/]: /
skeeter
nascarFTW
Yes, It’s a Real Email Address
Tackle Shop NOC
/
Where should log files be stored? [/var/log/logzilla]:
How long before I archive old logs? (in days) [7]:
Do you plan to log Windows events from SNARE to this server? [n]: y
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 111
Installing LogZilla (Cont.)
Updating file paths
Updating log paths
Generating /var/www/logzilla/html/config/config.php
All data will be installed into the syslog database
Ok to continue? [y]: y
Adding LogZilla logrotate.d file to /etc/logrotate.d
Ok to continue? [y]: y
Adding LogZilla to syslog-ng
Ok to continue? [y]:
Where is your syslog-ng.conf file located? [/etc/syslog-ng/syslog-ng.conf]:
Adding syslog-ng configuration to /etc/syslog-ng/syslog-ng.conf
Found 1 sources
Which source definition would you like to use? [s_all]:
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 112
Installing LogZilla (Cont.)
========================================
Cron Setup
========================================
Cron is used to run backend indexing and data exports.
Install will attempt to do this automatically for you by adding it to /etc/cron.d
Ok to continue? [y]: y
Will this copy of LogZilla be used to process more than 1 Million messages per day?
Note: Your answer here only determines how often to run indexing. [n]: n
Cronfile added to /etc/cron.d
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 113
Installing LogZilla (Cont.)
========================================
SUDO Setup
========================================
Syslog-ng MUST be restarted, would you like to send a HUP signal to the process?
Ok to HUP syslog-ng? [y]: y
HUPing syslog-ng PID 315
LogZilla installation complete!
In order for the Apache user to be able to apply changes to syslog-ng, sudo access needs to be provided in
/etc/sudoers
Note that you do not HAVE to do this, but it will make things much easier on your for both licensing and Email
Alert editing.
If you choose not to install the sudo commands, then you must manually SIGHUP syslog-ng each time an
Email Alert is added, changed or removed.
Ok to continue? [y]:
Please provide the username that Apache runs as [www-data]:
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 114
Installing LogZilla
Login to http://<your_url> and check for data
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 115
Troubleshooting
root@log# printf "`date \"+%Y-%m-%d %H:%M:%S\"`\ttest\t190\tCRON\tTest\n" |
/var/www/logzilla/scripts/db_insert.pl -d5 -v
Print a test message directly into the Perl pre-processor
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 116
LogZilla Command Line Test (Cont.)
Starting /var/log/logzilla/db_insert.log for /var/www/logzilla/scripts/db_insert.pl at pid 3570
Using Database: syslog
Debug level: 5
Table: logs
Adminuser:
PW:
DB: syslog
DB Host:
DB Port:
Deduplication Feature = 0
Logging results to /var/log/logzilla/db_insert.log
Printing results to screen (STDOUT)
INCOMING MESSAGE:
2011-03-30 18:35:47 test 190 CRON Test
HOST: test
PRI: 190
FAC: 23
SEV: 6
PRG: CRON
MSG: Test
MNE: None
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 117
LogZilla—Getting Help
Main Website
http://www.logzilla.pro
Forum
http://forum.logzilla.pro
Feedback (vote on new features!) and support
http://support.logzilla.pro
Installation Guide
http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.2
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 118
General Useful Links
Clayton’s NMS Wiki
http://nms.gdd.net
LinkedIn Syslog User Group
- An open forum for syslog Q&A
http://www.linkedin.com/groups?mostPopular=&gid=3729372
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 119
Key Takeaways
People tend to be a bit overwhelmed by the amount of data they have to parse through
- Proper implementation of tools, metrics and processes will solve that problem
Configure ALL devices consistently and properly; Make sure any new device deployment ALSO has the correct configuration
Properly designed, syslog can be one of the BEST sources for proactive network management available
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 121
Please complete your Session Survey
Don't forget to complete your online session evaluations after each session.
Complete 4 session evaluations & the Overall Conference Evaluation
(available from Thursday) to receive your Cisco Live T-shirt
Surveys can be found on the Attendee Website at www.ciscolivelondon.com/onsite
which can also be accessed through the screens at the Communication Stations
Or use the Cisco Live Mobile App to complete the
surveys from your phone, download the app at
www.ciscolivelondon.com/connect/mobile/app.html
We value your feedback
http://m.cisco.com/mat/cleu12/
1. Scan the QR code
(Go to http://tinyurl.com/qrmelist for QR code reader
software, alternatively type in the access URL above)
2. Download the app or access the mobile site
3. Log in to complete and submit the evaluations
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 125
Syslog Collector (using syslog-ng)
Search and Reporting Tool (using LogZilla)
Event Correlation (using Simple Event Correlator)
Event Correlation
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 126
Installing Simple Event Correlator (SEC)
SEC is available in the apt repositories:
aptitude install sec
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
Writing extended state information... Done
The following NEW packages will be installed:
sec
root@log#
Setting up sec (2.4.2-1) ...
SEC disabled in /etc/default/sec
Use sudo If You’re Not Logged in as Root (―sudo aptitude…)‖
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 127
#Defaults for sec
RUN_DAEMON="no"
DAEMON_ARGS="-conf=/etc/sec.conf -input=/var/log/syslog -
pid=/var/run/sec.pid -detach -syslog=daemon"
Change to ―yes‖
Installing Simple Event Correlator (SEC)
Edit the SEC config to allow it to start
vi /etc/default/sec root@log#
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 128
Installing Simple Event Correlator (SEC)
SEC uses a configuration file and takes input from a file or a named pipe
vi /etc/sec.conf
# Example
# Recognize a pattern and log it.
#
type=Single
ptype=RegExp
pattern=foo\s+(\S+)
desc=$0
action=logonly
root@log#
First step is to create a config file:
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 129
SEC Rules
SEC includes several different types of rules that are useful in event correlation.
This rule is of type Single.
# Example
# Recognize a pattern and log it.
#
type=Single
ptype=RegExp
pattern=foo\s+(\S+)
desc=$0
action=logonly
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 130
SEC Rules
RegExp is the pattern type, select RegExp for (―Regular Expression‖) matching or SubStr, for simpler string matching
# Example
# Recognize a pattern and log it.
#
type=Single
ptype=RegExp
pattern=foo\s+(\S+)
desc=$0
action=logonly
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 131
SEC Rules
foo\s+(\S+) is the actual pattern - in this case a perl regular expression pattern
# Example
# Recognize a pattern and log it.
#
type=Single
ptype=RegExp
pattern=foo\s+(\S+)
desc=$0
action=logonly
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 132
SEC Rules
desc is a variable definition for the pattern description (captured from the foo pattern using parentheses).
In this case a perl numbered variable, $0, is set to the entire matched pattern
# Example
# Recognize a pattern and log it.
#
type=Single
ptype=RegExp
pattern=foo\s+(\S+)
desc=$0
action=logonly
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 133
SEC Rules
The action statement describes the action taken when the pattern is recognized.
In this case, the logonly action simply writes the pattern to the logfile if one is indicated on the command line, or to standard output if not
# Example
# Recognize a pattern and log it.
#
type=Single
ptype=RegExp
pattern=foo\s+(\S+)
desc=$0
action=logonly
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 134
SEC Rules
Save the file and execute the following command:
sec -conf=/etc/sec.conf -input=- root@log#
SEC (Simple Event Correlator) 2.4.2
Reading configuration from /etc/sec.conf
1 rules loaded from /etc/sec.conf
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 135
SEC Rules
This example will take input from directly from the terminal. Type the following lines of input:
foo
foo bar
foo bar
baz
bar foo baz
bar foo
Notice that SEC Responds by Replying Every Time a Pattern Is Matched
bar foo baz
Response
Response
No Response
No Response
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 136
SEC Actions
SEC has over a dozen different actions it can perform once it matches a pattern in the input stream
Some of the actions depend on their context
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 137
SEC Actions
write
Writes the specified text to the named filename.
E.g.: action=write - Hello from SEC. Matched text was $0
shellcmd
Causes SEC to execute a shell command.
E.g.: action=shellcmd mycommand.sh
spawn
Identical to the shellcmd action, but output (e.g. from an exit status in that shell script) from the command is fed back into SEC for pattern matching
E.g.: action=shellcmd mycommand.sh
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 138
SEC Actions
assign and eval
Both assign and eval deal with ``% <letter>'' variables. They are internal SEC variables that can be used in rules
E.g.: action=assign %f Joe bob likes,
action=eval %h ($t = ―fishing and nascar")
action=write - %f %h at %t
Note: These Are All Parts of
Separate Rules, But Kept Out
for Brevity
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 139
SEC Actions
event
event allows the insertion of input to SEC from inside SEC itself
event is feedback mechanism - one controlled by SEC's own rules.
The time parameter is the number of seconds to wait before inserting the event text into SEC's input stream.
E.g.: action=event 5 baz is now matched. ; write - foo matched at %t. baz event in 5 seconds...
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 140
Sample Rules—Suppress
# System configuration events
# suppressed because we don't care about it
type=suppress
ptype=substr
pattern=%SYS-5-CONFIG_I:
desc=device configuration
What’s Wrong with This Rule?
(Should We Suppress
Configuration Changes?)
Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 141
Sample Rules—Time Based
# Looks for a reload followed by a restart event
#
type=pairWithWindow
ptype=regexp
pattern=\d+:\d+:\d+.*?(\S+)\s+\d+:.*?%SYS-5-RELOAD:
desc=(CRITICAL) $1 RELOAD_PROBLEM
action=pipe '%s' mail -s 'cisco event' [email protected]
ptype2=regexp
pattern2=($1).*?%SYS-5-RESTART:
desc2=(NOTICE) $1 RELOAD_OK
action2=pipe '%s' mail -s 'cisco event' [email protected]
window=300
Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 142
Sample Rules—Escalation
# This rule escalates to CRITICAL if there are more than 5
# neighbor changes in 5 seconds
#
type=SingleWithThreshold
ptype=substr
pattern=(MINOR) OSPF adjacency change
desc=(CRITICAL) More than 5 OSPF neighbor changes in 5
seconds
action=pipe '%s' mail -s 'cisco event' [email protected]
thresh=5
window=5
Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 143
Sample Rules—Link Up/Down Pairs
# This rule deals with link down events
#
type=PairWithWindow
ptype=RegExp
pattern=\d+:\d+:\d+.*?(\S+)\s+\d+:.*?%LINK-3-UPDOWN: Interface
(\S+), changed state to down
desc=(MINOR) $1 INTERFACE $2 DOWN and not up in one minute
action=pipe '%s' mail -s 'cisco event' [email protected]
ptype2=RegExp
pattern2=($1)\s+\d+:.*?%LINK-3-UPDOWN: Interface ($2), changed
state to up
desc2=(WARNING) %1 INTERFACE %2 BOUNCE
action2=event %s
window=60
Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 144
Sample Rules—Link Up/Down Pairs
# when the first bounce event is seen, create a reporting trigger
#
type=Single
continue=TakeNext
ptype=regexp
pattern=(\S+) INTERFACE \S+ BOUNCE
context=!INTERFACE_BOUNCE_WAIT_$1
desc=interface bounce summary event for router $1
action=create INTERFACE_BOUNCE_WAIT_$1 10 (report
INTERFACE_BOUNCE_$1 mail -s 'cisco events'
[email protected]; delete INTERFACE_BOUNCE_$1)
Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 145
Sample Rules—Link Up/Down Pairs
# accumulate all interface bounce events into a context
#
type=Single
ptype=regexp
pattern=(\S+) INTERFACE (\S+) BOUNCE
desc=interface bounce for router $1 interface $2 detected
action=add INTERFACE_BOUNCE_$1 %t: %s
Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 146
SEC—Getting Help
SEC Main Page
http://simple-evcorr.sourceforge.net/
Email list
http://simple-evcorr.sourceforge.net/#mailinglist
Good install and explanation guide
http://sixshooter.v6.thrupoint.net/SEC-examples/article.html