CSN08: You Can't Correlate What You Don't Have

Scott Carlson & Rick YetterApollo Group Inc.

Apollo Group

The Apollo Group Challenge

Apollo Group is a publicly traded parent company that owns the University of Phoenix and a number of other subsidiaries in the education arena. With 300 physical locations in six countries, 500,000 students, 50,000 faculty and 22,000 employees, Apollo Group has a formidable challenge in securing all its systems, data and endpoints.

Reference: http://www.arcsight.com/collateral/case_studies/ArcSight_CaseStudy_Apollo.pdf

2

3

What’s this about?

• ArcSight products are awesome, as long as you send them information. The products can’t do anything unless you send them as much data as possible!

• Normal steps to implementation1. Define Use Cases2. Send Logs3. Build Correlation Rules in ArcSight ESM4. …5. Profit!!!

4

Our Environment

• 4,500 Servers– Oracle Enterprise Linux, Red Hat Linux– Windows 2000, 2003, 2008– Solaris 9,10

• 60% Virtualized on VMWARE• Multiple international locations & data centers• Firewalls (Cisco, Juniper, Checkpoint)• Proxy (BlueCoat)• IDS (SourceFire)• AV/HIPS/DLP (McAfee)• ….

5

It’s the Logs that matter

6

“The powerful correlation engine of ArcSight ESM sifts through MILLIONS OF LOG RECORDS to find the critical incidents that matter” (www.arcsight.com)

Server Stuff

Security Events Change Monitoring Failure Events Application Logs Web Logs Host Firewalls Active Directory Activity

Network Stuff

Firewalls Proxies Intrusion Detection Antivirus Data Loss Prevention Email Traffic & Alerts Wireless Network Change Monitoring

Where to store the logs

• Long term storage is critical for large companies– Determine retention requirements (30 days, 1 year, infinite)

• Determine who may need the logs, do you need them online?– SysAdmin, Forensics, InfoSec, Legal

• Do you need non-repudiation? • Determine Storage method

– Splunk - Filesystem(s) full of Raw Log files– ArcSight Logger - Alternate Logging Product

• If you build your own– SAN versus NAS versus Local JBOD. You need to log even if things

break!

7

2

Syslog Relay

• Red Hat Linux• Syslog-ng v4 running on multiple ports

– For receiving logs from multiple sources with unique filters• Local JBOD w/12TB configured as RAID-5

– Make sure you can log even if your SAN is borked!• Additional security of SAMHAIN, tripwire, Solid Core to protect

your files from modification• 64GB of ram• Lots of processors

• Built In Syslog• Syslog-NG• SNARE or Epilog agent, kiwi• File-Reader ArcSight Connector• Something entirely custom, just put it in a FILE!

– Syslog format or CEF Format, you pick.

You found something without logs???? Well, Ask the developer or company to add logging!!!!

How to Get the Logs

9

2

Single Endpoint

• For smaller environments, or environments with fewer layer-2 boundaries

• Should configure server with redundancy in mind in case of failure

• Can use file reader connector to read from local logs

• Single destination, easy to script• May not scale• Limited to small number of

networks unless you traverse firewalls

2

Single Endpoint with DR Site

• Makes a copy of all logs to an alternate site• Saves you in case of catastrophic failure• Adds bandwidth to the WAN or remote site link

Local Collect & Forward

• Individual Syslog collection in each major network block or international location

12

How to send the Logs

• Configure syslog– *.debug @loghost.mydomain.com (Solaris)– *.* @loghost (Linux)

• Configure SNARE– Destination Snare Server address = loghost– Destination Port = 514– Enable SYSLOG Header = Selected

• Read the Fine Manual of your product to enable logging with a remote destination. If that’s not there, write to a file!

13

Decision points

• What’s your Double-Send point?– Host

• Not available in all “free logging tools”• Some things cannot double send (network gear, appliances)

– Relay• Adds cross-data center traffic times # Relay

– Central• Easy to control flow, exposure is at this point in each DC• Blind to logs if central server is gone

• What about things that don’t have syslog?– File Reader to multiple ArcSight ESM Targets is a possibility

14

Redundancy and Double-Sending

• Fail-over scenarios in use for Apollo Group using Syslog-ng– Redundancy at the Syslog Relay level

• Logs are sent from Snare agents on Windows or by Syslog to relays– Each Syslog relay has a VM hot standby in case of a hardware

failure.• Each Syslog relay is configured to send all information received to

multiple central servers for redundancy and fault tolerance. • Each Syslog relay retains all logs received for a period of 30 days

before being rotated out.

15

Syslog-NG Configuration

• Syslog-ng configuration (4 simple steps)– Simple Configuration

• Source– Where are the logs coming from? UDP, TCP, File

• Destination– Where are you going to send the logs? Disk, output to TCP or

UDP?» Can you handle the TCP Overhead?

• Filters– Keep what you want, discard the rest!

• Log– Log the source, process it, send it to the destination.

• Encrypted communications must use TCP

16

Syslog-NG Configuration Sample

• source s_local {• internal();• unix-stream("/dev/log");• file("/proc/kmsg" program_override("kernel:"));• udp(ip(0.0.0.0) port(514) flags(store-legacy-msghdr));• udp(ip(10.11.12.13) port(514) flags(store-legacy-msghdr));• };

• #• # Local filters• #• filter f_boot { facility(local1); };• filter f_messages { level(info..emerg); };• filter f_secure { facility(authpriv); };• filter f_mail { facility(mail); };• filter f_cron { facility(cron); };• filter f_emerg { level(emerg); };• filter f_spooler { level(crit..emerg) and facility(uucp, news); };• filter f_local7 { facility(local7); };• # Snare *NIX Filters• #• #filter f_filter_nix { match(":") and not match("snmp") and not match("printd") and not match("-6-302013") and not match("-6-

302015") and not match("kernel") and not match("lpstat") and not match("Application") and not match("System") and not host("10.29.10.100") and not match("dhcpd") and not match("xinetd") and not match("puppetmasterd") and not match("crond") and not match("multipathd") and not match("modprobe"); }; 17

Syslog-ng destinations (local)

• #• # Local destinations• #• destination d_messages { file("/u01/log/messages"); };• destination d_secure { file("/u01/log/secure"); };• destination d_maillog { file("/u01/log/maillog"); };• destination d_cron { file("/u01/log/cron"); };• destination d_console { usertty("root"); };• destination d_spooler { file("/u01/log/spooler"); };• destination d_bootlog { file("/u01/log/boot.log"); };• #

18

Syslog-ng destinations (remote)

• # Remote Destinations• #• destination d_forward { udp("10.3.4.5" port(514) keep_alive(no)); };• #• # Local logs • #• log { source(s_local); filter(f_emerg); destination(d_console); };• log { source(s_local); filter(f_secure); destination(d_secure); };• log { source(s_local); filter(f_mail); destination(d_maillog); flags(final); };• log { source(s_local); filter(f_cron); destination(d_cron); flags(final); };• log { source(s_local); filter(f_spooler); destination(d_spooler); };• log { source(s_local); filter(f_boot); destination(d_bootlog); };• log { source(s_local); destination(d_messages); };• log { source(s_local); destination(d_forward); };

19

Crazy Use Case #1

Get the unique customer # out of a sub-url string within the debug log of a firewall in order to perform tracking/troubleshooting

• Debug Logging ON on firewall (LOTS OF TRAFFIC!)• Logs send to Syslog• Syslog filter or external program called to trim out the customer

number and write it to a separate file

20

Crazy Use Case #2

Forward non-security events directly to your NOC Console, email queue, or whatever

• Syslog filter or external program called to grab the events you’re interested in, and send them to external mailer (mail –s “alert”) or a syslog-ng filter

• Don’t forget the System Administrators• IMHO 90% of problems are misconfigured systems

21

Crazy Use Case #3

Gather logs from a proxy server, at 5 minute intervals, and make sure that they’re going to your DR Site with minimal delay, add a filter to find naughty surfing.

• Proxy server sends logs via SCP to syslog relay• Syslog relay writes file to local JBOD• Syslog-ng or local script scrapes naughtiness from file• Cron job runs at 5 minute intervals to SCP completed files to DR

Watch out for incomplete files!Make sure your formatting is good!!!

22

Multiple ArcSight ESM Instances

• Double sending all logs allows you have two independent ArcSight ESM instances, in multiple data centers capable of performing your SOC duties at a moments notice.

23

Q&A

24