cisco csirt case study: forensic investigations with netflow

© 2014 Lancope, Inc. All rights reserved.

Cisco CSIRT: Security Analytics and Forensics with NetFlow

Presented by:

Michael Scheck, Information Security Manager, Cisco

Paul Eckstein, CSIRT Engineering Manager, Cisco


The Heartbleed Bug is a serious vulnerability in the popular OpenSSL cryptographic software library. This weakness allows stealing the information protected, under normal conditions, by SSL

April 8, 2014: Heartbleed Vulnerability


Cisco CSIRT Response to Heartbleed • Preparation

• Scanned 1.2M vulnerable servers - 300 needed repair

• Helped develop signatures for Sourcefire and Cisco IDS

• Deployed signatures to IDS

• Monitoring and response • Discovered 25 attacks: 21 benign, 4 malicious • Researched attack via NetFlow analysis to discern

normal connections from those that were anomalous and malicious

3


Heartbleed Benign Host

4


Heartbleed Malicious Host

5


NetFlow@Cisco History


A

B

C

C B

A

C A

B

NetFlow Basics

7

Presenter

Presentation Notes

In conjunction with IDS, Netflow is one of our top two data sources. We process 21 billion netflow records each day across the enterprise. So what is it? Netflow is analogous to a phone record in that it shows who communicated with whom at what time, and for how long. We can therefore see: source address, destination address, number of packets transferred during that session, and a timestamp of the session. - Best of all - It’s free


NetFlow Collection and Analysis Solutions

8

OSU FlowTools nfdump Lancope StealthWatch

License Open source from Ohio State

Open source from SourceForge

Commercial

NetFlow versions

V5 v5 and up v5 and up

IPv6 ready? Yes Yes Yes

Syntax Command-line, like ACLs

Command-line, like tcpdump

GUI

Support Ad-hoc via Google Code

Up-to-date Up-to-date


NetFlow at Cisco Before StealthWatch • OSU FlowTools • 25+ systems running in parallel

- Speeds up query time, but routers have to point at each collector

• 20+ Tb of physical storage

- Files were stored in native nfdump/flowtools compressed format

• No flow aggregation • Some connections passed through multiple

devices, causing duplicate flows • Routers splitting up long running flows

9

Presenter

Presentation Notes

Investigators and analysts running ad-hoc queries Detect known bad communications Network Forensics during an incident Variables had to be defined at the time of the query Usually done as a paste of subnets/IP’s No true behavioral detection, just thresholds Top talkers scanning


NetFlow Challenge:Support • Support of open source tools • OS support • Training staff • Feature requests • Protocol changes (NetFlow and IP) • Difficult to monitor for flow loss

10


NetFlow Investigation with OSU FlowTools Query

bot.acl file uses familiar ACL syntax. create a list named ‘bot’ [mynfchost]$ head bot.acl ip access-list standard bot permit host 69.50.180.3 ip access-list standard bot permit host 66.182.153.176

[mynfchost]$ flow-cat /var/local/flows/data/2007-02-12/ft* | flow-filter -Sbot -o -...

Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP 0213.08:39:49.911 0213.08:40:34.519 58 10.10.71.100 8343 98 69.50.180.3 31337 0213.08:40:33.590 0213.08:40:42.294 98 69.50.180.3 31337 58 10.10.71.100 83

11


NetFlow Investigation with OSU FlowTools Custom NetFlow Report Generator

Presenter

Presentation Notes

This can all be simplified with a GUI frontend to the CLI commands.


Our Installation


Internet

Data Center

ISP Gateways

NetFlow Collector

DC Gateways

Corporate Backbone

NetFlow exported at network choke

points

NetFlow Export at Cisco Collect at chokepoints for egress detection

14

Presenter

Presentation Notes

Martin will discuss topological placement of netflow in more detail in the next section ‘Collect’ Architecture – From where do we get our Netflow? Same as IDS: -Furthest aggregation point from the backbone Behind the firewall At each in/egress point Cisco does have a new product called ‘Cisco Netflow Generation Appliance’ which was built as a work-around for devices that do sampling so customers can still get full netflow. This is in conjunction and yet another alternate solution to the Lancope flow collectors that Cisco is reselling.


NetFlow Collection at Cisco with StealthWatch

15


Common collection infrastructure

• Redundant forwarding

• Regional storage

• Global search

• Applies to netflow, log collection

16


Lancope Devices and Count

StealthWatch Management Console

FlowReplicator FlowSensor FlowCollector

2

2 10 13

17


NetFlow Retention

18

SJC 4-18 months

RCDN 10 months

RTP 4 months

LON 26 months

BGL 5-9 months


Problems Solved


30s 30s 30s

NetFlow Challenge: Flow Timeouts One 90s flow creates 6 flows

30s timeout 90/30 = 3 x 2 collectors

30s 30s 30s

NetFlow creates 3 flows NetFlow creates 3 flows

Lab gateway ISP gateway

20


Business Benefit #1 Storage Capacity

30s 30s 30s 30s 30s 30s

NetFlow creates 3 flows NetFlow creates 3 flows

Lab gateway ISP gateway

21


Business Benefit #2 Ease of support • IPv4/IPv6 both supported • NetFlow v5/v9 both supported • All supported on the same

system, on the same port! • No system administration

required • Alarms built in for monitoring of

lost flows

22


Business Benefit #3 Ease of use

24


• Other variables: host groups, time range, interfaces, ports • Defaults to 2000 flow records returned • Much simpler than syntax for CLI (example

below)

Flow Table Query

1. Create a file called‘flow.acl’with a named access list:

linux-machine# cat ip access-list standard botnet permit ip 10.31.33.7 >flow.acl

2. Run a query for the time period you are interested in using the ACL linux-machine# flow-cat /var/local/flows/data/2006-12-01/ft* | flow-filter -f ~/flow.acl -Sbotnet -o -Dbotnet | flow-print -f5

25


Flow Table Output

26


FlowTable Results

Server, DNS, and Country

Traffic Type & Volume

27

Presenter

Presentation Notes

Part 2 from previous slide – NetFlow allows us to see the volume and type of traffic to servers, to aid our investigation.


NetFlow Challenge: Limited Detection Capability • No concept of host groups for query • Effective for forensics • Can do basic DOS detection • Any other queries required writing

algorithms

29


Suspected Data Loss

High File Sharing Index

Max Flows Served

Business Benefit #4: Analytics

30


Use Cases


NetFlow CNC discovery

32

2. Investigate other internal hosts communicating with the same CnC

1. Detect host communicating with external Command-and-Control

3. Uncover other malicious, external entities from the compromised hosts

Targeted Monitoring:DoS Detection

33

34

Targeted Monitoring – Data Loss

35

Targeted Monitoring: Data Loss


StealthWatch Host Locking

36

Send syslog for any traffic seen between insides hosts and known C&C servers


StealthWatch Host Locking

37

Modify known C&C server list via API


CRiTs [email protected]

38

Presenter

Presentation Notes

“Collaborative Research Into Threats” Created by MITRE Tracks Indicators by various values Source Type Relationships Role Based Access Control Handles file samples/pcap uploads Various service “plugins”


CRiTS Indicator Actions

39

Prevent

DNS RPZ

host IDS BGP

Detect

Syslog

In Progress

passive DNS

Share

Govt

Current

Future

CSIRT

Mandiant

ESA

HIPS LUPA/ PCAP

WSA

Partner

CRITS

MD5

IPV4 Regkey

AV SBG

CDSA

Lancope

CRiTS NetFlow Alarms

40


Splunk Integration – SMC Alarms Requirement: integrate flow events with other logs for a single investigation interface

Solution: send relevant alarms as syslog messages to in-house Splunk™ architecture

Presenter

Presentation Notes

When to remove an IP from a list?


StealthWatch Splunk Alerts

Link to StealthWatch host snapshot


API Use Cases Requirement Problem API Script Solution Pull all flows for given time period

SMC Flow Collector query limit

Run consecutive, small queries then concatenate

Keep SMC host groups up to date

Manual configuration, old data

Query internal source of truth, push subnet lists to host groups automatically

Look up events for a particular IP for a specific timeframe

No user attribution (yet) Find IP and lease time from internal source of truth, query StealthWatch for related events

43


Network Subnets Mapped from IPAM


Network Subnets Map to Lancope Zones

45


Splunk integration: getFlows

Find NetFlow events via Lancope API with the respective src/dst


Splunk Integration - getFlows

47


Next Steps How to get started:

1. Find a collection/query system for NetFlow

2. Export NetFlow from chokepoints

3. Map your network context from IPAM into zones for query

4. Configure alarms for specific zones

5. Setup performance monitoring to mitigate flow loss from exporters

6. Integrate with your portfolio via API

7. Train your users and administrators – attend Lancope webinars and training

48


Contact information: Mike: [email protected] Paul: [email protected]

mailto:[email protected]

mailto:[email protected]

cisco csirt case study: forensic investigations with netflow

Technology