Securing and Monitoring 10GbE WAN Links

Steven CarterCenter for Computational Sciences

Oak Ridge National Laboratory

Disclaimer

•Oak Ridge National Laboratory does not endorse any particular product. This presentation merely details our experience and chosen course of action (i.e. I am not a patsy for Force10).

Requirements•Wire rate intrusion detection (i.e. 20Gb/s)•Little or no latency•Low administrative/development overhead•Flexible (used for IDS and protocol monitoring)•Scalable (We have 5+ 10G links that we would like to monitor)•Affordable

Approaches•Divide and Conquer: Use a piece of

network equipment (e.g. Juniper Router) to divide the stream of packets by some attribute (e.g. destination port) into smaller, more easily handled streams for processing.

Approaches (Cont.)

•Host intensive: Send the full (or possibly filtered) stream to the host CPU for inspection.

•NIC intensive: The NIC does the packet inspection.

The Contenders

•Intel, Neterion, Chelsio 10G NICs•Endace DAG 6.2SE•Force10 P-Series (formally

MetaNetworks)

Initial Pros/Cons• Standard 10G NICS

• Inexpensive• Single host unable to keep up with full rate, full duplex connection

• Endace DAG 6.2SE• Offload allows single host to inspect more traffic (~13Gb/s), but you need a beefy host.• Timestamps• Only available with 1310nm optics• Expensive

Initial Pros/Cons (cont)

• Force10 P-Series• Less expensive• Compete offload• Scalable• Can block packets if used in-line• Supports too few snort rules (700 shared between 2 channels)• Long compile time• PCI Bus (1Gb/s b/w the card and the host)

Initial Test Setup

HostHostHostHost

P-SeriesP-Series

DAGDAG

SwitchSwitch HostHostHostHost SwitchSwitch

Optical TapPort Mirror

HostHostHostHost

SimulatedNefarious Traffic

SaturatingTraffic (~10Gb/s)

DAG Results•Circular Buffer started overflowing

~5Gb/s (could likely be tuned better)

•Not a generic network interface (Either use the provided dag* utilities or a special version of libpcap)

•Only one tool can be used at a time

P-Series Results•Able to handle full rate (~10Gb/s)• Interface presented as generic

interface (i.e. can run Bro, Snort, and tcpdump simultaneously)

•Supports too few snort rules (700 shared between 2 channels)... you have to choose well

•Long compile time (long test cycles)

Our Decision• The DAG 6.2SE is way too expensive for what

you get. We could not afford to use it on 5+ links

• The Force10 P-Series had the best strategy and would scale best to fit our needs. Although the card doubled in price, the next generation is slated to have stateful firewall features, more real estate, and a PCI-X (should be PCIe) interface. This makes for a very cost effective, flexible, firewall, IPS, and protocol analysis solution.

Working Around the Rule Limitation

• Send known low-rate traffic (ICMP, DNS, HTTP, etc.) to the host CPU to be compared against full complement of Snort rules.

• Send the first few packets of every connection to the host CPU to be compared against full compliment of Snort rules (either via state register or through the API).

• Use the rules on the card for high-rate traffic.

Final Setup

•3U Dual 2.8Ghz Opteron•8 GB RAM•3TB of internal RAID 5 storage•2 P-Series cards (room for a third)

Final Testing

BorderBorderRouterRouter

HostHost

P-SeriesP-Series HostHostSwitchSwitch

Saturating Traffic (~9Gb/s)

“Real” Internet Traffic

Conclusion•The Force10 P-Series takes a good

approach to the problem. It allows us to secure and monitor several 10G links for a reasonable price. The next generation is even more promising allowing the merging of IPS with firewalling capabilities.

Questions?