an introduction to traffic analysis:

11/01/12 1

An Introduction to Traffic Analysis: A Pragmatic Approach

jonschipp@gmail.com

11/01/12 2

Who am I?

jonschipp@gmail.com

Jon Schipp

Unix Admin

Linux & Unix User Group

Southern Indiana Computer Klub

11/01/12 3

Why do we capture packets?

jonschipp@gmail.com

Network issues/Debugging

Attack detection

Record keeping

Fun

11/01/12 4

Obtaining Network Traffic

jonschipp@gmail.com

• routers, firewalls, intermediate devices

• monitor/SPAN ports, port mirroring

• taps

• ip forwarding/relaying/tunneling

11/01/12 5

Network Interface Cards

jonschipp@gmail.com

Get a quality card

NAPI is good (New API), improved driver framework for packet processing.

Direct Memory Access (DMA) is good

Intel PRO/ MT models are generally good

11/01/12 6

Typical Frame Processing

jonschipp@gmail.com

Frame reaches NIC Ethernet preamble is removed If interface is set in promiscuous mode, take in all frames Else, only process when dst MAC is me (unicast), or broadcast, or multicast (if on) FIFO to kernel ring buffer, CPU intervention or DMA NIC generates an interrupt, interrupt handler is called FCS is calculated, if bad, dropped Passed to host stack → ip_input → tcp/udp module → userspace

11/01/12 7

Frame Processing

jonschipp@gmail.com

[http://www.ece.rice.edu/~willmann/teng_nics_hownicswork.html]

11/01/12 8

FreeBSD Packet Processing

jonschipp@gmail.com

[www.net.t-labs.tu-berlin.de/~fabian/papers/da.pdf]

11/01/12 9

FreeBSD Processing cont.

jonschipp@gmail.com

3 copies due to double buffer

Deals with smaller buffers compared to Linux

Half of the double buffer is copied to user space

Packet is passed to each BPF device, /dev/bpf[0-9] (where application via libpcap binds to)

Application reads from HOLD buffer, data is copied from the STORE buffer into the HOLD buffer

11/01/12 10

mbuf kernel structure

jonschipp@gmail.com

FreeBSD - data and headers are stored in mbufs and mbuf clusters

man mbuf: The total size of an mbuf, MSIZE, is a constant defined in <sys/param.h>.

Mbuf usage:

Sysctl value & description for mbuf clusters:

Show size and limit of an mbuf cluster with vmstat:

11/01/12 11

Linux Frame Processing

jonschipp@gmail.com

[www.net.t-labs.tu-berlin.de/~fabian/papers/da.pdf]

11/01/12 12

Linux Processing cont.

jonschipp@gmail.com

2 copies

Deals with larger buffers compared to FreeBSD Smart queue, pointers

Packets copied individually, not whole buffers full of packets

If packets are available, wake up user spacer(libpcap based) application

11/01/12 13

sk_buff kernel structure

jonschipp@gmail.com

Linux - data and headers are stored in sk_buffs /usr/include/linux/skbuff.h

11/01/12 14

Keeping Up?

jonschipp@gmail.com

Device Polling

NAPI Interrupt Mitigation

Packet Throttling

Shared memory, mmap() , and Zero-Copy

PF_RING & netmap

11/01/12 15

Capture Mechanisms/Sockets

jonschipp@gmail.com

Berkeley Packet Filter (BPF)

Filter packets before they get to user space

Linux Socket Filter (LSF) Extended BPF (kinda) and PF_RING (Linux)

Others: CSPF, NDIS, xPF, MPF, DPF, Swift and so on...

11/01/12 16

libpcap

jonschipp@gmail.com

C library for packet capture Runs on almost all the modern unices winpcap for windows

When data reaches user space, it's stored in the libpcap buffer

Provides link layer access to data available on the network through interfaces attached to the system.

11/01/12 17

tcpdump tests, average

jonschipp@gmail.com

6,000,000 packets in 60 seconds using iperf, loss OS defaults, hardware: Dell PowerEdge 2850, Xeon (Quad), 4GB RAM tcpdump -nni em0 -w test96.pcap | FreeBSD: 0%, Linux: 8%

tcpdump -nni em0 -w /dev/null | FreeBSD: 0%, Linux: 0%

tcpdump -nni em0 -s0 -w test65535.pcap | FreeBSD: 1.6%, Linux: 22% tcpdump -nni em0 -s0 /dev/null | FreeBSD: 0%, Linux: .02%

11/01/12 18

libpcap buffer

jonschipp@gmail.com

(FreeBSD) libpcap library initializes libpcap buffer to 32kb, if bpf value is less than 32KB

if ((ioctl (fd, BIOCGBLEN, (caddr_t)&v) < 0) || v < 32768) v = 32768; Linux initializes its buffer size at 512KB Increase BPF buffer size globally, for all apps net.bpf.bufsize, net.bpf.maxbufsize

Libpcap will initialize its buffer to size in net.bpf.bufsize

Set buffer for tcpdump instance, use -B 524288 (512kb)

11/01/12 19

FreeBSD, packet drops

jonschipp@gmail.com

netstat

Drop count from NIC's em0 driver:

Drops reported by BPF (out of space):

Example source lines for receiving drop count:

Interface stats and drop counts:

11/01/12 20

Linux, packet drops

jonschipp@gmail.com

static int get_dev_fields(char *bp, struct interface *ife){ switch (procnetdev_vsn) { case 3: sscanf(bp, "%llu %llu %lu %lu %lu %lu %lu", &ife->stats.rx_bytes, &ife->stats.rx_packets, &ife->stats.rx_errors, &ife->stats.rx_dropped,

...

ifconfigDrops reported by kernel (out of space):

Drops reported by NIC, NIC dependent: $ ethtool -S eth0

11/01/12 21

tcpdump/libpcap drops

jonschipp@gmail.com

“Packets captured” – Packets processed by tcpdump “Received by filter” – Passed the filter (LSF, BPF) “Dropped by kernel” - Not enough space in kernel buffer FreeBSD (kernel drops):

libpcap gets its drop count from the kernel (BPF)

ps_drop from pcap_stats() is bs_drop from BIOCGSTATS

Linux (kernel drops)

libpcap gets its drop count from PF_PACKET’s PACKET_STATISTICS

ps_drop from pcap_stats()

ps_ifdrop – Ubuntu addendum/patch (Linux , Tru64 Unix only) from /proc/net/dev

“Dropped by interface”

11/01/12 22

Reporting & Stats

jonschipp@gmail.com

Measure net load, drop count, packets per second, bits per second etc.

Find your network baseline; what you are currently handling

Tools:

tcpstat - report network interface statistics

ifpps - fetch and format kernel network statistics

atsar - system activity report

11/01/12 23

tcpstat

jonschipp@gmail.com

$ tcpstat -i eth0 -o "Time: %S\tpps: %p\tpacket count: %n\tnet load: %l\tBps: %B\n"

11/01/12 24

ifpps

jonschipp@gmail.com

# ifpps –dev eth0

11/01/12 25

sar

jonschipp@gmail.com

sar – system activity report

$ sar -l 5 5 $ sar -L 5 5

11/01/12 26

graphing with gnuplot

jonschipp@gmail.com

[http://www.frenchfries.net/paul/tcpstat/]

11/01/12 27

Stress Testing

jonschipp@gmail.com

Network performance and capture efficacy testing

Can I write 1Gbit line rate to disk?

Tools:

hping2 - send any (almost) arbitrary TCP/IP packets to network hosts

trafgen - a high-performance zero-copy network packet generator

iperf - perform network throughput tests ( not shown )

11/01/12 28

Packets Per Second

jonschipp@gmail.com

[http://www.cisco.com/web/about/security/intelligence/network_performance_metrics.html]

[1,000,000,000 b/s / (84 B * 8 b/B)] == 1,488,096 f/s (maximum rate)[1,000,000,000 b/s / (1,538 B * 8 b/B)] == 81,274 f/s (minimum rate)

11/01/12 29

Gigabit Line Rate for UDP

jonschipp@gmail.com

[http://www.cisco.com/web/about/security/intelligence/network_performance_metrics.html]

[http://wiki.networksecuritytoolkit.org/nstwiki/index.php/LAN_Ethernet_Maximum_Rates,_Generation,_Capturing_%26_Monitoring]

11/01/12 30

hping3

jonschipp@gmail.com

Example stats from sender with ifpps:

Example stats from sender with tcpstat:

Start a UDP flood with hping3, min size without data:

[http://www.hping.org/]

11/01/12 31

hping3

jonschipp@gmail.com

Start a UDP flood with hping3 with data from file:

[http://www.hping.org/]

Packet dump:

11/01/12 32

trafgen

jonschipp@gmail.com

Linux, a zero-copy traffic generator

Uses PF_Packet's TX_RING extension i.e. a virtual memory ring buffer, that is directly mapped between kernel & user space

Permission from Daniel Borkmann [http://netsniff-ng.org/]

11/01/12 33

trafgen config files

jonschipp@gmail.com

$P1 { # Dst MAC -> work 192.168.1.3 0xc4,0x2c,0x03,0x0b,0x65,0x80 # Src MAC 0x90,0xe6,0xba,0x70,0xbd,0x0a # Proto 0x08, 0x00,# Network Layer - IPv4 # IP Version & IHL 0x45, # Type of Service 0x00, # Total Length 0x00,0x1c, # Identification 0x6a,0xae, # IP Flags (3 bits) & Fragment Offset 0x00,0x00, # TTL 0x40, # Protocol 0x11, # Header Checksum 0x8c,0xa6, # IP Source Address 0xc0,0xa8,0x01,0x29, # IP Destination Address 0xc0,0xa8,0x01,0x03, # UDP - Source Port 0x05,0x32, # UDP Destination Port 0x05,0x39, # Length 0x00,0x08, # UDP Checksum 0x71,0xf6, # Padding & Data 0x00,0x00,0x00,0x00,0x00,0x00, 0x00,0x00,0x00,0x00,0x00,0x00,}

11/01/12 34

trafgen config files

jonschipp@gmail.com

$P1 { # Dst MAC -> work 192.168.1.3 0xc4,0x2c,0x03,0x0b,0x65,0x80 # Src MAC 0x90,0xe6,0xba,0x70,0xbd,0x0a # Proto 0x08, 0x00, # IP Version & IHL 0x45, # Type of Service 0x00, # Total Length 0x00,0x1c, # Identification 0x6a,0xae, # IP Flags (3 bits) & Fragment Offset 0x00,0x00, # TTL 0x40, # Protocol 0x06, # Header Checksum 0x40,0x46, # IP Source Address 0xc0,0xa8,0x01,0x29, # IP Destination Address 0xc0,0xa8,0x01,0x03, # TCP - Source Port 0x08,0x7f, # TCP Destination Port 0x00,0x50, # Sequence Number 0x59,0x3d,0xa6,0xde, # Acknowledgement Number 0x2e,0x5c,0x0d,0xae, # Offset & Reserved 0x50, # TCP Flags 0x02, # Window 0x02,0x00, # Checksum 0xe5,0x70, # Urgent Pointer 0x00,0x00, # Padding & Data # 0x00,0x00,0x00,0x00,0x00,0x00}

1.)

2.)

3.)

11/01/12 35

trafgen – packet generation

jonschipp@gmail.com

11/01/12 36

Capture

jonschipp@gmail.com

Collecting the data and writing it to disk

Can we handle it all?

Tools:

bpf filters - a packet filtering language

netsniff-ng - a high-performance zero-copy capturing program

tcpdump – the de facto command-line packet capturing tool

11/01/12 37

BPF filters

jonschipp@gmail.com

Examples:

Basic Filters:

Hosts:

ether aa:bb:cc:dd:eeether src aa:bb:cc:dd:eeether dst aa:bb:cc:dd:eehost 192.168.1.1src host 192.168.1.1dst host 192.168.1.1

Ports:

port 80src port 80dst port 25portrange 0-1023

Network:

net 192.168.1.0/24src net 192.168.1.0/24dst net 192.168.1.0/24

Protocol:

arpipip6tcpudpicmp

Advanced Filters:

tcp[13] = 0x02tcp[13] & 2 = 2ip[12:4] = ip[16:4]

ip and tcp and port 80 and dst host (192.168.1.1 or 192.168.1.2)

icmp and ether dst host 00:01:02:03:04:05

Combinations:

Size:

less 64greater 500

udp port 53 and not src net (192.168.1.0/24 or 192.168.2.0/24)

port 25 and tcp[20:4] = 0x4d41494cport 80 and tcp[32:4] = 0x47455420

11/01/12 38

BPF Filters – 1

jonschipp@gmail.com

11/01/12 39

BPF Filters – 2

jonschipp@gmail.com

11/01/12 40

BPF Filters – 3

jonschipp@gmail.com

[http://www.visi.com/~mjb/Drawings]

Compare protocol field ( udp = 0x11 )Load Halfword from IP ID field ( bitwise & to detect fragmentation)Load 1 byte from offset 14 ( IHL )Calculate IP header length0101 AND1111 (0x0f) ------0101 ( 5 * 4 = 20 bytes) IP header size^ value stored as xld halfword, dst port = [ x + 16 ] = [36] byte offset [36] = 0x35 = 53 decimanl

11/01/12 41

Capture SYN &

jonschipp@gmail.com

[http://www.visi.com/~mjb/Drawings]

# tcpdump -d 'tcp[13] & 2 = 2' >/dev/null | grep -B 1 -A 2 0x2

# tcpdump -nnr 05-11-2012_12\:30_eth3.pcap -c 3 'tcp[13] & 2 = 2' | grep -E '(S|S\.)'

11/01/12 42

IP Options: RR Example

jonschipp@gmail.com

[http://www.visi.com/~mjb/Drawings]

# tcpdump -Xvvnni eth5 'ip[0] & 0x0f > 5'

# ping -R 192.168.1.1 -c 1

11/01/12 43

Capture HTTP GET Method

jonschipp@gmail.com

[http://www.visi.com/~mjb/Drawings]

# tcpdump -Xnnr ~jon/mypcaps/05-11-2012\:30_eth3.pcap -c 3 -s 96 'port 80 and tcp[32:4] = 0x47455420'

# printf '\x47\x45\x54\x20\n' | hexdump -c

11/01/12 44

netsniff-ng

jonschipp@gmail.com

Linux, libpcap independent, zero-copy mechanism

Kernel must be compiled with CONFIG_PACKET_MMAP

Uses PF_PACKET's RX_RING buffer when receiving

Uses PF_PACKET's TX_RING buffer when sending (replay)

Permission from Daniel Borkmann [http://netsniff-ng.org/]

11/01/12 45

netsniff-ng: what i do

jonschipp@gmail.com

Capture and write with netsniff-ng and analyze later with other tools

An easy to use, high-speed alternative to entire shared memory systems suchas PF_RING and netmap

Permission from Daniel Borkmann [http://netsniff-ng.org/]

Fault: At the moment, writing BPF filters for netsniff-ng is tedious

11/01/12 46

netsniff-ng: a quick look

jonschipp@gmail.com

$ netsniff-ng –dev –num 1 –ring-size 50MB -b 0 -H

11/01/12 47

netsniff-ng: writing to disk

jonschipp@gmail.com

Pcaps are written to disk in unix epoch time:

Write a new pcap to disk every 60 seconds:

11/01/12 48

netsniff-ng: creating filters

jonschipp@gmail.com

1.)

2.)

11/01/12 49

tcpdump

jonschipp@gmail.com

Dump ethernet header ( -e ), everything in hex and ascii ( -XX ) and grab only the first 96 bytes of each by setting the snap length ( -s ):

Write 10 packet to disk ( -c ) and do not resolve port numbers and name ( -nn ), write to file test.pcap ( -w ):

[http://www.tcpdump.org/]

11/01/12 50

tcpdump & libpcap

jonschipp@gmail.com

Runs on most OSs, uses libpcap, mmap'd for libpcap versions 1.0+

Kernel must be compiled with CONFIG_PACKET_MMAP, should be default in most if not all Linux kernel's 2.6+. For FreeBSD 8.0+, set $ sysctl net.bpf.zerocopy_enable=1

libpcap provides the only full fledged BPF compiler

Most packet capturing tools use libpcap for low-level network access

[http://www.tcpdump.org/]

11/01/12 51

Analysis

jonschipp@gmail.com

Analyzing the data that we have collected

Making sense of it

Tools:

ntop – a web-based traffic monitoring tool with many graphs

iftop – shows data rate and other metrics per connection

tcpflow – a tcp/ip session reassembler

tcpick – a tcp stream sniffer and connection tracker

speedometer – measure and display rate of data across an interface

snort – A free lightweight network intrusion detection system

11/01/12 52

ntop

jonschipp@gmail.com

ntop -d -L -u ntop –access-log-file=/var/log/ntop/access.log -b -C –output-packet-path=/var/log/ntop-suspicious.log –local-subnets 192.168.1.0/24,192.168.2.0/24,192.168.3.0/24 -o -M -p /etc/ntop/protocol.list -i br0,eth0,eth1,eth2,eth3,eth4,eth5 -o /var/log/ntop

[http://www.ntop.org/products/ntop/]

11/01/12 53

snort

jonschipp@gmail.com

[www.snort.org/]

# snort -r 05-11-2012_12\:30_eth3.pcap -c /etc/snort/snort.read.conf -l .

Read file ( -r ),use configuration file ( -c ), write alerts to the cwd ( -l ),

Files:

11/01/12 54

iftop

jonschipp@gmail.com

Find bandwidth hogs

Per connection bandwidth statistics

BPF filters via libpcap and an easy to use regex screen filter

iftop - display bandwidth usage on an interface by host

Fault: inability to read pcaps

11/01/12 55

iftop

jonschipp@gmail.com

iftop - display bandwidth usage on an interface by host One connection displayed per line

11/01/12 56

iftop

jonschipp@gmail.com

Interactive: press “h” to cycle through views, traffic show in both directions, per connection, one line each

11/01/12 57

iftop

jonschipp@gmail.com

$ iftop -i eth0 -F 192.168.1.0/255.255.255.0

Show traffic originating from network 192.168.1.0/24 to any not from 192.168.1.0/24

11/01/12 58

iftop

jonschipp@gmail.com

$ iftop -i eth0 -f 'port (80 or 443)' $ iftop -i eth0 -f 'ip dst 192.168.1.5'

Example BPF filters

11/01/12 59

iftop – screen filter & config file

jonschipp@gmail.com

Press the “l” key to a set a screen filter with regex

Configuration file: ~/.iftoprc

11/01/12 60

tcpflow

jonschipp@gmail.com

[http://www.circlemud.org/jelson/software/tcpflow/]

a tcp/ip session reassembler:$ tcpflow -i eth2 -e -c 'port 25'

11/01/12 61

tcpflow

jonschipp@gmail.com

[http://www.circlemud.org/jelson/software/tcpflow/]

a tcp/ip session reassembler:

$ file ./*

# tcpflow -i eth0 -b 96 -e -c port 80Color ( -e ), stdout ( -c ), snap length ( -b )

11/01/12 62

tcpick

jonschipp@gmail.com

tcp stream sniffer and connection tracker

[http://tcpick.sourceforge.net/]

# tcpick -r 05-11-2012_12\:30 eth3.pcap -C -h -yP -e 15 "port ( 21 or 20 )"

Read file ( -r ),color output ( -C ), display ports/ip/flags ( -h ),print data to stdout ( -yP ), packet count ( -e ), and set BPF filter

11/01/12 63

tcpick

jonschipp@gmail.com

tcp stream sniffer and connection tracker

[http://tcpick.sourceforge.net/]

# tcpick -r 05-11-2012_12\:30 eth3.pcap -C -h -wR -e 10 "port 25"

Read file ( -r ),color output ( -C ), display ports/ip/flags ( -h ),write to cwd ( -wR ),packet count ( -e ), and set BPF filter

Server and Client flows:

11/01/12 64

speedometer

jonschipp@gmail.com

speedometer is a simple bandwidth utilization sensing tool that displays the current throughput usage in a moving bar graph fashion.

$ speedometer -rx eth0

11/01/12 65

speedometer

jonschipp@gmail.com

Speedometer can handle multiple interfaces at once

$ speedometer -rx eth0 -rx eth2 -rx eth3 -rx eth4 $ speedometer -rx eth0 -c -rx eth2 -c -rx -eth3 -c -rx -eth4

11/01/12 66

ngrep – network grep

jonschipp@gmail.com

$ ngrep -S 49 -qI 05-11-2012_12\:30 eth3.pcap “GET .*.jpg” 'port 80' -n 3

Displays the first 49 bytes ( -S ) of packet, quiet mode ( -q ), read input from file ( -I ),grab first 3 packets ( -n )

Note: ( -S ) is not the same as the snap length ( -s ) which specifies the size to capture.

11/01/12 67

ngrep – web traffic log

jonschipp@gmail.com

# ngrep -I bad_user.pcap -q -W single -t "GET" ip src 192.168.1.1 | awk '{ print $2, $3, $11, $9}' | sed 's/\.\{1,3\}User-Agent//' | grep -v -E '(ad|cache|analytics|wxdata|voicefive|imwx|weather.com|counterpath|cloudfront|2mdn.net|click|api|acuity|tribal|pixel|touchofclass|flickr|ytimg|pulse|twitter|facebook|graphic|revsci|digi|rss|cdn|brightcove|atdmt|btrll|metric|content|trend|serv|content|global|fwmrm|typekit|[a-z]*-[a-z]*\.com|pinit|cisco|tumblr)' | sed '/ [ \t]*$/d' > url.txt

11/01/12 68

Contact

jonschipp@gmail.com

Questions, suggestions, polite criticism: jonschipp@gmail.com

More info:

sickbits.networklabs.org/other/packetcapt dclinux.org