an introduction to traffic analysis:
TRANSCRIPT
11/01/12 2
Who am I?
Jon Schipp
Unix Admin
Linux & Unix User Group
Southern Indiana Computer Klub
11/01/12 3
Why do we capture packets?
Network issues/Debugging
Attack detection
Record keeping
Fun
11/01/12 4
Obtaining Network Traffic
• routers, firewalls, intermediate devices
• monitor/SPAN ports, port mirroring
• taps
• ip forwarding/relaying/tunneling
11/01/12 5
Network Interface Cards
Get a quality card
NAPI is good (New API), improved driver framework for packet processing.
Direct Memory Access (DMA) is good
Intel PRO/ MT models are generally good
11/01/12 6
Typical Frame Processing
Frame reaches NIC Ethernet preamble is removed If interface is set in promiscuous mode, take in all frames Else, only process when dst MAC is me (unicast), or broadcast, or multicast (if on) FIFO to kernel ring buffer, CPU intervention or DMA NIC generates an interrupt, interrupt handler is called FCS is calculated, if bad, dropped Passed to host stack → ip_input → tcp/udp module → userspace
11/01/12 7
Frame Processing
[http://www.ece.rice.edu/~willmann/teng_nics_hownicswork.html]
11/01/12 8
FreeBSD Packet Processing
[www.net.t-labs.tu-berlin.de/~fabian/papers/da.pdf]
11/01/12 9
FreeBSD Processing cont.
3 copies due to double buffer
Deals with smaller buffers compared to Linux
Half of the double buffer is copied to user space
Packet is passed to each BPF device, /dev/bpf[0-9] (where application via libpcap binds to)
Application reads from HOLD buffer, data is copied from the STORE buffer into the HOLD buffer
11/01/12 10
mbuf kernel structure
FreeBSD - data and headers are stored in mbufs and mbuf clusters
man mbuf: The total size of an mbuf, MSIZE, is a constant defined in <sys/param.h>.
Mbuf usage:
Sysctl value & description for mbuf clusters:
Show size and limit of an mbuf cluster with vmstat:
11/01/12 11
Linux Frame Processing
[www.net.t-labs.tu-berlin.de/~fabian/papers/da.pdf]
11/01/12 12
Linux Processing cont.
2 copies
Deals with larger buffers compared to FreeBSD Smart queue, pointers
Packets copied individually, not whole buffers full of packets
If packets are available, wake up user spacer(libpcap based) application
11/01/12 13
sk_buff kernel structure
Linux - data and headers are stored in sk_buffs /usr/include/linux/skbuff.h
11/01/12 14
Keeping Up?
Device Polling
NAPI Interrupt Mitigation
Packet Throttling
Shared memory, mmap() , and Zero-Copy
PF_RING & netmap
11/01/12 15
Capture Mechanisms/Sockets
Berkeley Packet Filter (BPF)
Filter packets before they get to user space
Linux Socket Filter (LSF) Extended BPF (kinda) and PF_RING (Linux)
Others: CSPF, NDIS, xPF, MPF, DPF, Swift and so on...
11/01/12 16
libpcap
C library for packet capture Runs on almost all the modern unices winpcap for windows
When data reaches user space, it's stored in the libpcap buffer
Provides link layer access to data available on the network through interfaces attached to the system.
11/01/12 17
tcpdump tests, average
6,000,000 packets in 60 seconds using iperf, loss OS defaults, hardware: Dell PowerEdge 2850, Xeon (Quad), 4GB RAM tcpdump -nni em0 -w test96.pcap | FreeBSD: 0%, Linux: 8%
tcpdump -nni em0 -w /dev/null | FreeBSD: 0%, Linux: 0%
tcpdump -nni em0 -s0 -w test65535.pcap | FreeBSD: 1.6%, Linux: 22% tcpdump -nni em0 -s0 /dev/null | FreeBSD: 0%, Linux: .02%
11/01/12 18
libpcap buffer
(FreeBSD) libpcap library initializes libpcap buffer to 32kb, if bpf value is less than 32KB
if ((ioctl (fd, BIOCGBLEN, (caddr_t)&v) < 0) || v < 32768) v = 32768; Linux initializes its buffer size at 512KB Increase BPF buffer size globally, for all apps net.bpf.bufsize, net.bpf.maxbufsize
Libpcap will initialize its buffer to size in net.bpf.bufsize
Set buffer for tcpdump instance, use -B 524288 (512kb)
11/01/12 19
FreeBSD, packet drops
netstat
Drop count from NIC's em0 driver:
Drops reported by BPF (out of space):
Example source lines for receiving drop count:
Interface stats and drop counts:
11/01/12 20
Linux, packet drops
static int get_dev_fields(char *bp, struct interface *ife){ switch (procnetdev_vsn) { case 3: sscanf(bp, "%llu %llu %lu %lu %lu %lu %lu", &ife->stats.rx_bytes, &ife->stats.rx_packets, &ife->stats.rx_errors, &ife->stats.rx_dropped,
...
ifconfigDrops reported by kernel (out of space):
Drops reported by NIC, NIC dependent: $ ethtool -S eth0
11/01/12 21
tcpdump/libpcap drops
“Packets captured” – Packets processed by tcpdump “Received by filter” – Passed the filter (LSF, BPF) “Dropped by kernel” - Not enough space in kernel buffer FreeBSD (kernel drops):
libpcap gets its drop count from the kernel (BPF)
ps_drop from pcap_stats() is bs_drop from BIOCGSTATS
Linux (kernel drops)
libpcap gets its drop count from PF_PACKET’s PACKET_STATISTICS
ps_drop from pcap_stats()
ps_ifdrop – Ubuntu addendum/patch (Linux , Tru64 Unix only) from /proc/net/dev
“Dropped by interface”
11/01/12 22
Reporting & Stats
Measure net load, drop count, packets per second, bits per second etc.
Find your network baseline; what you are currently handling
Tools:
tcpstat - report network interface statistics
ifpps - fetch and format kernel network statistics
atsar - system activity report
11/01/12 23
tcpstat
$ tcpstat -i eth0 -o "Time: %S\tpps: %p\tpacket count: %n\tnet load: %l\tBps: %B\n"
11/01/12 27
Stress Testing
Network performance and capture efficacy testing
Can I write 1Gbit line rate to disk?
Tools:
hping2 - send any (almost) arbitrary TCP/IP packets to network hosts
trafgen - a high-performance zero-copy network packet generator
iperf - perform network throughput tests ( not shown )
11/01/12 28
Packets Per Second
[http://www.cisco.com/web/about/security/intelligence/network_performance_metrics.html]
[1,000,000,000 b/s / (84 B * 8 b/B)] == 1,488,096 f/s (maximum rate)[1,000,000,000 b/s / (1,538 B * 8 b/B)] == 81,274 f/s (minimum rate)
11/01/12 29
Gigabit Line Rate for UDP
[http://www.cisco.com/web/about/security/intelligence/network_performance_metrics.html]
[http://wiki.networksecuritytoolkit.org/nstwiki/index.php/LAN_Ethernet_Maximum_Rates,_Generation,_Capturing_%26_Monitoring]
11/01/12 30
hping3
Example stats from sender with ifpps:
Example stats from sender with tcpstat:
Start a UDP flood with hping3, min size without data:
[http://www.hping.org/]
11/01/12 31
hping3
Start a UDP flood with hping3 with data from file:
[http://www.hping.org/]
Packet dump:
11/01/12 32
trafgen
Linux, a zero-copy traffic generator
Uses PF_Packet's TX_RING extension i.e. a virtual memory ring buffer, that is directly mapped between kernel & user space
Permission from Daniel Borkmann [http://netsniff-ng.org/]
11/01/12 33
trafgen config files
$P1 { # Dst MAC -> work 192.168.1.3 0xc4,0x2c,0x03,0x0b,0x65,0x80 # Src MAC 0x90,0xe6,0xba,0x70,0xbd,0x0a # Proto 0x08, 0x00,# Network Layer - IPv4 # IP Version & IHL 0x45, # Type of Service 0x00, # Total Length 0x00,0x1c, # Identification 0x6a,0xae, # IP Flags (3 bits) & Fragment Offset 0x00,0x00, # TTL 0x40, # Protocol 0x11, # Header Checksum 0x8c,0xa6, # IP Source Address 0xc0,0xa8,0x01,0x29, # IP Destination Address 0xc0,0xa8,0x01,0x03, # UDP - Source Port 0x05,0x32, # UDP Destination Port 0x05,0x39, # Length 0x00,0x08, # UDP Checksum 0x71,0xf6, # Padding & Data 0x00,0x00,0x00,0x00,0x00,0x00, 0x00,0x00,0x00,0x00,0x00,0x00,}
11/01/12 34
trafgen config files
$P1 { # Dst MAC -> work 192.168.1.3 0xc4,0x2c,0x03,0x0b,0x65,0x80 # Src MAC 0x90,0xe6,0xba,0x70,0xbd,0x0a # Proto 0x08, 0x00, # IP Version & IHL 0x45, # Type of Service 0x00, # Total Length 0x00,0x1c, # Identification 0x6a,0xae, # IP Flags (3 bits) & Fragment Offset 0x00,0x00, # TTL 0x40, # Protocol 0x06, # Header Checksum 0x40,0x46, # IP Source Address 0xc0,0xa8,0x01,0x29, # IP Destination Address 0xc0,0xa8,0x01,0x03, # TCP - Source Port 0x08,0x7f, # TCP Destination Port 0x00,0x50, # Sequence Number 0x59,0x3d,0xa6,0xde, # Acknowledgement Number 0x2e,0x5c,0x0d,0xae, # Offset & Reserved 0x50, # TCP Flags 0x02, # Window 0x02,0x00, # Checksum 0xe5,0x70, # Urgent Pointer 0x00,0x00, # Padding & Data # 0x00,0x00,0x00,0x00,0x00,0x00}
1.)
2.)
3.)
11/01/12 36
Capture
Collecting the data and writing it to disk
Can we handle it all?
Tools:
bpf filters - a packet filtering language
netsniff-ng - a high-performance zero-copy capturing program
tcpdump – the de facto command-line packet capturing tool
11/01/12 37
BPF filters
Examples:
Basic Filters:
Hosts:
ether aa:bb:cc:dd:eeether src aa:bb:cc:dd:eeether dst aa:bb:cc:dd:eehost 192.168.1.1src host 192.168.1.1dst host 192.168.1.1
Ports:
port 80src port 80dst port 25portrange 0-1023
Network:
net 192.168.1.0/24src net 192.168.1.0/24dst net 192.168.1.0/24
Protocol:
arpipip6tcpudpicmp
Advanced Filters:
tcp[13] = 0x02tcp[13] & 2 = 2ip[12:4] = ip[16:4]
ip and tcp and port 80 and dst host (192.168.1.1 or 192.168.1.2)
icmp and ether dst host 00:01:02:03:04:05
Combinations:
Size:
less 64greater 500
udp port 53 and not src net (192.168.1.0/24 or 192.168.2.0/24)
port 25 and tcp[20:4] = 0x4d41494cport 80 and tcp[32:4] = 0x47455420
11/01/12 40
BPF Filters – 3
[http://www.visi.com/~mjb/Drawings]
Compare protocol field ( udp = 0x11 )Load Halfword from IP ID field ( bitwise & to detect fragmentation)Load 1 byte from offset 14 ( IHL )Calculate IP header length0101 AND1111 (0x0f) ------0101 ( 5 * 4 = 20 bytes) IP header size^ value stored as xld halfword, dst port = [ x + 16 ] = [36] byte offset [36] = 0x35 = 53 decimanl
11/01/12 41
Capture SYN &
[http://www.visi.com/~mjb/Drawings]
# tcpdump -d 'tcp[13] & 2 = 2' >/dev/null | grep -B 1 -A 2 0x2
# tcpdump -nnr 05-11-2012_12\:30_eth3.pcap -c 3 'tcp[13] & 2 = 2' | grep -E '(S|S\.)'
11/01/12 42
IP Options: RR Example
[http://www.visi.com/~mjb/Drawings]
# tcpdump -Xvvnni eth5 'ip[0] & 0x0f > 5'
# ping -R 192.168.1.1 -c 1
11/01/12 43
Capture HTTP GET Method
[http://www.visi.com/~mjb/Drawings]
# tcpdump -Xnnr ~jon/mypcaps/05-11-2012\:30_eth3.pcap -c 3 -s 96 'port 80 and tcp[32:4] = 0x47455420'
# printf '\x47\x45\x54\x20\n' | hexdump -c
11/01/12 44
netsniff-ng
Linux, libpcap independent, zero-copy mechanism
Kernel must be compiled with CONFIG_PACKET_MMAP
Uses PF_PACKET's RX_RING buffer when receiving
Uses PF_PACKET's TX_RING buffer when sending (replay)
Permission from Daniel Borkmann [http://netsniff-ng.org/]
11/01/12 45
netsniff-ng: what i do
Capture and write with netsniff-ng and analyze later with other tools
An easy to use, high-speed alternative to entire shared memory systems suchas PF_RING and netmap
Permission from Daniel Borkmann [http://netsniff-ng.org/]
Fault: At the moment, writing BPF filters for netsniff-ng is tedious
11/01/12 46
netsniff-ng: a quick look
$ netsniff-ng –dev –num 1 –ring-size 50MB -b 0 -H
11/01/12 47
netsniff-ng: writing to disk
Pcaps are written to disk in unix epoch time:
Write a new pcap to disk every 60 seconds:
11/01/12 49
tcpdump
Dump ethernet header ( -e ), everything in hex and ascii ( -XX ) and grab only the first 96 bytes of each by setting the snap length ( -s ):
Write 10 packet to disk ( -c ) and do not resolve port numbers and name ( -nn ), write to file test.pcap ( -w ):
[http://www.tcpdump.org/]
11/01/12 50
tcpdump & libpcap
Runs on most OSs, uses libpcap, mmap'd for libpcap versions 1.0+
Kernel must be compiled with CONFIG_PACKET_MMAP, should be default in most if not all Linux kernel's 2.6+. For FreeBSD 8.0+, set $ sysctl net.bpf.zerocopy_enable=1
libpcap provides the only full fledged BPF compiler
Most packet capturing tools use libpcap for low-level network access
[http://www.tcpdump.org/]
11/01/12 51
Analysis
Analyzing the data that we have collected
Making sense of it
Tools:
ntop – a web-based traffic monitoring tool with many graphs
iftop – shows data rate and other metrics per connection
tcpflow – a tcp/ip session reassembler
tcpick – a tcp stream sniffer and connection tracker
speedometer – measure and display rate of data across an interface
snort – A free lightweight network intrusion detection system
11/01/12 52
ntop
ntop -d -L -u ntop –access-log-file=/var/log/ntop/access.log -b -C –output-packet-path=/var/log/ntop-suspicious.log –local-subnets 192.168.1.0/24,192.168.2.0/24,192.168.3.0/24 -o -M -p /etc/ntop/protocol.list -i br0,eth0,eth1,eth2,eth3,eth4,eth5 -o /var/log/ntop
[http://www.ntop.org/products/ntop/]
11/01/12 53
snort
[www.snort.org/]
# snort -r 05-11-2012_12\:30_eth3.pcap -c /etc/snort/snort.read.conf -l .
Read file ( -r ),use configuration file ( -c ), write alerts to the cwd ( -l ),
Files:
11/01/12 54
iftop
Find bandwidth hogs
Per connection bandwidth statistics
BPF filters via libpcap and an easy to use regex screen filter
iftop - display bandwidth usage on an interface by host
Fault: inability to read pcaps
11/01/12 55
iftop
iftop - display bandwidth usage on an interface by host One connection displayed per line
11/01/12 56
iftop
Interactive: press “h” to cycle through views, traffic show in both directions, per connection, one line each
11/01/12 57
iftop
$ iftop -i eth0 -F 192.168.1.0/255.255.255.0
Show traffic originating from network 192.168.1.0/24 to any not from 192.168.1.0/24
11/01/12 58
iftop
$ iftop -i eth0 -f 'port (80 or 443)' $ iftop -i eth0 -f 'ip dst 192.168.1.5'
Example BPF filters
11/01/12 59
iftop – screen filter & config file
Press the “l” key to a set a screen filter with regex
Configuration file: ~/.iftoprc
11/01/12 60
tcpflow
[http://www.circlemud.org/jelson/software/tcpflow/]
a tcp/ip session reassembler:$ tcpflow -i eth2 -e -c 'port 25'
11/01/12 61
tcpflow
[http://www.circlemud.org/jelson/software/tcpflow/]
a tcp/ip session reassembler:
$ file ./*
# tcpflow -i eth0 -b 96 -e -c port 80Color ( -e ), stdout ( -c ), snap length ( -b )
11/01/12 62
tcpick
tcp stream sniffer and connection tracker
[http://tcpick.sourceforge.net/]
# tcpick -r 05-11-2012_12\:30 eth3.pcap -C -h -yP -e 15 "port ( 21 or 20 )"
Read file ( -r ),color output ( -C ), display ports/ip/flags ( -h ),print data to stdout ( -yP ), packet count ( -e ), and set BPF filter
11/01/12 63
tcpick
tcp stream sniffer and connection tracker
[http://tcpick.sourceforge.net/]
# tcpick -r 05-11-2012_12\:30 eth3.pcap -C -h -wR -e 10 "port 25"
Read file ( -r ),color output ( -C ), display ports/ip/flags ( -h ),write to cwd ( -wR ),packet count ( -e ), and set BPF filter
Server and Client flows:
11/01/12 64
speedometer
speedometer is a simple bandwidth utilization sensing tool that displays the current throughput usage in a moving bar graph fashion.
$ speedometer -rx eth0
11/01/12 65
speedometer
Speedometer can handle multiple interfaces at once
$ speedometer -rx eth0 -rx eth2 -rx eth3 -rx eth4 $ speedometer -rx eth0 -c -rx eth2 -c -rx -eth3 -c -rx -eth4
11/01/12 66
ngrep – network grep
$ ngrep -S 49 -qI 05-11-2012_12\:30 eth3.pcap “GET .*.jpg” 'port 80' -n 3
Displays the first 49 bytes ( -S ) of packet, quiet mode ( -q ), read input from file ( -I ),grab first 3 packets ( -n )
Note: ( -S ) is not the same as the snap length ( -s ) which specifies the size to capture.
11/01/12 67
ngrep – web traffic log
# ngrep -I bad_user.pcap -q -W single -t "GET" ip src 192.168.1.1 | awk '{ print $2, $3, $11, $9}' | sed 's/\.\{1,3\}User-Agent//' | grep -v -E '(ad|cache|analytics|wxdata|voicefive|imwx|weather.com|counterpath|cloudfront|2mdn.net|click|api|acuity|tribal|pixel|touchofclass|flickr|ytimg|pulse|twitter|facebook|graphic|revsci|digi|rss|cdn|brightcove|atdmt|btrll|metric|content|trend|serv|content|global|fwmrm|typekit|[a-z]*-[a-z]*\.com|pinit|cisco|tumblr)' | sed '/ [ \t]*$/d' > url.txt
11/01/12 68
Contact
Questions, suggestions, polite criticism: [email protected]
More info:
sickbits.networklabs.org/other/packetcapt dclinux.org