First responder Course:7 Incident Scope Assessment

Phil HugginsFebruary 2004

Agenda

Description Strategy Meeting Documentation Debugging Log Processing Basic Host Analysis Rootkits Information Gathering

Description

The Scope Assessment Phase has the following goals: To confirm the existence of the incident To identify which systems (if any) are involved in the

incident To estimate the damage (if any) done to involved

systems To identify if the attack is still underway To identify the complexity of the incident To gather any other data needed to make decisions on

how to respond Sources of data:

Logs Network Monitoring Assessment Analysis

Strategy Meeting

Large or complex incidents may require an initial strategy meeting to coordinate efforts

This tends to be a more technically focused meeting than the initial team meeting discussed yesterday

This meeting will identify who is responsible for: Verifying the initial report Verifying that similar systems were not affected Watching for an additional incident Deploying additional monitoring tools

Remember the Key Ideas

Document everything (even mistakes)

Trust nothing on the suspect system Suspect systems should be modified

as little as possible Chain of Custody forms should be

generated for all evidence

Documentation: Investigator Log

Documentation: Chain of Custody

How to Debug a System

Debugging is simply “finding what’s wrong with stuff”

Obvious principles but MUST be applied

Book Recommendation: Debugging by David J. Agans, ISBN 0-

8144-7168-4 http://www.debuggingrules.com/

9 Fundamental Debugging Rules Understand the system Make it fail Quit thinking and look Divide and conquer Change one thing at a time Keep an audit trail Check the plug Get a fresh view If you didn’t fix it, it ain’t fixed.

Log Processing The goal is to find new clues and validate other

findings Using information that is already known about the

incident, consult logs for additional clues Extract logs that reference suspect systems from

devices between the gateway and the suspect systems (using grep)

If a time frame is known, extract all logs from that time on gateways and remote access devices (stone-stepping scenario)

Identify additional hosts that have similar log activity or that may have been used as a stepping stone

Generate MD5 values of extracted logs Ensure that logs from the incident timeframe are

not overwritten

Basic Host Analysis

In some cases, an analysis needs to be performed on a compromised system before a forensic acquisition occurs

The goal of this analysis is to identify the scope of damage and quickly gather additional clues

The analysis may answer: Have hiding mechanisms, such as a rootkit, been

installed Who recently logged on and from where Were log files modified What files were recently created or modified

If it is suspected that there are “time bombs” or other “traps”, then the system should be unplugged and only examined with a trusted kernel

Remember the Key Ideas (again)

Document everything The “Access Time” of files will be updated when

you view their contents, record which files you look at so those times can be explained

Send log files to an evidence server via netcat, calculate an MD5 value, and analyze that copy

Trust nothing on the suspect system Use only trusted tools from an response kit CD-

ROM Kernel Module rootkits will hide data even with

original binaries

Remember the Key Ideas (again)

Suspect systems should be modified as little as possible Use a tool such as mac-robber (http://www.sleuthkit.org/mac-

robber/index.php) or mac-daddy (www.incident-response.org) to collect the MAC times of files before they get modified during the analysis

Use a tool such as The Sleuth Kit (http://www.sleuthkit.org/index.php) to analyze the file system from the raw device (the MAC times will not be modified)

Use tools such as Afind from The Forensic Toolkit Version 2.0 (http://foundstone.com under resources and free tools) to search for recently edited files on Windows systems..

Stop schedulers from running commands on system Do not write files to the disk, it will overwrite deleted content.

Instead pipe data using netcat to the evidence server or to a floppy disk

On Evidence Server:# nc -l -p 9000 > wtmp.log

On Suspect system:# cat wtmp.log | nc -w 5 10.0.0.1 9000

Basic Host Analysis

Volatile data acquisition procedures should be done first to collect the data before it could be modified (we will cover this later) netstat ps / pslist lsof / handle / fport etc.

Examine the output (on the evidence server) for suspicious processes, open ports, and logged on users

Timeline of File Activity

All files have at least 3 times associated with them (Modified, Access, and Change)

Timelines can be created with file activity at any time

For UNIX hosts, The Sleuth Kit can collect the data from the raw device and not modify the file system

An alternative is mac-robber or mac-daddy, which will modify the access times of directories

Both approaches will send data to an evidence server where it is processed and analyzed

Create Timeline of File Activity

Sleuth Kit:# fls -f solaris -m / -r /dev/rdsk/c0t0d0s0 | nc -w 5 10.0.0.1 9000

# ils -f solaris -m /dev/rdsk/c0t0d0s0 | nc -w 5 10.0.0.1 9000

…. (repeat for each partition) mac-robber:# mac-robber / | nc -w 5 10.0.0.1 9000 mac-daddy:# perl mac-daddy.pl / | nc -w 5 10.0.0.1 9000

Create Timeline of File Activity

On the evidence server (a new file for each partition with the Sleuth Kit):

# nc -l -p 9000 > mac_1.dat Sleuth Kit and mac-robber require a

processing tool from the Sleuth Kit:# mactime -b mac_1.dat 01/01/2002 > mac_1.tl

Refer to the timeline.README document in the Sleuth Kit for details (www.sleuthkit.org)

View Timeline of File ActivityAutopsy Interface

Keyword Searching DIBS Mycroft V3

http://www.dibsusa.com/products/mycroft.html Very fast and cheap

Rootkits

Rootkits are installed by attackers to: Hide files and processes that they created Collect data (such as logins and passwords)

from the network or local system Provide a back-door method of gaining access

to the system Remove evidence of previous attack

There are two major varieties of data hiding: Classical binary modification Kernel Modules

Classical Rootkits

The original system binaries are modified to read a configuration file

The configuration file contains a list of processes or files to hide

These can be detected by comparing the MD5 value of current binary with one from a non-compromised system (change management)

In basic versions of this, running ‘strings’ on the binary will show the location of the configuration file (/dev/ptx0)

Classical Rootkits Contents of a process config file (LRK 4)2 slice22 snif2 pscan2 imp3 qd2 bs.sh3 nn3 egg.lin Contents of a file hiding config file (LRK 4)tcp.logslice2scanapaddy.awk.fakeid

Classical Rootkits

Strings of a trojaned (LRK 4) ps binary:<…>90t:u&Vh/dev/ptypNR PID STACK ESP EIP TMOUT ALARM

PID TTY MAJFLT MINFLT TRS DRS SIZE SWAP

<…> /dev/ptyp file is a regular file, not a device /dev/ptyp0, /dev/ptyp1, etc. are valid

devices

Classical Rootkit Detection

Compare MD5 values of binaries with: Trusted system with same patch level Solaris Fingerprint Database (

www.sun.com) NIST NSRL (www.nsrl.nist.gov) Linux RPM (with -V a flag)

Compare output of system binaries with trusted binaries on a CD

chkrootkit (www.chkrootkit.org) signature analysis

Kernel Module Rootkits

Kernel Module rootkits modify the kernel system call table instead of modifying the binaries

These rootkits prevent the kernel from giving information on the processes and files that are in a configuration file

These are harder to detect because the MD5 of the binaries remain constant

Kernel Module Rootkits Normally, tools like ‘ps’ and ‘ls’ use

the API to request a list of processes or files from the Kernel

Kernel Module Rootkits A rootkit goes between the Kernel and API Now, the API requests a list of processes

or files from the Rootkit, which forwards the request to the Kernel and then filters out the “hidden” data.

Basic Host Analysis - Rootkits Trojan ‘sshd’ and ‘tcpd’ servers also exist

to allow access ‘ifconfig’ can be trojaned to hide the

Promiscuous flag Padding can be added to the end of new

binaries to match the CRC value of the original Use an accepted hashing algorithm such as

MD5 or SHA-1

General Rootkit Detection

New open network ports (nmap port scan)

Promiscuous network interface (AntiSnif)

Updated patch levels Modified logs Anti Virus software

Kernel Module Rootkit Detection

Different output from ‘nmap’ than ‘netstat’

Different output from ‘ls’ than the Sleuth Kit or Encase

Carbonite chkrootkit Kstat Intrusion Prevention Systems

Rootkit Examples

UNIX Windows

t0rn NetBus

Adore (LKM) Back Orifice

SLKM (LKM) Sub Seven

Linux Root Kit(LRK) NT Rootkit

Romanian Vanquish

Acquatica HE4Hook

Looking for Rootkits

Check MD5 values of ‘ls’, ‘ps’, ‘netstat’, ‘sshd’ binaries

Compare output of nmap port scan and netstat

Look for text files in /dev/ or directories that start with a ‘.’ in UNIX

Compare output of ‘ls’ with that of the Sleuth Kit

Examine a file activity timeline created by the Sleuth Kit (not mac-robber or mac-daddy)

“Hidden” Files

Data can also be hidden while not using rootkits

UNIX files and directories that start with a ‘.’ are not shown by default:# find / -name “.*” -print

NTFS Alternate Data Streams are not shown by default:C:\> echo “test” > file1.txtC:\> echo “hidden test” > file1.txt:hidden

Crucial ADS, sfind, and the Sleuth Kit will show their existence

Logs

Copy logs to evidence server for analysis (using netcat as previously described)

Look at wtmp logs on UNIX and run integrity checks to see if it has been modified

Look at other logs and correlate entries with remotely stored copies or network device logs

Copy Event Logs from Windows and open in Event Viewer (will be missing some application log text)

Don’t forget to generate MD5 values

Remote Preview

To analyze a UNIX system, a CD with The Sleuth Kit, Autopsy, and other utilities can be created for remote analysis.

Autopsy is HTML-based, so it is run from the CD and listens on a given port

The investigator connects to the port on the suspect system and can browse the file system through the raw device

This means that no files are modified and that rootkits will be bypassed

EnCase Preview offers a similar function for Windows systems

Information Gathering

Internet-based Research Sanitize your location & be careful

where you visit Use a dial up account NOT the

corporate network Mailing lists may contain additional

information - google searches If IRC information or IP addresses are

found, it is not recommended that you join the IRC channel or do a port scan of the host

Information Gathering

Converting an IP address to a hostname or a hostname from an IP

‘dig’ collects data about domains and networks from DNS records

Information Gathering ‘whois’ returns contact information for an IP address

Information Gathering

traceroute may show where a host is located (based on hostnames of back-bone devices)

Online Information Gathering http://samspade.org/

Powerful collection of ‘network detective’ tools run from the web site.

Windows tool for download. http://www.arin.net

American IP Allocation Database http://www.ripe.net/db/whois/whois.html

European IP Allocation Database http://www.apnic.net

Asia Pacific IP Allocation Database

Phase Conclusions

This phase should answer: Which systems are involved and to what

extent? How critical is each involved system? Which systems do we need to acquire? Is the attack still in progress? Is there an ongoing threat? Do we want to prosecute? Are more monitoring and logging needed for

the investigation? Are there any suspects? Is this from an insider?

Assessment Summary

This phase collects data to identify the scope of the incident

The types of activities of this phase will depend on the type of incident

The data collected will be used in the Response Phase, which will decide whether it is necessary to use additional monitoring or do an acquisition

Documentation and non-intrusive analysis are crucial

Chain of Custody is important if prosecution is likely