“rooting out” rootkits

“Rooting Out” Rootkits

David Taylor & John Lupton

ISC Information Security

Security-SIG, 15 December 2005

ISC/Information Security

Rooting Out Rootkits15 December 2005

ISC/Information Securitysecurity@isc.upenn.edu

rootkit: (n)

• A collection of software “tools” - utilities, scripts, data files, etc.

• Installed on a target computer following compromise (usually remote, but locally possible as well)

• Used not only for operations on that machine, but also as a “stash” to retrieve when breaking into other machines

rootkit (n): (cont.)

• Originally a (mostly) Unix/Linux threat– “rooted” (recompiled) versions of common utilities,

e.g. ls, ps, netstat – Re-written to hide presence and activity of other

rootkit files– Usually cleverly hidden in file system

• Windows rootkits have (surprise!) become much more common in recent years– Structure and operation different than U/L rootkits,

but still do essesntially the same sorts of things:

rootkit activity

• Hide files, processes, network connections• Wipe logs (“cover your tracks”)• Install backdoors• Sniff networks• Replace binaries and executables• And??…Whatever else the attackers wants!

rootkit evolution

• As operating system kernels have evolved, so have the ways rootkits are written to take advantage– Linux: LKM’s (Loadable Kernel Modules)

– A compromised kernel means the machine is “0wn3d” to its very foundation

• Windows rootkits often create and install a specialized system driver and configuration files that access API “hooks” allowing attacker to name the process and determine where and how to hide it.

Example: Hacker Defender

Typical configuration file for Hacker Defender:

[H<<<idden T>>a/"ble]

h"xdef"*

r|c<md\.ex<e::

/[/H/idd\en Ser:vi"ces]Ha>:ck"er//Def\ender

*[Set/tin/:\gs] /

P:assw\ord=hxdef-rulez

Ba:ckd:"oor"Shell=hxdef$.exe

Fil:eMappin\gN/ame=_.-=[Hacker Defender]=-._

Serv:iceName=HackerDefender100

Se|rvi:ceDisp<://la"yName=HXD Service 100

Ser>vic:eD||escr<ip:t"ion=powerful NT rootkit

Dri<ve\rN:ame=HackerDefenderDrv100

D:riv>erFileNam/e=hxdefdrv.sys

rootkit vs. anti-virus

• Sometimes, if the hacker is careless, the rootkit will be caught and quarantined by anti-virus software

• For a knowledgeable hacker, this is only a temporary setback– Can usually turn A-V on and off at will– Don’t think deleting it out of quarantine will

solve the problem - ”I’ll be back…” (The Terminator)

Rootkit Detectors

• Like anti-virus software and firewalls…useful and effective up to a point

• Can detect many well-known, widely distrubuted rootkits

• Many rootkits are known only to one person - the one who wrote it

Rootkit Detectors: Dave T’s Picks

Blacklight (Free Beta)

http://www.f-secure.com/blacklight/

RootkitRevealer

http://www.sysinternals.com/Utilities/RootkitRevealer.html

Rkdetector

http://www.rkdetector.com/

UnHackMe

http://www.greatis.com/unhackme/

“Be vewwy, vewwy quiet…I’m hunting wootkits”

• Before you start:– Have on hand statically re-compiled versions of

common operating system utilities• If you suspect the presence of a rootkit, you cannot trust any

element of the file system on the machine

• You might also want to check the MD5 hash of your “trusted” copies against a pre-written list and/or known good copies on other machines

• Keep these trusted utilities on CD-ROM, stored in a secure place until needed

– Decide whether this is a “live” patient or an “autopsy”

“Is it live, or is it Memorex?…”

• On a live system, you can check:– Active processes– Open files– Changes in file sizes, attributes and access

times– Active network connections– Sniff the network for traffic to and from the

“patient”

“Autopsy”

• If circumstances dictate the system be taken down and rebuilt immediately– Use dd or similar utility to make image file– rootkit presence can still be found by

examining files, directories, attributes, “metadata”, etc

– Can be done post facto, at leisure

Document what you do

• Whether “live” or “autopsy” mode, keep a log of what you do, when you do it and what you find– May come in handy if situation arises again– You may find evidence of a crime

• Might not even relate to rootkit, e.g. presence of child pornography

Check available logs

• logins, ssh, sendmail, ftp, http, etc…

• On target system, are likely to be wiped, but not always

• Many systems configured to use remote logging utilities– “wiped” logs may exist elsewhere

• Look for anomalies, e.g.:– user ‘davet’ shows up running ftp sessions, and you know he:

a) Doesn’t know what ftp is

b) Isn’t smart enough to use it if he did

c) Is dead, and you forgot to delete the account

Check cron jobs

• cron runs processes, programs, scripts, etc. at predetermined times and intervals

• Similar to Scheduled Tasks in Windows

• Typical location is /var/spool/cron• Anything there that looks unfamiliar or

suspicious?

[root@dobro bin]# ps -elfF S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD4 S root 1 0 0 76 0 - 1187 - Nov15 ? 00:00:00 init [5]1 S root 2 1 0 94 19 - 0 ksofti Nov15 ? 00:00:01 [ksoftirqd/0]1 S root 3 1 0 65 -10 - 0 worker Nov15 ? 00:00:00 [events/0]1 S root 4 3 0 71 -10 - 0 worker Nov15 ? 00:00:00 [khelper]1 S root 5 3 0 74 -10 - 0 worker Nov15 ? 00:00:00 [kacpid]1 S root 34 3 0 65 -10 - 0 worker Nov15 ? 00:00:00 [kblockd/0]1 S root 35 1 0 75 0 - 0 hub_th Nov15 ? 00:00:00 [khubd]1 S root 46 3 0 75 0 - 0 pdflus Nov15 ? 00:00:00 [pdflush]1 S root 47 3 0 75 0 - 0 pdflus Nov15 ? 00:00:02 [pdflush]1 S root 49 3 0 67 -10 - 0 worker Nov15 ? 00:00:00 [aio/0]1 S root 48 1 0 75 0 - 0 kswapd Nov15 ? 00:00:07 [kswapd0]1 S root 122 1 0 84 0 - 0 serio_ Nov15 ? 00:00:00 [kseriod]1 S root 192 3 0 65 -10 - 0 worker Nov15 ? 00:00:00 [ata/0]1 S root 194 1 0 85 0 - 0 - Nov15 ? 00:00:00 [scsi_eh_0]1 S root 195 1 0 85 0 - 0 - Nov15 ? 00:00:00 [scsi_eh_1]1 S root 208 3 0 66 -10 - 0 worker Nov15 ? 00:00:00 [kmirrord]1 S root 209 3 0 66 -10 - 0 worker Nov15 ? 00:00:00 [kmir_mon]1 S root 217 1 0 75 0 - 0 kjourn Nov15 ? 00:00:10 [kjournald]1 S root 1134 1 0 75 0 - 0 - Nov15 ? 00:00:00 [khpsbpkt]0 S root 1141 1 0 69 -10 - 900 - Nov15 ? 00:00:00 udevd1 S root 1569 1 0 76 0 - 0 - Nov15 ? 00:00:00 [knodemgrd_0]1 S root 2009 3 0 66 -10 - 0 kaudit Nov15 ? 00:00:00 [kauditd]1 S root 2060 1 0 75 0 - 0 kjourn Nov15 ? 00:00:00 [kjournald]1 S root 2563 1 0 76 0 - 906 - Dec11 ? 00:00:03 syslogd -m 05 S root 2567 1 0 76 0 - 633 syslog Nov15 ? 00:00:00 klogd -x5 S rpc 2586 1 0 75 0 - 1186 - Nov15 ? 00:00:00 portmap 5 S rpcuser 2606 1 0 81 0 - 1449 - Nov15 ? 00:00:00 rpc.statd 1 S root 2639 1 0 76 0 - 4963 - Nov15 ? 00:00:00 rpc.idmapd1 S root 2710 1 0 79 0 - 634 - Nov15 ? 00:00:00 /usr/sbin/acpid5 S ntp 2804 1 0 76 0 - 4638 - Nov15 ? 00:00:00 ntpd -u ntp:ntp -p

Use ps -auxww or -elf to see

what processes are running

lsof can show which files a process has open…

[root@dobro bin]# lsof -p 2563COMMAND PID USER FD TYPE DEVICE SIZE NODE NAMEsyslogd 2563 root cwd DIR 253,0 4096 2 /syslogd 2563 root rtd DIR 253,0 4096 2 /syslogd 2563 root txt REG 253,0 37992 5144639 /sbin/syslogdsyslogd 2563 root mem REG 253,0 105080 42975258 /lib64/ld-2.3.4.sosyslogd 2563 root mem REG 253,0 1489097 42975260 /lib64/tls/libc-2.3.4.sosyslogd 2563 root mem REG 253,0 56791 42975257 /lib64/libnss_files-2.3.4.sosyslogd 2563 root 0u unix 0x000001012afb0e00 5893 /dev/logsyslogd 2563 root 2w REG 253,0 430682 30002610 /var/log/messagessyslogd 2563 root 3w REG 253,0 0 30002360 /var/log/securesyslogd 2563 root 4w REG 253,0 1248 30002361 /var/log/maillogsyslogd 2563 root 5w REG 253,0 190062 30002364 /var/log/cronsyslogd 2563 root 6w REG 253,0 0 30002362 /var/log/spoolersyslogd 2563 root 7w REG 253,0 0 30002363 /var/log/boot.log

eth0 Link encap:Ethernet HWaddr 00:12:3F:64:7A:DA inet addr:192.168.0.2 Bcast:255.255.255.255 Mask:255.255.255.0 inet6 addr: fe80::212:3fff:fe64:7ada/64 Scope:Link UP BROADCAST RUNNING PROMISCUOUS MTU:1500 Metric:1 RX packets:247288 errors:0 dropped:0 overruns:0 frame:0 TX packets:382125 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:143347639 (136.7 MiB) TX bytes:59167774 (56.4 MiB) Base address:0xdcc0 Memory:dfee0000-dff00000

lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:8626 errors:0 dropped:0 overruns:0 frame:0 TX packets:8626 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:6840040 (6.5 MiB) TX bytes:6840040 (6.5 MiB)

Check ifconfig

…there may be a sniffer running on eth0

In most installations, the running mode is normally MULTICAST…

Use netstat to check network connections

[root@dobro bin]# netstat -atup Active Internet connections (servers and established)Proto Recv-Q Send-Q Local Address Foreign Address Statetcp 0 0 *:32769 *:* LISTENtcp 0 0 *:mysql *:* LISTENtcp 0 0 *:netbios-ssn *:* LISTENtcp 0 0 *:sunrpc *:* LISTENtcp 0 0 *:auth *:* LISTENtcp 0 0 localhost.localdomain:ipp *:* LISTENtcp 0 0 192.168.0.2:ipp *:* LISTENtcp 0 0 *:microsoft-ds *:* LISTENtcp 0 1728 192.168.0.2:ssh 24.168.97.666:35424 ESTABLISHEDudp 0 0 *:32768 *:*udp 0 0 172.16.213.1:netbios-ns *:*udp 0 0 172.16.245.1:netbios-ns *:*udp 0 0 192.168.0.2:netbios-ns *:*udp 0 0 *:netbios-ns *:*udp 0 0 172.16.213.1:netbios-dgm *:*udp 0 0 172.16.245.1:netbios-dgm *:*udp 0 0 192.168.0.2:netbios-dgm *:*udp 0 0 *:netbios-dgm *:*udp 0 0 *:662 *:*udp 0 0 *:bootpc *:*udp 0 0 *:sunrpc *:*udp 0 0 *:ipp *:*udp 0 0 192.168.0.2:ntp *:*udp 0 0 localhost.localdomain:ntp *:*udp 0 0 *:ntp *:*raw 0 0 *:icmp *:*

Inodes

• Key part of file system “metadata” structure

• Sequentially numbered “container” that contains file name, permissions, and location(s) in file system (i.e., disk)

• Term “inode” most commonly applied to Unix/Linux, but same principle used in Windows/NTFS

Inode numbering

[root@dobro bin]# ls -li l*16433200 -rwxr-xr-x 1 root root 20088 Jun 20 07:45 link16433242 -rwxr-xr-x 1 root root 31880 Jun 20 07:45 ln16433221 -rwxr-xr-x 1 root root 100952 Jun 15 2004 loadkeys16433184 -rwxr-xr-x 1 root root 28024 Sep 14 04:48 login16433231 -rwxr-xr-x 1 root root 87608 Jun 20 07:45 ls

[root@dobro bin]# ls -li /home/lupton/*.*30409514 -rw-rw-r-- 1 lupton lupton 987 Dec 14 12:16 /home/lupton/ifconfig.txt30409513 -rw-rw-r-- 1 lupton lupton 581 Dec 14 12:16 /home/lupton/ifconfig.txt~30409511 -rw-rw-r-- 1 lupton lupton 1160 Dec 14 12:15 /home/lupton/lsof-p2563.txt30409274 -rw-rw-r-- 1 lupton lupton 0 Oct 31 11:52 /home/lupton/mandolin.iso30409510 -rw-rw-r-- 1 lupton lupton 1746 Dec 14 12:10 /home/lupton/netstat-a--inet.txt30409327 -rw-rw-r-- 1 lupton lupton 4535 Dec 14 11:46 /home/lupton/pself.txt30408835 -rw-rw-r-- 1 lupton lupton 1656 Nov 21 10:49 /home/lupton/upd.txt

Inode behavior

• Inode number remains same when file is edited

• If file is deleted, and new file with same name is written to disk, will usually retain Inode number

• Inode number changes when file is replaced or overwritten by new file

MAC Times

• “MAC”: “Modified/Accessed/Changed”

• “M-time”: date/time file contents last modified

• “A-time”: date/time file was last accessed

• “C-time”: date/time inode information last changed (chmod, new blocks written, defragmentation, etc.)

MAC Implications

• Hackers like to use rootkits to hide things AND plant “rooted” versions of standard binaries

• If the M- or C-times of standard utilities (e.g. ls, ps, netstat) have been altered, it may indicate a bogus version

• Similarly, if the inode number appears to have changed, it may be a “rooted” version

[root@dobro bin]# ls -li /home/lupton/inode_test.txt30409516 -rw-rw-r-- 1 lupton lupton 77 Dec 14 13:24 /home/lupton/inode_test.txt[root@dobro bin]# cat /home/lupton/inode_test.txtThis is the first file, before deletion.

To be saved as "inode_test.txt"...[root@dobro bin]# rm /home/lupton/inode_test.txtrm: remove regular file `/home/lupton/inode_test.txt'? Y

Now, I write and save a new file with the same name…

[root@dobro bin]# ls -li /home/lupton/inode_test.txt30409516 -rw-rw-r-- 1 lupton lupton 56 Dec 14 13:27 /home/lupton/inode_test.txt[root@dobro bin]# cat /home/lupton/inode_test.txtThis is the second version, after deleting the first...

Next, we overwrite the second version with another file…

[root@dobro bin]# mv /home/lupton/inode_bogus.txt /home/lupton/inode_test.txtmv: overwrite `/home/lupton/inode_test.txt'? y[root@dobro bin]# ls -li /home/lupton/inode_test.txt30409285 -rw-rw-r-- 1 lupton lupton 28 Dec 14 13:32 /home/lupton/inode_test.txt[root@dobro bin]# cat /home/lupton/inode_test.txtThis is the "bogus" version[root@dobro bin]#

Follow the changing inode…

MAC Timeline

• Almost always a major part of a full forensic examination

• Correlates filenames and the dates/times their M, A and/or C were altered

• Usually lengthy and time consuming to look through, but can often reveal exactly when and how a rootkit was installed

Basic MAC evaluation using ls

• List inode # with M-time: ls -li• List inode # with A-time: ls -luti• Long listing with C-time: ls -lci

ls -li, -luti, -lci

[root@dobro bin]# ls -li l*16433200 -rwxr-xr-x 1 root root 20088 Jun 20 07:45 link16433242 -rwxr-xr-x 1 root root 31880 Jun 20 07:45 ln16433221 -rwxr-xr-x 1 root root 100952 Jun 15 2004 loadkeys16433184 -rwxr-xr-x 1 root root 28024 Sep 14 04:48 login20767391 -rwxr-xr-x 1 root root 90654 Aug 4 2005 ls

[root@dobro bin]# ls -luti l*16433231 -rwxr-xr-x 1 root root 87608 Dec 14 13:20 ls16433200 -rwxr-xr-x 1 root root 20088 Dec 9 04:02 link16433242 -rwxr-xr-x 1 root root 31880 Dec 9 04:02 ln16433221 -rwxr-xr-x 1 root root 100952 Dec 9 04:02 loadkeys16433184 -rwxr-xr-x 1 root root 28024 Dec 9 04:02 login

[root@dobro bin]# ls -lci l*16433200 -rwxr-xr-x 1 root root 20088 Nov 4 12:27 link16433242 -rwxr-xr-x 1 root root 31880 Nov 4 12:25 ln16433221 -rwxr-xr-x 1 root root 100952 Nov 4 12:24 loadkeys16433184 -rwxr-xr-x 1 root root 28024 Nov 4 12:25 login16433231 -rwxr-xr-x 1 root root 87608 Nov 4 12:22 ls

What’s wrong with this picture?

[root@dobro bin]# ls -a. dd igawk nisdomainname tar.. df ipcalc pgawk tcsh.. dmesg kbd_mode ping touchalsaunmute dnsdomainname kill ping6 tracepatharch doexec ksh ps tracepath6ash domainname link pwd tracerouteash.static dumpkeys ln red traceroute6aumix-minimal echo loadkeys rm trueawk ed login rmdir umount

Take a closer look…

[root@dobro]# ls -a. dd igawk nisdomainname tar.. df ipcalc pgawk tcsh.. dmesg kbd_mode ping touch

How can there be two ‘..’ directories?…

How did this happen?

[root@dobro bin]# mkdir ..\

This is actually:

mkdir <space><dot><dot><backslash><space><enter>

It creates a directory actually named “dot-dot-space”

What’s in this “mystery” directory?

[root@dobro bin]# cd ..\[root@dobro .. ]# ls -ltotal 24-rw-r--r-- 1 root root 0 Dec 15 12:19 rootkit_file_01-rw-r--r-- 1 root root 0 Dec 15 12:19 rootkit_file_02-rw-r--r-- 1 root root 0 Dec 15 12:19 rootkit_file_03-rw-r--r-- 1 root root 0 Dec 15 12:19 rootkit_file_04-rw-r--r-- 1 root root 0 Dec 15 12:19 rootkit_file_05-rw-r--r-- 1 root root 0 Dec 15 12:19 rootkit_file_06

What happens when…

[root@dobro bin]# mkdir ..\ \ \ \ \

(i.e., dot-dot with 5 spaces)

What does ls show?

[root@forensic_laptop bin]# ls -a. date hostname nice sync.. dd igawk nisdomainname tar.. df ipcalc pgawk tcsh.. dmesg kbd_mode ping touch

Yet another ‘..’ directory…

Anything in it?

[root@dobro bin]# cd ..\ \ \ \ \[root@dobro .. ]# ls -ltotal 24-rw-r--r-- 1 root root 0 Dec 15 12:23 evilroot_01-rw-r--r-- 1 root root 0 Dec 15 12:23 evilroot_02-rw-r--r-- 1 root root 0 Dec 15 12:23 evilroot_03-rw-r--r-- 1 root root 0 Dec 15 12:23 evilroot_04-rw-r--r-- 1 root root 0 Dec 15 12:23 evilroot_05-rw-r--r-- 1 root root 0 Dec 15 12:23 evilroot_06

Where do you find rootkit files?

• Sometimes, right under your nose– Watch out for “stupid tricks” with ‘..’

directories and filenames beginning with ‘.’– Use both -l and -a flags when using ls

•-l does not list ‘.’ and ‘..’ entries:[root@forensic_laptop bin]# ls -ltotal 6864-rwxr-xr-x 1 root root 15528 Jul 19 07:34 alsaunmute-rwxr-xr-x 1 root root 2812 Sep 14 04:52 arch-rwxr-xr-x 1 root root 98356 Jun 15 2004 ash-rwxr-xr-x 1 root root 522116 Jun 15 2004 ash.static-rwxr-xr-x 1 root root 12964 Jun 15 2004 aumix-minimallrwxrwxrwx 1 root root 4 Oct 31 13:10 awk -> gawk-rwxr-xr-x 1 root root 13068 Jun 20 07:52 basename

Look for things that are odd, out of place, or you don’t

recognize

•Hackers usually aren’t going to name the files “rootkit_01”, etc.

•They have an entire file system to hide them in

OK, I found a rootkit - or I’m pretty sure there’s one in there somewhere…

What do I do??

OK, I found a rootkit - or I’m pretty sure there’s one in there somewhere…

What do I do??

REBUILD!!!

You cannot trust a system that has a rootkit installed.

Rebuilding is not just the best option…

IT’S THE ONLY OPTION

See:www.microsoft.com/technet/community/columns/secmgmt/sm0504.mspx

By Jesper Johansson (Microsoft Security Manager)

Questions? Comments?

“rooting out” rootkits

live system

file systemwindows rootkits

rootkit activityhide

rootkit filesusually

ul rootkits

rootkit evolutionas

foundationwindows rootkits

ways rootkits

Documents

rooting out rejection and hidden bitterness

a hizmet approach to rooting out violent extremism

windows nt rootkits

root out rootkits: an inside look at mcafee deep ... - intel...

rootkits jonathan barella chad petersen. overview what are...

database rootkits

advance rooting

rootkits & spyware

annex 1b rooting out hunger in malawi with nutritious...

rooting out relationship killers

designing bsd rootkits

rootkits. agenda introduction definition of a rootkit types...

rooting out hunger in malawi with nutritious orange-fleshed...

rooting phylogenies

cisco ios rootkits

user-mode rootkits

android rooting

root out rootkits - intel · rootkits are an increasingly...

rootkits. - udc

rooting out hunger in malawi with potato and sweetpotato