bcit comp 8506 final project report by wesley kenzie july 2010

BCIT Computer Systems Technology COMP 8506 Selected Topics in Network Security and Design Due Date: July 15, 2010 HoneyNet Project: Windows 2000 Server VM ______________________________________________________________________________

______________________________________________________________________________Copyright © 2010 Pandher, Tom, Infeld, Kenzie. All Rights Reserved. of 64

HoneyNet Project: Windows 2000 Server VM 1. Introduction ....................................................................................... page 2 2. Design ............................................................................................... page 3 3. Chronology ......................................................................................... page 16 4. Analysis ............................................................................................. page 22 5. Conclusions ........................................................................................ page 56 6. Prologue ............................................................................................. page 58 7. Appendix ............................................................................................ page 61



1. Introduction One of the virtual machines we created on our dedicated server was based on a Microsoft Windows 2000 Server iso image that we found at a torrent download site. It was thought that an older version of a Windows Server operating system would be a valued addition to our HoneyNet, since it likely would have a number of vulnerabilities as a standard feature – no extra installation required. After installation at about 3:50pm on July 3, 2010, we did not run Windows Update on it to bring it up-to-date, however it did include Service Pack 4, so it was presumed to be relatively current. Interestingly, this OS was only retired by Microsoft on July 13, 2010 – as per the chart below - so presumably there are still organizations that have continued to use this operating system for operation of their servers to this day.



2. Design 2.1 After this VM was created, the System Summary showed as follows. We created it with 512MB of RAM and 8GB of disk space. It was given the name “Tor2” and was made part of the “proxyup.net” domain. It was setup as an IIS web server, Active Directory Domain Controller, DNS Server, Directory Server, Terminal Services Server, Print Server, and File Server.



2.2 The VM was created with a single network interface as shown below. It was setup using the VMware “bridged” method, and given its own static, public IP address 212.117.166.35 which was updated in the Domain Name System as the address for the domain name “tor.proxyup.net”.



2.3 An nmap scan of the VM’s external-facing network interface (from the perspective of the host at 212.117.166.234, but this would have been the same from any other internet device) shows the significant number of open ports that were exposed as part of the default installation of the OS.



2.4 Only one user was created on the VM, which was “Administrator”. It was created with no password.



2.5 IIS was configured to serve up a single index.html web page, since the intention was to keep it simple, and to appear to the world as relatively unsophisticated (i.e. stoopid).



2.6 Windump.exe was initially installed (available from http://www.winpcap.org/windump/install/default.htm ) and implemented by means of a “start_windump.bat” batch file. Each time the OS was rebooted this batch file was manually run, which consisted of the following command: windump –e –f –n –s 0 –C 100 –vvv –x –w c:\log\windump The intention was to capture packets (similar to tcpdump) that might be of interest for later investigation. However, in looking at all the windump .pcap files created over the 4 days from July 3 to July 7, none of them have any packets of interest, other than about 10,000 “LLC” protocol packets sent out about every 10 seconds. It is possible that WinDump was not able to capture other packets, or that WinDump was not running when other packets of interest were active. This batch file is located on the accompanying DVD in the “Tor2” \Documents and Settings\Administrator\My Documents directory. The log files are also on the DVD in the “Tor2” \log directory.



2.7 Event logging and Local security policies were initially set to the default setting for Windows 2000 Server. This meant – among other things – that event logging was not done for “system events”, “object access” or “logon events”; that event logs were limited in size to 512KB; and that event logs were overwritten after 7 days or after filling up - whichever occurred first.



2.8 The default settings also meant that the OS would automatically reboot if a system failure occurred, and memory dump files would not be saved across reboots, as shown below:



2.9 Another security weakness of the default Windows 2000 Server OS that was discovered while trying to setup remote logging was that it cannot be done - other than by using third party software. Kiwi Syslog Server was thus installed on the VM (available from http://www.solarwinds.com ) to automatically convert event logs to syslog format, and to forward syslogs to a remote server. Around this time, however, there were a number of performance and stability issues on the VM that made it difficult to determine the cause. In hindsight it was not likely something caused by the Kiwi Syslog Server software, but we did not have enough experience with it to recommend it other than as one possible solution. Two days after Kiwi Syslog Server was installed, it was replaced by Snare EventLog Agent for Windows (available from http://www.intersectalliance.com ) which is another third party program designed to handle this important security task. It was our belief at the time that this would be a better solution than Kiwi Syslog Server, but in hindsight we cannot say with any certainty that either of these were better than the other.



3. Chronology 3.1 Installation of the VM was completed on the afternoon and evening of July 3. This included installation of Firefox, Filezilla, Notepad++, WinPcap, Windump, MySQL, and the VMware Tools package. (Over the next four days, 7-zip, Kiwi Syslog Server, Wireshark, and Snare EventLog Agent for Windows were also installed, as shown by the Program Files directory listing below.)



3.2 The morning following the initial installation, Sunday, July 4 at 10:59am, an unusual message was discovered from the previous evening, titled “Messenger Service”: “Message from NWAVE to ADMINISTRATOR on 7/3/2010 11:07:16 PM. CPIB691: User ADMINISTRATOR (NET) has successfully connected to i5/OS NetServer.”

The date and time of this message is important, because there is evidence (discovered later) that this is very close to the date and time that all the html files on the VM were hacked/altered. But as to who “NWAVE” is, or “CPIB691”, or “i5/OS NetServer” – these have never (yet) been discovered. This message was the first indication that our VM may have fulfilled its purpose as a Honeypot – and it had done so within 8 hours of being setup.



3.3 The second discovered message showed up later that evening, on July 4 as follows:

Unfortunately, the System Event log prior to 11:02:23 AM on July 4 was corrupted and unreadable. Also, the Security Event log prior to 1:49:10 PM on July 7 was either lost or not turned on. Also, the Application Event log between 7:23:37 PM on July 3 and 6:18:21 PM on July 5 was either lost or not turned on.



3.4 What is known is that between 9:10 PM on Sunday evening, July 4 and about noon, on Tuesday, July 6, the VM was regularly rebooting, with a message similar to the following. Anecdotally, it seemed that the VM was not able to run for more than about one hour before it would shutdown and then reboot. There was no obvious reason for this to be happening.



3.5 At about noon on Tuesday, July 6 the DNS and DHCP services were removed, and this resulted in no shutdowns over the next 2+ hours. Wireshark was then installed and started to see what sort of communications was going on. This activity was recorded to the “wireshark_snippet_201007061500.pcap” file on the accompanying DVD in the “Tor2” log directory. Over a period of just 1 minute and 41 seconds 55,074 packets were captured, and at first glance it appeared that most of these packets involved the IP address of the server VM, 212.117.166.35. A decision was then made to run Wireshark overnight to a series of .pcap files, and so at 15:23:01 on July 6 this was done, as shown in the “wshark_00001_20100706152301.pcap” file on the accompanying DVD. Unexpectedly Wireshark only ran for 16 minutes and 17 seconds before it crashed. In that short time it captured 525,355 packets. This crash was not discovered until the next day, July 7, at about 1:30pm, at which time the server VM was shutdown manually and the network interface disabled so that it had no further outside access.



3.6 Interestingly, the first few attempts to shutdown the server VM at this point were not successful.



4. Analysis 4.1 A subset of the latest Wireshark pcap file was created by adding a display filter “ip.addr == 212.117.166.35” in order to isolate all packets either sent out or received by the VM server. This totaled 351,465 packets, or about 67% of the total packets captured. These packets are contained in the “wshark_00001_ip_addr_212_117_166_35.pcap” file on the accompanying DVD. The following summary shows the breakdown of these packets by protocol showing about 20% as ICMP, 80% as TCP/IP, and a mere 0.01% as UDP.



4.2 A clear indication that the server was infected with some form of malware was a further filtering of these captured packets on “tcp.dstport == 445”. Fully 236,729 of the 351,465 packets in and out of the server (roughly 67%) involved port 445. This is about 242 packets per second!

Clearly some sort of malware had infected this machine.



4.3 SANS has a summary of vulnerabilities involving port 445 at http://isc.sans.edu/port.html?port=445. One of the recent comments here indicates that a password should always be set for the “Administrator” user. This had not been done as previously indicated – the password for Administrator had been left blank, and no other user accounts had been created.

The SANS discussion of port 445 indicates that there have been multiple worms over the past few years that have used this port.



4.5 When copying some of the files from the server VM to a Windows 7 Ultimate desktop computer, running F-Secure Internet Security 2010 software, the following warning message was displayed, indicating the malware to be the W32/Downadup.AB Worm.



4.6 A 7z compressed file contained some suspected infected files was uploaded to https://www.virustotal.com for analysis, and 24 of 42 virus detection engines identified some form of worm, trojan or virus:



4.7 In looking around the server VM hard drive, I discovered in the IIS web server “wwwroot” directory some new and altered files. Three new unknown .exe applications were there: kkvwbsrw.exe, lrbtjhnn.exe, and ttjtnrek.exe as shown below:



4.8 An internet search on kkvwbsrw.exe came up with several hundred hits, including the following from threatexpert.com:



4.9 Clearly there was some form of malware on the server VM. Equally clearly was the fact that there were several different interpretations of what this malware was. This conclusion was further re-enforced by the fact that internet searches on the other 2 .exe files came up with almost no hits.



4.10 Next the “index.html” file in c:\inetpub\wwwroot was analyzed, and finally the details of how this malware worked was uncovered. See below for the inserted <OBJECT> in this file:

And notice that index.html file was time stamped at exactly the same date and time as the kkvwbsrw.exe file. Clearly this was related.



4.11 The _vti_inf.html file was investigated next, which was time stamped at July 3, 2010 at 11:50 PM, just 8 hours after the initial installation at 3:52 PM. This file had the same time stamp as the other 2 .exe files:



4.12 The postinfo.html file was also investigated, showing the same pattern of inserted <OBJECT> commands:



4.13 And what was this application/x-oleobject? For this we went to the Windows Registry:



Notice the Registry entry defines a “LocalServer32” that points back to the .exe file c:\Inetpub\wwwroot\lrbtjhnn.exe !



4.14 Further investigation showed that every .html and .htm file on the server VM was altered in the same way – by insertion of an <OBJECT> command and creating a corresponding .exe file. Notice the date and time stamp below for the “debug” and “iisHelp” directories, for example:



4.15 As well as the addition of the vxxblbhe.exe, hjjbcxkx.exe and bnsklnec.exe application files; and the date and time stamps of the ciadmin.htm, ciquery.htm and ixqlang.htm files:



This was obviously how the malware spread to other machines. When a browser opened up one of these .htm or .html pages, the corresponding .exe program was run.



4.16 Next, an analysis of the Windows 2000 Server event log files was undertaken. However, these were inconclusive, and relatively confusing. There were gaps in the event log data and none of the messages reviewed clarified how the malware was initiated. Further, importing the Windows 2000 event logs into a Windows 7 system that has a much improved event viewer and considerably better analysis functionality proved of no benefit.



4.17 Next, an analysis of the snort alert log files was initiated. These snort alerts were captured by an instance of snort running on the host of the dedicated server for this project. This host machine was running a Debian Lenny operating system, with VMware Server 2.0 installed to host the various virtual machines used in this HoneyNet Project. The host has its own public IP address, 212.117.166.234 which is connected to the domain proxyup.net. The “host” name of the host as assigned by the dedicated server provider was “R005”. Snort was configured to restart every 12 hours, as the following snapshots show. Because each VM was configured to use “bridged” networking, all network activity was captured by the instance of snort running on the eth0 interface.



4.18 Over the period from May 20 to July 1, over 164 megabytes of alert log data was collected. It is not currently understood why no alert data was collected after July 1, despite the fact that snort continued to run after July 1, during which time tcpdump log files were also being generated. A snortsnarf analysis of this alert log data was undertaken, but as of the due date of this project has not finished. Snortsnarf has been processing this data for over 72 hours and counting. This alert log data is contained on the accompanying DVD in the “alert.combined.log.gz” file in the /R005/log/snort/ directory. The snort.conf configuration file is in the /R005/etc/snort directory.



4.19 The tcpdump log data generated by snort running on the host operating system was investigated next, in an attempt to identify the source and behaviours of the malware discovered on the Windows 2000 Server VM. What this tcpdump log data shows is very interesting. Just after the infection, as shown in tcpdump.log.1278308104, which starts at 22:35:07 on July 4, 2010, there are a large number of different IP addresses sending ICMP packets to 212.117.166.35 on port 445. And there are a few malformed IP packets sent from our infected server in a form of covert channel communications. See packet 3 in the image below. An IP packet has been sent to 144.223.245.130 with a 139-byte data payload that clearly indicates that some form of covert communication message is being sent:



Note the contents of the data payload: Priority Count: 5. Connection Count: 96. IP Count: 48. Scanned IP Range: 82.90.1.167:124.219.250.32. Port/Proto Count: 0. Port/Proto Range: 0:0.



4.20 Packets 28, 29, 37, and 38 (as shown below) were likewise sending data in IP packets. The data in packet 28 is as follows: Priority Count: 5. Connection Count: 16. IP Count: 1. Scanner IP Range: 212.117.166.35:212.117.166.35. Port/Proto Count: 5. Port/Proto Range: 139:445.

Note that this data says “Scanner IP Range” rather than the previous “Scanned IP Range”.



4.21 Packet 29 (as shown below) is seen to send the following data: Open Port: 445.



4.22 Packet 116 (as shown below) shows an IP packet coming to our infected machine: Priority Count: 7. Connection Count: 20. IP Count: 18. Scanner IP Range: 64.120.173.103:210.242.145.125. Port/Proto Count: 19. Port/Proto Range: 1952:8118:



4.23 In total, in this one tcpdump.log.1278308104 snort log file, from July 4, 2010 22:35:08 to July 5, 2010 3:03:01 90,856 of the total 117,988 packets (77% of total) consisted of malformed IP packets sending data between infected machines. This works out to about 5.6 IP packets per second over this 4 hour and 27 minute time period. Incredibly, there were over 6,100 different IP addresses in this collection of IP packet communications in this 4.5 hour period.



4.24 The first packet coming from our infected server VM appears to be at 17:36:46 on July 3, which is packet 301 in log file tcpdump.20100703150614.pcap as shown below:

Installation of the server VM was done at around 3:52pm that same day. Less than 2 hours later the server VM was sending out covert communications to IP address 83.55.113.159, which is likely a residential internet user in Barcelona, Spain.



4.25 So was the initial Windows 2000 Server iso image pre-infected? We tested this by possibility by creating a new VM on our home office Windows 7 from the same iso image while using Wireshark to record all packet activity during installation and afterwards for the next 12 hours. No malformed IP packets were either sent or received. No packets were sent out to port 445. It seems likely, therefore, that the iso image was not pre-infected.



4.26 So where did the infection of the server VM come from? One interesting possibility came to light when reviewing the contents of earlier tcpdump log files – earlier than the July 3 date on which it was installed. As shown below, on June 27 at 15:03:23 a malformed IP packet ostensibly coming from the OpenDNS server at 208.67.222.222 was sent to our server VM IP address!



4.27 The host machine’s tcpdump log files showed many other possibilities as to the original source of the malware, but at this point nothing conclusive has been determined. The answer likely lies in the “tcpdump.20100705.todate.filtered.pcap” file in the /R005/log/snort directory on the accompanying DVD. This pcap file, from June 23 at 22:37:40 until July 6 at 03:03:00 contains 1,758,740 packets, including many from the host IP address and the other virtual machines in this HoneyNet Project that implicate one or more of them as being the source of the malware found on the Windows 2000 VM.



5. Conclusions 5.1 Using virtual machines for honeypot and honeynet purposes is an excellent way to go. Setup of new VM’s with different operating systems usually takes just 15-20 minutes. A single dedicated server can handle multiple simultaneous, independent or inter-connected VM’s. Network topologies can be implemented in many different ways, with one or more network interfaces per VM. Intrusion Detection Systems and other monitoring programs can be setup on one or more VM’s – and on the host in some cases – to keep track of network activity on other VM’s. 5.2 A potential downside of using virtual machines is that there is a single point of failure, with everything running on a single computer. This was not, however, an issue for us, since the machine held up, even in the face of individual VM’s becoming corrupted and infected with malware. 5.3 A default Windows 2000 Server, even at the SP4 level, is a nightmare from a security risk perspective. In our case it took only about 2 hours or less for it to become infected with some form of malware that quickly proceeded to spread itself to thousands of other machines. 5.4 A firewall, separate from Windows 2000 Server, is essential to ensure that only authorized users and authorized services are provided entry. 5.5 Most Windows 2000 Server services should be restricted to internal, private subnets. 5.6 Things go wrong, and therefore redundancies should be planned and implemented from the start. 5.7 Use of a third-party event-to-syslog conversion and remote logging program on Windows is very useful, and highly recommended. 5.8 Analysis of log data collected on a honeypot is very time consuming and tedious. Automated tools cannot be relied on to find all malware. 5.9 Severity of malware discovered on this Windows Server 2000 VM would be measured as: (Target Criticality + Attack Lethality ) – (System Countermeasures + Network Countermeasures). Running a public-facing IIS web server as we did, with file sharing and other services exposed, would likely give it a Target Criticality of 4. As a honeypot, of course, this would really be 1 rather than 4. Attack Lethality would be equal to about 2, since it did not disable or destroy anything – the malware appears to have been designed to deface web pages and to propagate itself. On the other hand, it is not possible to determine what future lethality might be associated with this malware – would it enable access by its author(s) at some future date for more destructive purposes?



System Countermeasures on this VM would be 1, as low as possible. Installing a proper firewall would increase this above 1. Closing most public services would increase it as well, as would establishing proper passwords and account policies. Network Countermeasures on the HoneyNet would be 1 as well. We were only recording attacks, not preventing them in any way. A proper IDS to automatically shut off access based on certain triggers would increase this above 1, as would implementing a choke firewall to protect all internal VM’s.



6. Prologue 6.1 Prior to installation of the Windows 2000 VM on July 3, another VM had been setup to on the same IP address 212.117.166.35. This VM was running Ubuntu Server 8.04 and was setup as a public email relay using port 25. It attracted a very large amount of email traffic, but unfortunately it crashed and became corrupted. Analysis of the data collected by this previous VM has been deferred, but its existence explains much of the tcpdump log data collected by snort prior to July 3.



6.2 A remote logging server was setup to receive syslog data from each of the virtual machines in the HoneyNet Project. The rsyslog.conf file on the accompanying DVD in the /R005/etc/ folder shows how the host machine was configured to do this. A snapshot is shown below, where line 53 indicates the remote logging location and port.

Unfortunately, the benefits of logging remotely were not realized, due to an unexpected disk failure on the 212.117.166.26 server on Friday, July 9. Recovery from this disk failure is still a work in process, and at the time of preparation of this report, none of this remotely logged data is available for review or analysis.



6.3 Subsequent to all the data discussed in this report, the Windows 2000 VM was turned back on and reconnected to the Internet. Wireshark was turned on and allowed to run for a short time, however it crashed and the VM rebooted for unknown reasons a few times. At one point, while it was still running, the following snapshot was taken showing some of the Remote IPC shares in use. It is believed that all of these were initiated by the malware on the VM.



7. Appendix The accompanying DVD set has the following files: R005 (files and directories from host machine) /cache /apt /debconf /fontconfig /ldconfig /man /modass /etc /alternatives /apm /apt /bash_completion.d /ca-certificates /calendar /cron.d /cron.daily /cron.hourly /cron.monthly /cron.weekly /cups /dbus-1 /default /defoma /dhcp3 /doc-base /dpkg /emacs /exim4 /fonts /gconf /groff /gtk-2.0 /init.d /initramfs-tools /iproute2 /ld.so.conf.d /ldap /logcheck /logrotate.d /lsb-base



/lvm /lynx-cur /modprobe.d /mysql /network /opt /pam.d /pango /perl /ppp /prelude /privoxy /python /python2.5 /rc0.d /rc1.d /rc2.d /rc3.d /rc4.d /rc5.d /rc6.d /rcS.d /rsyslog.d /security /skel /snort /ssh /ssl /sysctl.d /terminfo /tor /tripwire /udev /vga /vim /vmware /vmware-tools /vmware-fix /wireshark /X11 /xdg /home /ap /ei007 /gtom



/wkenzie /lib /apt /aptitude /arpwatch /bittorrent /dbus /defoma /dhcp3 /dpkg /exim4 /gconf /initscripts /libuuid /logrotate /misc /mysql /python-support /rkhunter /tor /tripwire /ucf /urandom /vim /vmware /x11 /xkb /local /log /apt /bittorrent /dsniff /exim4 /fsck /mysql /news /nmap /privoxy /snort /snort_vmnet8 /tor /vmware /mail /opt /run



/dbus /exim4 /network /oinkmastser /sshd /sudo /vmware /vmware-hostd-ticket /src /snortsnarf Windows-2000-server-sp4.iso Tor2 (files and directories from the Windows Server 2000 VM) /Documents and Settings /Administrator /All Users /All Users.WINNT /Default User /Default User.WINNT /IUSR_HOMEPNY /NetShowServices /Inetpub /log /WINNT