log analysis and intrusion detection

Log Analysis and Intrusion Detection

By

Srikrishna Gudavalli

Venkata Naga Vamsi Krishna

Ravi Kiran Yellepeddy

Log Analysis (Windows And linux)

What is log analysis?

Describes an event (or) process activity in detail on the system.

Examples : • user authentication event log• ftp authentication .

Setup for LogAnalysis

• Application Log

Specific to particular application.

eg:MS word,Windows Media Player

• Security Log

Specifically logs all the security features.

• System Log

Logs all the system related activities.

Log Files

• Lokasi bergantung, umumnya pada /var/log

• Contoh/var/log/mail.log/var/log/messages/var/log/daemon.log/var/log/apache/access-log/var/log/apache/error-log/var/adm/utmp/var/adm/wtmp

5

Log Analysis and Correlation

• Syslogs, messages logs, other Unix host logs

Security/Auth Log

Mar 9 13:07:49 nile in.telnetd[1315]: connect from 68.62.72.193Mar 9 13:09:24 nile in.rlogind[1321]: connect from 68.62.72.193Mar 9 13:09:27 nile in.ftpd[1326]: connect from 68.62.72.193Mar 9 13:09:28 nile in.rshd[1329]: connect from 68.62.72.193Mar 9 13:09:28 nile in.telnetd[1333]: connect from 68.62.72.193Mar 9 13:09:31 nile in.fingerd[1334]: connect from 68.62.72.193Mar 9 13:12:13 nile in.fingerd[1352]: connect from 68.62.72.193Mar 9 13:12:13 nile in.rlogind[1357]: connect from 68.62.72.193Mar 9 13:12:14 nile in.rshd[1360]: connect from 68.62.72.193Mar 9 13:12:16 nile in.telnetd[1365]: connect from 68.62.72.193Mar 9 13:12:18 nile in.ftpd[1368]: connect from 68.62.72.193Mar 9 13:15:23 nile in.ftpd[1382]: connect from 68.62.72.193Mar 9 13:15:24 nile in.telnetd[1384]: connect from 68.62.72.193Mar 9 13:15:27 nile in.rshd[1396]: connect from 68.62.72.193Mar 9 13:15:28 nile in.rlogind[1398]: connect from 68.62.72.193Mar 9 13:15:29 nile in.fingerd[1400]: connect from 68.62.72.193Mar 9 13:26:43 nile login: ROOT LOGIN ON tty1Mar 9 13:37:15 nile in.ftpd[1447]: connect from 68.62.72.193Mar 9 13:37:44 nile in.fingerd[1448]: connect from 68.62.72.193Mar 9 17:17:19 nile in.telnetd[1521]: connect from 12.87.62.43Mar 9 17:17:26 nile login: LOGIN ON 0 BY pstephen FROM 43.detroit-16-17rs.mi.dial-access.att.netMar 9 17:50:13 nile in.ftpd[1556]: connect from 216.205.122.231Mar 10 11:12:02 nile in.ftpd[8929]: connect from 200.68.32.185Mar 10 11:13:07 nile in.ftpd[8965]: connect from 68.62.72.193

6


• TCPDump logs

11:30:27.181108 eth0 < pcp01103425pcs.aubrnh01.mi.comcast.net.17697 > nile.ftp: . 1:1(0) ack 1 win 4288 (DF)11:30:27.190617 eth0 > arp who-has ubr01-a-rtr.aubrnh01.mi.comcast.net tell nile (0:0:86:54:50:5b)11:30:27.198369 eth0 < arp reply ubr01-a-rtr.aubrnh01.mi.comcast.net is-at 0:5:5f:e9:10:54 (0:0:86:54:50:5b)11:30:27.207662 eth0 < ns02.pntiac01.mi.comcast.net.domain > nile.1025: 20012 1/2/2 PTR pcp01103425pcs.aubrnh01.mi.comcast.net. (174) (DF)11:30:27.218149 eth0 < ns02.pntiac01.mi.comcast.net.domain > nile.1025: 20013 1/2/2 A pcp01103425pcs.aubrnh01.mi.comcast.net (151) (DF)11:30:27.230334 eth0 < ns02.pntiac01.mi.comcast.net.domain > nile.1025: 20014 1/2/2 PTR pcp01103425pcs.aubrnh01.mi.comcast.net. (174) (DF)11:30:27.231013 eth0 > nile.ftp > pcp01103425pcs.aubrnh01.mi.comcast.net.17697: P 1:80(79) ack 1 win 32120 (DF) [tos

0x10] 11:30:27.253084 eth0 < pcp01103425pcs.aubrnh01.mi.comcast.net.17697 > nile.ftp: P 1:16(15) ack 80 win 4209 (DF)11:30:27.253122 eth0 > nile.ftp > pcp01103425pcs.aubrnh01.mi.comcast.net.17697: . 80:80(0) ack 16 win 32120 (DF) [tos

0x10]

7


• Intrusion Detection Log (RealSecure)

Event Date Event NameProtocol

IDSourcce

Port Dest PortSrc Port Name

Dest Port Name Src Address Dest Address Engine IP

9/10/2001 11:27 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 10.1.230.102 192.168.9.2439/10/2001 11:27 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 10.1.230.102 192.168.9.2439/10/2001 11:27 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 10.4.18.245 192.168.9.2439/10/2001 11:27 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 10.4.18.245 192.168.9.2439/10/2001 11:27 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 10.4.18.245 192.168.9.2439/10/2001 11:27 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 10.4.18.245 192.168.9.2439/10/2001 11:44 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 192.168.6.75 192.168.9.2439/10/2001 11:44 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 192.168.6.75 192.168.9.2439/10/2001 11:44 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 192.168.6.75 192.168.9.2439/10/2001 11:44 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 192.168.6.75 192.168.9.2439/10/2001 11:44 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 10.1.151.231 192.168.9.2439/10/2001 11:44 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 10.1.151.231 192.168.9.2439/10/2001 11:44 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 10.1.151.231 192.168.9.2439/10/2001 11:44 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 10.1.151.231 192.168.9.2439/10/2001 11:44 SNMP_Activity 17 1030 162 1030 SNMPTRAP 192.168.10.199 10.1.151.246 192.168.9.243

8


• Intrusion Detection Log (SNORT Summary)

Apr 16 02:45:37 lisa snort[7483]: IDS13/portmap-request-mountd: 200.190.13.181:1372 -> 172.16.1.107:111Apr 16 07:17:06 lisa snort[7483]: IDS128/web-cgi-phf: 200.190.8.220:55220 -> 172.16.1.107:80Apr 16 14:54:20 lisa snort[7483]: IDS171/Ping zeros: 24.201.15.148 -> 172.16.1.101Apr 16 14:54:20 lisa snort[7483]: IDS171/Ping zeros: 24.201.15.148 -> 172.16.1.105Apr 16 14:54:20 lisa snort[7483]: IDS171/Ping zeros: 24.201.15.148 -> 172.16.1.107Apr 17 06:02:32 lisa snort[8255]: IDS198/SYN FIN Scan: 195.116.152.104:0 -> 172.16.1.101:111Apr 17 06:02:32 lisa snort[8255]: IDS198/SYN FIN Scan: 195.116.152.104:0 -> 172.16.1.107:111Apr 17 09:45:28 lisa snort[8255]: IDS198/SYN FIN Scan: 195.116.152.104:0 -> 172.16.1.105:111Apr 19 08:00:19 lisa snort[3515]: IDS/DNS-version-query: 212.25.75.196:1723 -> 172.16.1.101:53Apr 20 01:26:00 lisa snort[3515]: IDS212/dns-zone-transfer: 24.234.45.60:4075 -> 172.16.1.107:53Apr 20 03:49:38 lisa snort[3515]: IDS/DNS-version-query: 216.123.23.5:4349 -> 172.16.1.101:53Apr 20 03:49:39 lisa snort[3515]: IDS/DNS-version-query: 216.123.23.5:4350 -> 172.16.1.107:53Apr 20 21:48:55 lisa snort[12353]: IDS246/large-icmp: 129.142.224.3 -> 172.16.1.107Apr 20 21:48:55 lisa snort[12353]: IDS246/large-icmp: 129.142.224.3 -> 172.16.1.107Apr 20 22:48:13 lisa snort[12632]: IDS159/Ping Microsoft Windows: 216.228.4.204 -> 172.16.1.101Apr 20 22:48:13 lisa snort[12632]: IDS159/Ping Microsoft Windows: 216.228.4.204 -> 172.16.1.101Apr 20 23:00:33 lisa snort[12657]: IDS171/Ping zeros: 216.228.4.133 -> 172.16.1.101Apr 21 11:01:27 lisa snort[12777]: IDS/DNS-version-query: 207.236.55.76:4039 -> 172.16.1.101:53Apr 21 11:01:28 lisa snort[12777]: IDS/DNS-version-query: 207.236.55.76:4044 -> 172.16.1.107:53Apr 22 08:36:29 lisa snort[743]: IDS/DNS-version-query: 212.244.222.100:1368 -> 172.16.1.101:53Apr 22 08:36:29 lisa snort[743]: IDS/DNS-version-query: 212.244.222.100:1328 -> 172.16.1.107:53

Courtesy of The Honeynet Project

9


• Intrusion Detection Log (SNORT Raw Log)

[**] WEB-MISC 403 Forbidden [**]07/29-23:59:17.752579 0:0:C5:75:67:2C -> 0:AA:0:B7:E9:56 type:0x800 len:0x246209.235.0.178:80 -> 63.222.202.8:1550 TCP TTL:43 TOS:0x0 ID:22555 IpLen:20 DgmLen:568 DF***AP*** Seq: 0x85B19798 Ack: 0x4E439F5C Win: 0x7D78 TcpLen: 2048 54 54 50 2F 31 2E 31 20 34 30 33 20 46 6F 72 HTTP/1.1 403 For62 69 64 64 65 6E 0D 0A 44 61 74 65 3A 20 4D 6F bidden..Date: Mo6E 2C 20 33 30 20 4A 75 6C 20 32 30 30 31 20 30 n, 30 Jul 2001 033 3A 35 38 3A 35 38 20 47 4D 54 0D 0A 53 65 72 3:58:58 GMT..Ser76 65 72 3A 20 41 70 61 63 68 65 2F 31 2E 33 2E ver: Apache/1.3.31 39 20 28 55 6E 69 78 29 20 6D 6F 64 5F 73 73 19 (Unix) mod_ss6C 2F 32 2E 38 2E 33 20 4F 70 65 6E 53 53 4C 2F l/2.8.3 OpenSSL/30 2E 39 2E 36 61 20 6D 6F 64 5F 70 65 72 6C 2F 0.9.6a mod_perl/31 2E 32 35 20 6D 6F 64 5F 67 7A 69 70 2F 31 2E 1.25 mod_gzip/1.33 2E 31 39 2E 31 61 20 50 48 50 2F 34 2E 30 2E 3.19.1a PHP/4.0.36 0D 0A 43 6F 6E 6E 65 63 74 69 6F 6E 3A 20 63 6..Connection: c6C 6F 73 65 0D 0A 43 6F 6E 74 65 6E 74 2D 54 79 lose..Content-Ty70 65 3A 20 74 65 78 74 2F 68 74 6D 6C 3B 20 63 pe: text/html; c68 61 72 73 65 74 3D 69 73 6F 2D 38 38 35 39 2D harset=iso-8859-31 0D 0A 0D 0A 3C 21 44 4F 43 54 59 50 45 20 48 1....<!DOCTYPE H54 4D 4C 20 50 55 42 4C 49 43 20 22 2D 2F 2F 49 TML PUBLIC "-//I45 54 46 2F 2F 44 54 44 20 48 54 4D 4C 20 32 2E ETF//DTD HTML 2.30 2F 2F 45 4E 22 3E 0A 3C 48 54 4D 4C 3E 3C 48 0//EN">.<HTML><H45 41 44 3E 0A 3C 54 49 54 4C 45 3E 34 30 33 20 EAD>.<TITLE>403 46 6F 72 62 69 64 64 65 6E 3C 2F 54 49 54 4C 45 Forbidden</TITLE3E 0A 3C 2F 48 45 41 44 3E 3C 42 4F 44 59 3E 0A >.</HEAD><BODY>.3C 48 31 3E 46 6F 72 62 69 64 64 65 6E 3C 2F 48 <H1>Forbidden</H31 3E 0A 59 6F 75 20 64 6F 6E 27 74 20 68 61 76 1>.You don't hav65 20 70 65 72 6D 69 73 73 69 6F 6E 20 74 6F 20 e permission to 61 63 63 65 73 73 20 2F 63 67 69 2D 62 69 6E 2F access /cgi-bin/61 64 63 79 63 6C 65 2F 61 64 63 79 63 6C 65 2E adcycle/adcycle.63 67 69 0A 6F 6E 20 74 68 69 73 20 73 65 72 76 cgi.on this serv65 72 2E 3C 50 3E 0A 3C 48 52 3E 0A 3C 41 44 44 er.<P>.<HR>.<ADD52 45 53 53 3E 41 70 61 63 68 65 2F 31 2E 33 2E RESS>Apache/1.3.31 39 20 53 65 72 76 65 72 20 61 74 20 74 68 65 19 Server at the62 61 62 79 63 6F 72 6E 65 72 2E 63 6F 6D 20 50 babycorner.com P6F 72 74 20 38 30 3C 2F 41 44 44 52 45 53 53 3E ort 80</ADDRESS>0A 3C 2F 42 4F 44 59 3E 3C 2F 48 54 4D 4C 3E 0A .</BODY></HTML>.

10


• Correlating data from multiple sources– Normalizing

• Same events may have different names depending upon the source

– Translating IDS codes» Cisco NetRanger: 4052» ISS RealSecure: Chargen_Denial_of_Service

• Use to build a chain of evidence

11


• Correlating data from multiple sources– Deconfliction

• Same event shows up multiple times with same names– Certain types of denial of service attacks– Some penetration attacks

» Use care not to remove individual steps in an attack scenario

• Same event repeated so rapidly that the logging device reports a large number of the same event in a very short (sometimes sub-second) period of time

• Multiple rapid events that make an attack scenario such as a port scan

• Deconflicted events are used with normalized data to create an event timeline

12


• Correlating data from multiple sources– Creating chain of evidence and event timelines

• Using deconflicted and normalized events on multiple data sources, chart the chain of events into an event timeline

– Carefully note the timebase of various data sources and correct to a common timebase

– Note events and attack scenarios – correlate connected events into scenarios

• Document every assumption with evidence and, if possible, corroboration using both forensic and traditional investigation

13


• Forensic handling of deleted or modified logs– Useful only in certain types of systems

• Recovering deleted logs– System must support recovery of ambient data

• Recovering altered logs– Logging source must delete old log and create a

new one when the log is altered– System must support recovery of ambient data

Web Server Log Analysis

SD'98 (c) David Strom, Inc. 15

Different types of log files

• Access

• Error

• Referral

• Other


Access logs

• Domain name

• Date, time

• Server command processed and result

• URL of visitor

• Bytes transmitted


Sample access log data• rm258.fav.usu.edu [31/May/1995:09:03:23 +0600] "GET /NEI.html

HTTP/1.0" 302 396• rm258.fav.usu.edu [31/May/1995:09:03:28 +0600] "GET

/xculture/nei/nei.html HTTP/1.0" 200 2114• rm258.fav.usu.edu [31/May/1995:09:03:30 +0600] "GET

/gifs/sedlbutton.gif HTTP/1.0" 200 1336• 129.71.83.161 [31/May/1995:09:20:32 +0600] "GET /RELs.html

HTTP/1.0" 304 0• Leslie-Francis.tenet.edu [31/May/1995:09:36:06 +0600] "GET /

HTTP/1.0" 200 1867• ls973.ulib.albany.edu [31/May/1995:09:40:52 +0600] "GET

/viii1.html HTTP/1.0" 404 244


Errors reported in your logs

• Clients that time out (or leave in frustration!)

• Scripts that don’t produce any output

• Server bugs

• User authentication or configuration problems


Sample error log data• [Thu May 30 07:25:32 1996] send timed out for bamberg.sedl.org• [Thu May 30 07:57:41 1996] send timed out for kenya.sedl.org• [Thu May 30 08:23:11 1996] send timed out for ppp092.kyoto-

inet.or.jp• [Thu May 30 09:15:52 1996] access to

/usr/local/www/htdocs/scimath/compass/vol03 failed for 170.211.67.51, reason: File does not exist

• [Thu May 30 09:57:56 1996] send timed out for dd10-048.compuserve.com

• [Thu May 30 10:47:25 1996] read timed out for ncia110b.ncia.net


Referral logs

• Who links to your site?

• Who downloads your pages?


Sample referral log data• http://www.isisnet.com/ ->/change/welcome.html• http://www.ipl.org/ref/RR/EDU/Research-rr.html ->/welcome.html• http://www.tenet.edu/snp/main.html ->/policy/networks/toc.html• http://www.tenet.edu/new/main.html ->/policy/networks/toc.html• http://guide-p.infoseek.com/NS/Titles?qt=teacher+training

->/resources/SCIMAST/announcement.html• http://www.tenet.edu/new/main.html ->/policy/networks/toc.html• http://www.tenet.edu/new/main.html ->/policy/networks/toc.html• http://www.nwrel.org/national/regional-labs.html

->/welcome.html


Common log format

• Output by most standard servers

• Needed by most third-party log analyzers• hoohoo.ncsa.uiuc.edu/docs/setup/httpd/Overview.html


Extended/custom log formats

• Log whatever you wish in whatever order you wish

• Useful if you will read them regularly!

• But can’t work with the analyzers

• Now in IIS v4, NSCP v3, others.


What you can learn from your log files

• Hits per day

• Domain origins

• The path people take in and around your web

• Problem areas


HITS

• (How Idiots Track Success)

• Nobody uses this word anymore

• Doesn’t really measure individual users, just access

• Catching servers and proxies mess up these statistics


Domain origins

• Where users are coming from -- sometimes

• Just because they are from ibm.net doesn’t mean they work at IBM!

• Forgotten accounts, friends and family using the account

• Hacked user names

• Proxies don’t help here either


The path people take in and around your web

• Search engines help sometimes

• Which search site was the most popular front door

• Who links to you and why

• Is there a pattern or a random walk?


Problem areas to deal with

• Broken links (locally)

• Broken outbound links

• Time outs (sunspots?)


What you can’t learn from your logs

• Who are these people, anyway?– No specific user names– Is it a bot or a real human?

• How long did they view a page?– Most people don’t spend much time on your

web– Where did they go visit next?


What technologies are available?

• Built-in analyzer tools

• Sites that capture user info

• Secure sites with registration

• Build your own from perl

• Third-party tools


Built-in tools

• WebSite, website.ora.com

• IIS with Site Server, www.microsoft.com/iis

• Netscape servers, www.netscape.com

• Easy to use but limited


WebSite Professional v2

• Win NT, 95

• Best web server for learning about logs, best docs

• QuickStats module for instant analysis:– single report but nice set of information– shows today, last two days requests and

unique hosts– IP addresses of visitors, average

requests/hour


IIS Site Server

• NT Server v4 w/SP3 only

• Lots of preconfigured reports

• Two versions, Express and Full (customized reports)

• backoffice.microsoft.com/products/siteserver/express/


Netscape v3 web servers

• Various NT, Unix versions

• Reports for a few variables but nothing too extensive

• Best to use a third-party tool here


Sites that capture user info

• WebCounter, www.digits.com -- third-party hit counter

• Someone else does the programming and debugging

• But beyond your control


Secure sites with registration

• You know your users

• But many won’t register, or forget their passwords

• Requires scripting, database integration, more maintenance


Build your own from perl

• Needs some in-house support

• Works best with Unix-based webs

• Examples:– refstats,

members.aol.com/htmlguru/refstats.html– surfreport, bienlogic.com/SurfReport/


Third-party tools

• WebTracker, www.CQMInc.com/webtrack

• WebTrends, www.webtrends.com

• net.Genesis, www.netgen.com

• MarketWave, www.marketwave.com

• IIS Assistant, www.go-iis.com


Third-party tools (con’t)

• Can make very pretty reports

• Customizable

• Make sure they support your particular log format

• Not that expensive, mostly run on Windows

log analysis and intrusion detection

Documents

eth0 nile

nile login

domain nile

system log logs

eth0 arp

log fileslokasi bergantung

df tos 0x10

root login