Best Known Methods in Security Events Correlation

Mohammed Fadzil Haron GSEC GCIA

April 12, 2005

2 IT@Intel

Agenda

Correlation overviewKnowledge requirementsMethodologyData representationReaction

3 IT@Intel

Correlation defined

A relation existing between phenomena or things or between mathematical or statistical variables which tend to vary, be associated, or occur together in a way not expected on the basis of chance alone…[1]

[1] http://www.webster.com

4 IT@Intel

Overview Correlation is the next security big thing in

importance An important tool in the security analyst’s toolbox

for monitoring security events To be most effective, most – if not all – events

should be examined Defense in depth means more data from different

technologies, vendors, and products Huge amount of data to analyze; terabytes in size

and growing Reduce false-positive and false-negative findings

compared to use of a single product/technology Expensive manned 24x7 monitoring capabilities

5 IT@Intel

Ultimate goal

Et = Dt + Rt Exposure time (Et): The time the resource,

information, or organization is susceptible to attack or compromise.

Detection time (Dt): The time it takes for the vulnerability or the threat to be detected.

Reaction time (Rt): The time it takes for the individual, group, or organization to respond and eliminate or mediate the vulnerability or risk.

“Time Based Security” by Winn Schwartau

6 IT@Intel

Security events flow

7 IT@Intel

Axiom on correlation

1. You only see the tip of the iceberg

2. Know the environment and perimeter of defense well

3. Don’t trust the tool; trust your judgment4. “Automate whenever possible” [1]

5. Use the simplest data representation possible

6. Balance between over-correlated and under-correlated

7. Get the big picture8. “The truth is in the packet” [1]

[1] Toby Kohlenberg, Intel Corp.

8 IT@Intel

Knowledge requirements

Know your environmentKnow your perimeter of defenseAutomate tasksSimplify data representation

9 IT@Intel

Know your environment

Knowing the ins and outs of your network is a necessity

– External network, DMZ and internal network architecture– Other networks, such as VPN and dial-up– Logistical and geographical locations of servers and users– Different operation systems, applications and functionality

of servers and client machines– Network switches and routers in use– Logistical and geographical locations of critical servers

(DNS, WINS, DHCP) as well as high-valued servers (web servers, servers containing intellectual properties)

– You cannot know everything yourself, so know the individual experts on each piece of the network puzzle

10 IT@Intel

Example of environment knowledge usage Can isolate IP addresses of Internet, DMZ and

internal network for different categorization– Potential detection of external attack versus inside job

VPN and dial-up services introduce other threats and need to be given separate consideration

Allows assignment of customized severity levels for different services, such as DNS and servers housing intellectual property, for upgraded security needs

11 IT@Intel

Source of events

Host level – Syslog, HIDS/HIPS, eventlog, log files, apps logs, anti-virus signature level

Network level – NIDS/NIPS, NBAD, firewall, network routers and switch logs, active directory logs, VPN logs, third-party authentication logs

Audit – Vulnerability scanning, OS and patch level

Knowledgebase – Software vulnerabilities and exploits

12 IT@Intel

Know your perimeter of defense

FirewallIDSIPSAudit capabilitiesHost level defensesPENSVulnerability scanning dataAnd so on.

13 IT@Intel

Know your firewalls

Location – Outer-facing, inner-facing, DMZ, internal, internal isolated network

Type – Packet filter, stateful, application firewall/proxy

What’s allowed versus deniedCapabilities versus shortcomings

14 IT@Intel

Know your IDS/IPS

Which product deployed? NIDS, HIDS/HIPS, NIPS

Where were they deployed? What kind of traffic is being monitored?

What product/vendor deployed?Capabilities versus shortcomings

15 IT@Intel

Know your audit capabilities

Where are logs being kept? Syslog server or logs on host?

How long have logs being kept? Rotated? Know your syslog servers

16 IT@Intel

Host level defenses

Anti-virus logsMinimum security specification compliance

enforcement software logsOS, service packs, patches-level

information

17 IT@Intel

Automate tasks as much as possible

Daunting tasks to detect intrusion due to:– Amount of data involved reaching terabyte range

– Complexity of network environment architecture with Internet presence, DMZ, WAN, MAN, PAN, LAN, VOIP, VPN, Dial-up

– Complexity of perimeter of defense

– Large IP address ranges used internally, that is, using Class A 10.x.x.x

– Multiple internally isolated networks with different type of policies, and access controls

18 IT@Intel

What and where to automate

Data aggregation – at data source and event manager

Manual, repetitive tasks – at event manager and reaction

Data correlation – event managerSimplify data representation – event

manager consoleIncident notification – event manager

19 IT@Intel

Group your assets

Break down IP addresses into groups, such as internal, DMZ and others for Internet

Determine and group all critical servers, such as DNS, WINS, and DHCP

Determine and group all high valued servers, such as file shares, web servers, and FTP servers, and encrypted content servers for intellectual properties

20 IT@Intel

Types of correlation

Sets– String a group of events together to generate a

trigger

Sequences– String a group of events together in sequence or

particular order to generate a trigger

Statistical– Deviation of normal behavior, such as mean or

normal curve

21 IT@Intel

Methods of correlation Rule

– Manually constructed, easy to create/update. Usually explicit in nature and can be applied to set, sequence and threshold types. Contains three elements: condition, time interval, and response.

Heuristic– Similar to anti-virus signature. One signature can detect multiple

variations. More implicit than explicit in nature, thus potential for higher false positives/negatives.

Fuzzy Logic / Artificial Intelligence– Model approach to correlation that can dynamically adapt to

changing environment. Difficult to produce and still immature; very cutting-edge.

Hybrid– No one doing them all yet. Commonly used are heuristic and

rule.

22 IT@Intel

Correlation constraint

Time– Time should be considered when creating time

box correlation

– Correct time is critical in correlation

– Time synchronization is crucial

Context– Order of events sequence is important

– Context can be necessary in correlation rules

23 IT@Intel

Sample of correlation flow

INTERNETExternal

attacker’s IP address

Outer IDS detection

NO YES

Outer FirewallOuter Firewall Accept/Deny

Deny Accept AcceptDeny

DMZ IDS detection

DMZ IDS detection

NO YES

Inner Firewall

Inner Firewall

Inner Firewall

Inner FirewallDeny

Deny Deny DenyAccept

AcceptAccept

Accept

NO YES

Inner IDS detection

Inner IDS detection

Inner IDS detection

Inner IDS detection

NO NONOYES YES YES YESNO

24 IT@Intel

Graphical representation

Seeing is believingPros

– Can represent huge data in simple and easy to understand graphs

Cons– Not many tools (commercial/open source) with

this capability

– If exist, limited capabilities

25 IT@Intel

Effective graphics should…

Show the dataAvoid distorting dataPresent a large volume of data in small

spaceMake large data sets coherentShow several levels of detailProvide clear purpose of data presentationRepresent the data and not the underlying

technology, methodology, and design

26 IT@Intel

Forms of data representation

GraphsLink graphChartsData mapsTime seriesNarrative graphics (space and time)AnimationVisualizationVirtual reality

27 IT@Intel

Scanning graph(One source to many target relationship)

Mar 14 08:33:20 66.34.244.12:2827 -> xxx.yyy.1.1:18905 SYN ******S*

Mar 14 08:33:20 66.34.244.12:2830 -> xxx.yyy.1.2:18905 SYN ******S*

Mar 14 08:33:20 66.34.244.12:2833 -> xxx.yyy.1.3:18905 SYN ******S*

Mar 14 08:33:22 66.34.244.12:2836 -> xxx.yyy.1.4:18905 SYN ******S*

Mar 14 08:33:22 66.34.244.12:2839 -> xxx.yyy.1.5:18905 SYN ******S*

Mar 14 08:33:22 66.34.244.12:2842 -> xxx.yyy.1.6:18905 SYN ******S*

Mar 14 08:33:22 66.34.244.12:2845 -> xxx.yyy.1.7:18905 SYN ******S*

Mar 14 08:33:20 66.34.244.12:2848 -> xxx.yyy.1.8:18905 SYN ******S*

Harder to internalize

Scan activity easily recognized

28 IT@Intel

Link graph

Stage 1 of worm

propagation

29 IT@Intel

Link graph

Stage 2 of worm

propagation

30 IT@Intel

Link graph

Stage 3 of worm

propagation

31 IT@Intel

Moving average(Simple network anomaly detection)

0

20

40

60

80

100

120

140

160

180

1 2 3 4 5 6 7 8 9

Monitored Events

Moving Average

Increase in moving average, showing an increase in activities

Example: Monitoring port 445

32 IT@Intel

Animation movie

Inbound connection attempts to San Diego State University (SDSU) from external source (unauthorized)

Representing 332 GB of raw data, 3.4 billion raw syslog records, and 1 million events

Period of 1996-2002 (6 years) Available at

http://security.sdsc.edu/probes-animations/index.shtml

33 IT@Intel

Animation movie

34 IT@Intel

Reaction to correlated data

Enforcement for malware cleaningBlocking to minimize malware propagation

and attackInvestigation for malicious non-worm

activitiesLearning mode for improving data (reducing

false-positives and false-negatives)

35 IT@Intel

Conclusion

Correlation is a must tool for information security professionals

Time saved in detection will allow faster response time

Faster response time will minimize damages to your assets

36 IT@Intel

Questions?

37 IT@Intel

References

Event correlation; http://www.computerworld.com/networkingtopics/networking/management/story/0,10801,83396,00.html

“Protecting the Enterprise with Scalable Security Event Management, Part II - Intelligent Event Correlation”; Michael Mychalczuk; https://www.sans.org/webcasts/show.php?webcastid=90468

“Thinking about Security Monitoring and Event Correlation“; http://www.securityfocus.com/infocus/1231