george tadda fusion technology branch information directorate air force research laboratory e-mail:...

George TaddaFusion Technology Branch

Information DirectorateAir Force Research LaboratoryE-mail: [email protected]

Phone: 315-330-3957

INFORMATION FUSION FOR CYBER SITUATION AWARENESS

Approved for Public Release #

Outline

• Introduction

• Motivation

• Situation Awareness Reference Model

• Metrics

• Application of Lessons Learned


Work in Situation Awareness (SA)

• Used reference models to demonstrate/build prototype systems for:• Cyber (Defense & Security (D&S) ’05, SIMA ‘05)• Tactical (ISIF ’02)• Global (ISIF ’04)• Maritime • and Many Others

• Developed Metrics (D&S ’04) to Evaluate Level 2 Systems and applied them to Cyber (D&S ’05)– After much discussion we questioned the difference between tracking objects and situations and whether the majority of the metrics are just another way to measure integrity of tracks

• Additional Activities:– Jean Roy, under The Technical Cooperative Program, presented a

definition of situational analysis and included in "Concepts,

Models, and Tools for Information Fusion“– Snidaro, M. Belluz, G. Foresti, “Domain knowledge for

security applications”, ISIF’07 defined types of events (simple,

spatial, and transitive)– Dale Lambert, formalizing situation awareness through mathematics

07-210


Motivation(Reality of Today’s Environment)

…and MORE

Today WE Have…

Tac

tica

lC

yber

Glo

bal

Moving Objects 80/sec

Data

Data

Data

The Analyst/Operator

Drowning in data and Inundated with “dots” on map or messages. INFORMATION STARVED

INCOMPLETE, CONFLICTING DATA

SA is Highly Operator Dependent and 100% Mental Process

- Stress

- Fatigue

- Experience

LIMITED BY INDIVIDUAL’S ABILITIES

Class B Address Space

26,000 Alerts/day

3 – 4 Petabytes/day

(E-mail, Published Pages, etc)

1000’s of Objects

07-291


Motivation

SensemakingWhat is…

…and MORE

Today WE Have…

Tac

tica

lC

yber

Glo

bal

Data/Information Ratio (DIR)

(Examples)

Pre Iran/Kuwait

Conflict

Objects*: 16,203No. Units: 42 DIR: 386

Events Attacks DIR20,131 107 18819.531 66 296 8,681 62 14031,513 155 203

SKAION Datasets

(3s8, 3s26, 3s28, 3s29)

*No noise/clutter

A Measure of Success

TRENDS(Economic, Military)

Knowledge Of Atcks

SPATIAL(Obj Types/No.)

Unit Axx

xx

xx

xx

xx

Data Information

Data Information

Data Information

ALERTS

TRENDS(Network, Host)

EVENTS

Knowledge Of Sits

Knowledge Of Units

Plausible Futures(Intent, Opportunity,Capability)



Information Assessment




AnticipationMost Likely/

Worst Case eCOA

(STEP 2: From Complex Relations/Situation(s) -> Anticipation)(STEP 1: From Data -> Complex Relations/Situation(s))

07-291


Sharing the Stage(From A Model Perspective)

HumanComputerInteraction

Level 0Processing

Sub-object

DataAssessment

Level 1Processing

ObjectAssessment

Level 2Processing

SituationAssessment

Level 3Processing

ImpactAssessment

Level 4Processing

ProcessRefinement

Data BaseManagement System

SupportDatabase

FusionDatabase

DATA FUSIO N DO MAIN

• Most popular is the Joint Director’s of Laboratory (JDL) Model (Sensor-based)• Functional Model• 5 Levels (Level 0, 1, 2, 3, 4)• Published By Llinas, Hall, White (1992)• Most work concentrated on Level 0/1/4 (Dots on Map)• Little definition of Level 2/3 (What do they mean?)• Bottom-up, Data Driven

.

Information ProcessingMechanisms

Long TermMemory Stores Automaticity

Task/System Factors

DecisionState of the environment

feedback

Individual Factors

• Abilities• Experience• Training

• System Capability• Interface design• Stress and workload• Complexity, Automation

• Goals & Objectives• Preconceptions (expectations)

Perceptionof Elementsin CurrentSituation

SITUATIO N AW ARENESS

Projectionof Future

Status

Compre-hension

of Current Situation

Performanceof actions

• Receiving Much Attention Today from the Cognitive Community• Mental Model• 3 Levels: Perception, Comprehension, Projection• Developed by: M. Endsley (1995)• Extended by McGuinness and Foy for Resolution• Top Down, Goal Driven

FUSION - TACTICAL SITUATION AWARENESS

BALANCED

ata

Fo

cuse

d

Go

al F

ocu

sed

07-291


ModelAnalysis

Tools

SourcesData

Collection

Data Cleansing

ParsingExtraction

Evidence

Pe

rce

pti

on

Co

mp

reh

en

sio

n

Level 0/1

KnowledgeDiscovery

Tools

TargetModels

The “Problem”

Sources

Data

Requirements

KnowledgeDiscovery

Tools

AN

TIC

IPA

TIO

N

Potential New Relationships

Matches/Partial Matches

Info

Additional

*MissedQuestions

*Based on Model Unfolding

Situation Awareness Reference Model(Combining The “Best” Of Both Worlds)

• Based on JDL & Endsley’s Models

- Plus Initial Data Requirement

- Textual Inputs (Info Exploit)

• Define Problem/Goal – Top Down

- What/Where/Who…

• Processing Flow ( )

- Projection – The Alert(s)

- Comprehension

-- Model Analysis

- Perception

-- Data Collection

-- Parsing/Extraction

-- Data Cleansing

- JDL: Level 0/1

• Process Refinement ( )

- Missing Data

- Additional Data

- Input for Sensor Mgmt

• Off-Line Processing ( )

- Knowledge Discovery

Assessment

Situation

07-291


MissionClient/Host

Configurations

ModelAnalysis

Tools

SourcesData

Collection

Data Cleansing

ParsingExtraction

EvidenceP

erc

ep

tio

nC

om

pre

he

ns

ion

Level 0/1

KnowledgeDiscovery

Tools

TargetModels

The “Goal”

Sources

Data

Requirements

KnowledgeDiscovery

Tools

AN

TIC

IPA

TIO

N

Potential New Relationships

Matches/Partial Matches

Info

Additional

*MissedQuestions

Situation Awareness Reference Model(Applied to Cyber SA)

Assessment

Situation

Pe

rce

pti

on

Snort Dragon

Web LogsSys Logs

Evidence

(Alerts)

Network Stats

Post Proc

Host IDS

Data Cleansing

Open Source

Co

mp

reh

en

sio

n

ReconIntrusion

Attempt

Privilege

Escalation

A Priori KnowledgeAnticipation

ParsingExtraction

DataCollection

Potential

AttacksAtt

ack

A

Eq

uip

Fai

l

Att

ack

N

Model Matching

Algorithm

07-291


Situation Awareness Reference Model(Applied to Cyber Domain)

Comprehension

Attack A

ReconIntrusion

Attempt

Goal

Privilege

EscalationMulti-Stage

Attack

Model Template XA Priori

Knowledge

Model Template Y

Attack B

Equip Failure

Model Matching

Algorithm

Potential

Attacks

Perception

Snort

Dragon

Web Logs

Sys Logs

Host

IDSEvidence

Th

e N

etw

ork

Network Stats

Po

st

Pro

c

Anticipation

Business

Model

ImpactAssessment

Potential to

Advance to

Next Stage

Client/Host

Configurations

List Based

On Risk

Open

Source

TBD07-291


Lexicon(Background)

• Evidence

– IDS Alerts (i.e., Snort, Dragon)

– System Logs

– Service Logs (i.e., Apache, IIS)

– Network Flow Data

• Track – collection of all evidence available against one or more targets originating from one or more attackers

• Situation – set of tracks at a snapshot in time

• Situation Awareness of a Network – analyst’s mental model of the situation

• True Positive* – successful attack

• False Positive* – incorrectly identified attack

• Non-relevant Positive* – correctly identified attack that fails or is incomplete (i.e., try to exploit a ‘blocked’ vulnerability)

*Valeur et al, “A Comprehensive Approach to Intrusion Detection Alert Correlation, IEEE Transactions on Dependable and Secure Computing, Jul-Sep 04

06-081


Metrics Overview

• Confidence – measures the ability of the system to correctly identify the track(s)

– Recall: Percentage of tracks detected in relation to the “total known”

– Precision: Percentage of correct tracks detected in relation to number of detections

– Fragmentation: Percentage of tracks reported as multiple tracks that should have been reported as a single track

– Mis-Association: Percentage of tracks that are neither correct nor a fragment in relation to the number of detections

• Purity – characterizes the quality of the detections

– Mis-Assignment Rate: Percent of evidence incorrectly assigned to a given track

– Evidence Recall: Percentage of evidence detected in relation to the “total known” evidence

• Cost Utility – a single weighted measure of the system in identifying “important or key” tracks with respect to a concept of cost

• Timeliness – measures the ability of the system to respond within time requirements of a particular domain

06-081


GT0 Background Scan

GT1 Background Attack

GT2 Background Scan

GT3 Attack

GT4 Background Scan

R0 Background Scan

R1 UNASSIGNED

R2 Attack

R3 Background Scan

R4 Background Scan

R5 Background Attack

Weighted Cost =

Attack Score =

Ground Truth100 pts ATTACK

5 pts Background Scan

5 pts Background Attack

-50 pts False Positive

∑ Weighted Values for Ground Truth

∑ Weighted Values for Results

5

5

5

100

5

120

5

- 50

100

5

5

5

70

Proposed Attacks

Cost Utility(Weighted Cost and Attack Score)

NOTE: Sorted Based on Score

Given:

Weighted Cost = 70/120 = .5833

Attack Score = [(1)(6) – (2-(1-1))]/(1)(6) = .6667

[No. Attacks in Results][No. Results] – [[Sum of Positions of Attacks in Results] – [Geometric Sum ([No Attacks in Results] -1)]]

[No. Attacks in GT][No. Results]

06-081


The Infrastructure

Viewing Ground Truth

Metric Report

RE

PO

RT

S

Alerts correlated to selected Attack Track

List of Potential

Attacks

Filter by score

Play Buttons

Assignment Matrix

(Confidence, Purity, Cost)

Processing Results

.csv.html

ResultsUsing

AFRL Schema

Skaion Dataset

Cyber Fusion System

Ground Truth

AFRL Results

Analyzer ToolsAFRL

Ground Truth

Correlation

06-081


Work has Raised Many Questions … Resulting in Few Answers

• Where do groups, events, activities fit in?

– Can we not track a group, an activity (Why only Objects?)

– Is a group or activity only a complex object?

• What is a Situation? Is there more than one? Is it Context-based?

• Where does Knowledge Discovery exist? Forensics?

• What is Situation Assessment?

• Is Threat Assessment only of the future – what about current threat?

• What about forecasting or projecting the “future” state?

No one model answers ALL of these questions and

even addresses them!

07-210


…so Then What

• Treating Situation as a composite of activities and tracking activities as complex objects allows for a “cleaner” distinction between fusion levels

– Situation(s)-> Activity(s) -> Group(s)/Entity(s) -> Event(s): These are ALL OBJECTS THAT CAN BE TRACKED

– Object Assessment has really been performing Tracking & Identification – LET’S TRACK ALL TYPES OF OBJECTS

• Knowledge Discovery and a priori knowledge necessary and integral to building “complex” objects (e.g., Groups, Activities)

– Updating knowledge/relationships (models) is continuous and part of refinement process

• Define Situation Assessment based on Jean Roy’s Definition for Situational Analysis:

– Behavior Analysis – Activity Level Analysis

– Intent Analysis – Salience Analysis

– Capacity/Capability Analysis – Impact Analysis

– Threat Analysis07-210


…and

• Use Time to distinguish between JDL Level 2 and 3 as does Endsley’s comprehension and projection

– Same analysis is done for both levels only difference is time

– Thus JDL Level 2 is assessment of “current situation and JDL Level 3 is the assessment of the current situation projected forward in time.

• Process Refinement involves not only sensor movement/collection (sensor management) BUT fusion algorithm management (which algorithms and which parameters to use) and model management from ALL processes. Possible sources to refinement include:

L1: Prediction where object is moving/next event

L2: Missing data, increase certainty of current assessments

L3: Forecasted actions/placement to pre-position sensors

07-210


Revised Situational Awareness Reference Model (Based on Previous Suggestions)

*Based on JDL, Endsley’s, and Jean Roy’s work

Level 1: Object Tracking

and IdentificationLevel 2: Assessing the

Current Situation(s)

Level 3: Assessing the

Forecasted Situation(s)07-210


Wrap Up

• We proposed a revised Reference Model that includes many of the lessons learned to date

• Plans are to continue to apply this revised model to Air, Cyber and Space Situation Awareness – UNIVERSAL SITUATION AWARENESS

– …with emphasis on current and forecasted situation assessment

07-210

george tadda fusion technology branch information directorate air force research laboratory e-mail:...

Documents

public release

percentage of tracks

percentage of correct

data cleansing jdl

data collection

multiple tracks

timesituation awareness

situationtrue positive