HyperSpector: Virtual Distributed Monitoring Environments forSecure Intrusion Detection

Kenichi Kourai Shigeru Chiba

Tokyo Institute of Technology

Distributed intrusiondetection system (DIDS)

Useful to achieve self-monitoring of distributed systems◆ Towards self-protection

Consists of multiple IDSes◆ Including• Host-based IDS (HIDS)

• Network-based IDS (NIDS)◆ IDSes cooperate with each other

or with an analyzer

IDS

analyzer

distributed system

server

Threats against the DIDS Active attacks

◆ Directly take actions against IDSes by• Sending malicious packets to network ports

used by IDSes

• modifying IDS policy files or terminating IDS processes

Passive attacks◆ Wait until IDSes read data including

malicious code by• Sending malicious packets to monitored

servers

• Changing attributes of monitored files

server

IDS

activeattack

monitor

passiveattack

Traditional approach:Isolated monitoring

Isolates NIDSes from servers physically◆ Using NIDS hosts and a back-end

switch

NIDS hosts monitor packets by port mirroring◆ NIDS hosts are connected to

mirroring ports in a front-end switch◆ The front-end switch duplicates

and forwards packetsInternet

back-endswitch

front-endswitch

serverhost

NIDShost

DIDS

mirroringport

Internet

back-endswitch

front-endswitch

serverhost

NIDShost

DIDS

Security ofisolated monitoring

Prevents active attacks◆ The attacker cannot attack NIDS

hosts using mirroring ports

• Mirroring ports are only for monitoring

Confines the impact of passive attacks to within the DIDS◆ The attacker cannot access the

outside of the DIDS◆ Important because preventing

passive attacks is difficult

mirroringport

Problems inisolated monitoring

Need additional hardware◆ Lots of machines for NIDSes◆ A back-end switch◆ A front-end switch with port mirroring

Support only NIDSes◆ Legacy HIDSes do not support monitoring of

remote server hosts◆ Achieving secure monitoring of remote server

hosts from HIDS hosts is difficult

Our approach: HyperSpector

Virtual distributed monitoring environment◆ IDS VM and server VM• Isolate each other without

additional hardware

• The IDS VM can monitorthe server VM

◆ A virtual network• Connects the IDS VMs

• Isolated from a network usedby servers

serverVM

serverVM

serverVM

serverVM

virtual network

IDSVMIDSVM

IDSVMIDSVM

DIDS

Inter-VMmonitoring mechanisms

Requirements◆ Interfaces to legacy IDSes◆ Secure monitoring between VMs

HyperSpector provides three mechanisms◆ Software port mirroring (for packet capturing)◆ Inter-VM disk mounting (for file system checking)◆ Inter-VM process mapping (for process checking)

Software port mirroring

Virtual switch◆ Achieves port mirroring by

software◆ Connects its mirroring port to

the IDS VM• Using a virtual network

interface (VNI)

◆ Duplicates and forwards packets to the IDS VM

virtualswitchvirtualswitch

NIDS

BPFdevice

VNI

server VM IDS VM

VMM

mirroring port

outside

Inter-VM disk mounting

Inter-VM disk mounter◆ Mounts the file system of the

server VM on the IDS VM• As a shadow file system

◆ Forwards requests to a shadow file system to the server VM• Using VMM interfaces

inter-VMdisk mounter

inter-VMdisk mounter

HIDS

server VM IDS VM

VMM

VMMinterface

filesystem

shadowfile system

read

Inter-VM process mapping

Inter-VM process mapper◆ Maps the processes in the

server VM to the IDS VM• As shadow processes

◆ Forwards• Requests to shadow processes

to the server VM

• Notifications from the server VM to HIDSes

– Using VMM interfaces

inter-VMprocess mapper

inter-VMprocess mapper

HIDS

server VM IDS VM

VMM

VMMinterface

serverprocess

ptrace

shadowprocess

wakeup

serverVM

serverVM

serverVM

serverVM

virtual network

IDSVMIDSVM

IDSVMIDSVM

Security of HyperSpector Prevents active attacks

◆ From the server VMs◆ From hosts outside the DIDS

Confines the impact of passive attacks◆ The IDS VM cannot attack the

server VM◆ The IDS VM cannot attack

hosts outside the DIDS

DIDS

serverVM

serverVM

serverVM

serverVM

IDSVMIDSVM

IDSVMIDSVM

Security of the inter-VM monitoring mechanisms

Secure, because◆ The server VM cannot use inter-VM

monitoring mechanisms◆ The IDS VM cannot interfere with the

server VM• Inter-VM monitoring mechanisms are

only for monitoring◆ The IDS VM cannot send monitored

information outside the DIDS • Although it can view secret

information of servers...

serverVM

serverVM

IDSVMIDSVM

VMM

modify

outside hosts

monitor

request

Implementation We have implemented HyperSpector in the

FreeBSD kernel IDS VM and server VM

◆ Based on our portspace• The portspace virtualizes only

a network system, file system,and processes

◆ Secure enough• We assume the kernel and the

base system are not exploitable kernel

basesystem

IDS VMserverVM

VMM

net netfs fsfs

Implementation of the VMM

Implemented efficiently in the kernel◆ Virtual switch• Maps a network interface of the server VM to the IDS

VM in a read-only manner

◆ Inter-VM disk mounter• Mounts the file system of the server VM on the IDS

VM read-only, using the modified union file system

◆ Inter-VM process mapper• Makes the IDS VM share the processes of the server

VM in a read-only manner

Experiments

We measured overhead of HyperSpector◆ Experimental setup• Snort, Tripwire, or truss in the IDS VM

• thttpd in the server VM

• ApacheBench in the client host

◆ Hardware• 2 PCs (3.0 GHz Pentium 4,

1 GB of memory, Intel Pro/100+)

• 100Base-T network switch

IDSVM

IDSVM

serverVM

serverVM

clienthost

server host

Snort Monitors packets from

ApacheBench to thttpd◆ We measured the

throughput of thttpd◆ For comparison• The base system

• Isolated monitoring

Maximum overhead◆ 7.5% slower than the base system◆ 7% slower than isolated monitoring (over 2 KB file size)• 30% in 0 KB file size

Tripwire Checks the integrity of

the whole file system◆ 54,885 objects◆ We measured the time

of the integrity check• altering the file change

rate◆ For comparison• The base system

Overhead◆ 17 to 26% slower than the base system

Truss Traces system calls

issued by thttpd◆ We measured the

throughput of thttpd• Using ApacheBench

◆ For comparison• The base system

Overhead◆ 0.8 to 7.3% slower than the base system

Related work ReVirt [Dunlap’02], Livewire [Garfinkel’03]

◆ Enable IDSes to monitor servers running in a VM• The VM protects IDSes from active attacks via servers

◆ Do not consider other attacks against IDSes

Backdoors [Bohra’04]◆ Enables isolated monitoring for HIDSes• Using programmable NICs to monitor server state

◆ Needs much hardware◆ Insecure because HIDS hosts are network-reachable

These need to develop specialized IDSes

Conclusion

We proposed HyperSpector, which◆ Isolates IDSes from servers without additional

hardware• Using IDS VMs, server VMs, and a virtual network

◆ Provides secure Inter-VM monitoring mechanisms:• Software port mirroring, inter-VM disk mounting, and

inter-VM process mapping

◆ Prevents active attacks and confines the impact of passive attacks to within the DIDS

Future work

Support for active monitoring◆ Needs a mechanism to securely send probe

messages to servers

Support for DoS attacks◆ Needs to allocate sufficient resources to the IDS

VM even under overload

Automatic detection of compromised HyperSpector◆ Monitoring resource usage may help