A Hybrid, Stateful, and Cross-Protocol Intrusion Detection

System for Converged Applications

Department of Electrical Engineering

University of Cape Town

Bazara Barry and H. Anthony ChanBazara Barry and H. Anthony Chan

UCT-COE Seminar000 Page 2 April 21, 2023

Contents

Introduction Formal model System Architecture Related Work Implementation and Experiment Future Work

UCT-COE Seminar000 Page 3 April 21, 2023

Intrusion Detection Systems

Set of techniques and methods to detect suspicious activities at the network or host level

Two main categories of IDS exist:

–Knowledge-based (misuse)

–Behavior-based (anomaly)

A relatively new approach in Behavior-based detection is Specification-based anomaly detection.

UCT-COE Seminar000 Page 4 April 21, 2023

Intrusion Detection Systems

Specification-based detection can be classified into two categories: Syntax anomaly detection.

(checks if messages are well-formed)

Semantics anomaly detection. (monitors the sequence of commands)

Specifications are developed based on standards approved by organizations such as IETF.

UCT-COE Seminar000 Page 5 April 21, 2023

Convergence

Convergence in networks refers to the structures and processes that result from design and implementation of a common networking infrastructure that accommodates data, voice, and multimedia communications [1].

Convergence in applications refers to the building of applications that span over different protocols/specifications [2].

UCT-COE Seminar000 Page 6 April 21, 2023

Convergence

Sharing the same physical infrastructure with data networks makes convergence inherit all the security weaknesses of IP protocol.

VoIP standards separate signaling and media on different channels.

Standardized on Open Technologies: SIP, H.323, and thus vulnerable to attacks.

UCT-COE Seminar000 Page 7 April 21, 2023

Session Initiation Protocol (SIP)

An application layer protocol that is used for establishing, modifying and terminating multimedia sessions in an IP network [3].

SIP is susceptible to Denial of Service, Eavesdropping, Tearing down sessions, Registration Hijacking, and Session Hijacking.

UCT-COE Seminar000 Page 8 April 21, 2023

A Finite State Machine (FSM) is a model of behavior Composed of a finite number of states, transitions between those states, and actions.

An FSM extended with parameters, variables, predicates, and operations is what is understood by an Extended FSM (EFSM).

Communicating Extended Finite State Machines Model

UCT-COE Seminar000 Page 9 April 21, 2023

attack

0

2

1

attack

In_Packet()0/Out_packet()0

0

2

1

In_Packet()1/Out_packet()1

In_Packet()0/Out_packet()0

In_Packet()1/Out_packet()1

Network protocol 1Network protocol 2

Communicating Extended Finite State Machines Model

UCT-COE Seminar000 Page 10 April 21, 2023

System Architecture

Incoming VoIP traffic

Filter

Semanticssignatures

Syntaxsignatures

Admin

Packet verifier

SIPRTP

Behavior observer

Admin

UCT-COE Seminar000 Page 11 April 21, 2023

Advantages of System Architecture

Stateful Detection. Cross-Protocol Detection. Knowledge-based and Behavior-based detection. Syntax and Semantics Anomaly Detection.

UCT-COE Seminar000 Page 12 April 21, 2023

Stateful Cross-protocol

Knowledge-based

Semantics anomaly detection

Syntax anomaly detection

STAT[4]

NetSTAT[5]

WebSTAT [6]

SCIDIVE[7]

vIDS[8]

Our proposed IDS

Related Work

UCT-COE Seminar000 Page 13 April 21, 2023

Implementation and Experiment

The system is developed based on SIP servlet programming model and the SIP servlet API. The SIP servlet specification allows applications to perform a fairly complete set of SIP signaling based on SIP standards (e.g. RFC 3261).

The API gives the developer full control to handle SIP messages by allowing full access to headers and body, responding to or rejecting requests, and initiate requests.

UCT-COE Seminar000 Page 14 April 21, 2023

Implementation and Experiment

Five attacks are implemented to test the system: BYE Attack Re-INVITE Attack REGISTER flooding Attack CANCEL Attack Buffer Overflow Attack

UCT-COE Seminar000 Page 15 April 21, 2023

Future Work

Currently we are investigating the runtime impact of the system on VoIP applications.

The efficiency of the system will be improved by developing more abstract modules in the packet verifier and the Behavior observer to reduce the number of false positives.

A possible extension is to adopt the standard of (IDWG) for message exchange to turn it into a distributed system.

UCT-COE Seminar000 Page 16 April 21, 2023

References

1. T. Porter, Practical VoIP Security (Syngress Press, 2006), p.6.2. N. Khan, “The SIP Servlet Programming Model, ” Technology white paper, 31st January

2007, Available at: http://dec2dev.bea.com. [2007 April]. 3. SIP RFC (3261).4. P. Porras, “STAT -- A State Transition Analysis Tool For Intrusion Detection,” Technical

Report: TRCS93-25, University of California at Santa Barbara, 1993.5. G. Vigna and R. Kemmerer. NetSTAT: A Network-based Intrusion Detection Approach.

In Proceedings of the 14th Annual Computer Security Application Conference (ACSAC 1998), Scottsdale, Arizona, December 1998.

6. G. Vigna, W. Robertson, V. Kher, and R. Kemmerer. A Stateful Intrusion Detection System for World-Wide Web Servers. In Proceedings of the Annual Computer Security Applications Conference (ACSAC 2003), pages 34–43, Las Vegas, NV, December 2003.

7. Y. Wu, S. Bagchi, S. Garg, N. Singh, T. Tsai, “ SCIDIVE: A Stateful and Cross Protocol Intrusion Detection Architecture for Voice-over-IP Environments,” In Proceedings of the 2004 International Conference on Dependable Systems and Networks (DSN’04).

8. Hemant Sengar, Duminda Wijesekera, Haining Wang, and Sushil Jajodia, “VoIP Intrusion Detection Through Interacting Protocol State Machines,” In the proceedings of Dependable Systems and Networks, 2006. DSN 2006, Sheraton Society Hill, Philadelphia, PA, USA

Questions