tesis doctoral 0.4cmevent-driven principles and complex
TRANSCRIPT
Tesis Doctoral
Event-driven Principles and Complex Event Processing forSelf-adaptive Network Analysis and Surveillance Systems
Defense
Rudiger Gad
Universidad de Cadiz
2015-07-29
Introduction Thesis Overview Two Highlights Summary and Conclusion
Outline
1 Introduction
2 Thesis Overview
3 Two Highlights
4 Summary and Conclusion
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Outline
1 Introduction
2 Thesis Overview
3 Two Highlights
4 Summary and Conclusion
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Motivation
IT is used ubiquitously.
Non-operational IT?→ Severe Consequences!
Importance of IT: Critical
Computer Networks
Fundamental for IT OperationNon-operational Networks? → Non-operational IT!Importance of Networks: Critical
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Assure Operational Computer Networks
BasisInformation
DetailedAccurateUp-to-date. . .
Network Analysis and Surveillance(“Network Reconnaissance” or “Network Monitoring”)
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Network Analysis and Surveillance (NAaS)
Challenging
DistributionSizeChangeTimelinessData Volume. . .
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Event-driven Principles and Complex Event Processing (CEP) for NAaS
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Event-driven Principles and Complex Event Processing (CEP) for NAaS
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Event-driven Principles and Complex Event Processing (CEP) for NAaS
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Event-driven Principles and Complex Event Processing (CEP) for NAaS
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Event-driven Architecture (EDA) and CEP for NAaS
Powerful Capabilities
Existing Related Work
Related Work, Limitations
Focused on Specific Use CasesConceptual/Architectural FocusReal-world Applicability?
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Outline
1 Introduction
2 Thesis Overview
3 Two Highlights
4 Summary and Conclusion
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Aims
Overarching Approach
Convergence of Heterogeneous DataSources
Flexible
Applicability
Performance
Complexity vs. Usability
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Thesis Outline
Analysis of Important Properties and Requirements
Architecture for Overarching Event-driven NAaS
Evaluation Prototype
Flexibility and Convergence of Data SourcesPerformance
Improvements in Distributed Contexts
Coalescence Problems, Analysis and Solutions
Improvements for Individual Components
Threats to Validity
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Thesis Outline
Analysis of Important Properties and Requirements
Architecture for Overarching Event-driven NAaS
Evaluation Prototype
Flexibility and Convergence of Data SourcesPerformance
Improvements in Distributed Contexts
Coalescence Problems, Analysis and Solutions
Improvements for Individual Components
Threats to Validity
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
General Architecture
Important Properties and Requirements
Event-driven Architecture
Focus on the Essentials
Unified Internal Event Representation
Components
Sensors (S)Event Transformer (ET). . .
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
General Architecture
Important Properties and Requirements
Event-driven Architecture
Focus on the Essentials
Unified Internal Event Representation
Components
Sensors (S)Event Transformer (ET). . .
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Prototype Implementation and Evaluation of Flexibility and Convergence
Prototype Based on Architecture
Evaluation of Flexibility andConvergence of Data Sources
Step-wise Defined Goals
Basic SystemRelocating FunctionalityConvergence of Sensors. . .
Results
Flexible NAaSConvergence of Sensors for NAaS
Figure: Evaluation Prototype
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Prototype Implementation and Evaluation of Flexibility and Convergence
Prototype Based on Architecture
Evaluation of Flexibility andConvergence of Data Sources
Step-wise Defined Goals
Basic System
Relocating FunctionalityConvergence of Sensors. . .
Results
Flexible NAaSConvergence of Sensors for NAaS
Figure: Evaluation Prototype
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Prototype Implementation and Evaluation of Flexibility and Convergence
Prototype Based on Architecture
Evaluation of Flexibility andConvergence of Data Sources
Step-wise Defined Goals
Basic SystemRelocating Functionality
Convergence of Sensors. . .
Results
Flexible NAaSConvergence of Sensors for NAaS
Figure: Evaluation Prototype
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Prototype Implementation and Evaluation of Flexibility and Convergence
Prototype Based on Architecture
Evaluation of Flexibility andConvergence of Data Sources
Step-wise Defined Goals
Basic SystemRelocating FunctionalityConvergence of Sensors
. . .
Results
Flexible NAaSConvergence of Sensors for NAaS
Figure: Evaluation Prototype
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Prototype Implementation and Evaluation of Flexibility and Convergence
Prototype Based on Architecture
Evaluation of Flexibility andConvergence of Data Sources
Step-wise Defined Goals
Basic SystemRelocating FunctionalityConvergence of Sensors. . .
Results
Flexible NAaSConvergence of Sensors for NAaS Figure: Evaluation Prototype
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Performance Evaluation
Evaluation Setup
Packet Capturing as “Worst Case” Scenario
Results
Example ResultsIt works.Most Critical: Sensor
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Performance Evaluation
Evaluation Setup
Packet Capturing as “Worst Case” Scenario
Results
Example Results
It works.Most Critical: Sensor
0
20
40
60
80
100
100 150 200 250 300
Cap
ture
Rat
io [%
]
Packet Rate [kpps]
clj Pcap Sing.
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Performance Evaluation
Evaluation Setup
Packet Capturing as “Worst Case” Scenario
Results
Example ResultsIt works!Most Critical: Sensor
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Outline
1 Introduction
2 Thesis Overview
3 Two Highlights
4 Summary and Conclusion
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Sensors: Cooperation for Improving the Performance, Foundations
Paper: 28th IEEE AINA 2014Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Sensors: Cooperation for Improving the Performance, Foundations
Paper: 28th IEEE AINA 2014Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Sensors: Cooperation for Improving the Performance, Foundations
Paper: 28th IEEE AINA 2014Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Sensors: Cooperation for Improving the Performance, Foundations
Paper: 28th IEEE AINA 2014Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Sensors: Cooperation for Improving the Performance, Foundations
Paper: 28th IEEE AINA 2014Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Cooperative Sensors, Architecture and Application
Sensors: Hosts A to D
Controller: Host E
LogicData MergingData Consumer
Traffic Generation
Host A → Host D
Paper: IEEE ICC 2015
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Cooperative Sensors: Performance, Scalability, and Traffic Load
0
25
50
75
100
100 200 300 400 500 600
Cap
ture
Rat
io [%
]
Packet Rate [kpps]1 Sensor2 Sensors
3 Sensors4 Sensors
0
100
200
300
400
500
600
100 200 300 400 500 600
Tra
ffic
Loa
d [M
bps]
Packet Rate [kpps]1 Sensor2 Sensors
3 Sensors4 Sensors
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Cooperative Sensors: Improving Operation and Usability via Self-adaptivity
Problem
Complexity ofOperation & Usability
Solution
Self-adaptation
Example
On-demand Cooperation
Aims
Capture as much as possible.Avoid overload.Reduce # of sensors.
Apply cooperation as necessary.
0
100
200
300
400
500
0 10 20 30 40 50 60 70 80 90 100
[kpp
s]
Time [s]
Send 3Send 2Send 1Drp. 1Drp. 2Drp. 3Pkt. Rt.Recv.
Figure: Detailed Results of an Example Experiment
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Improvements for Individual Components
Example: Sensor
Packet Capturing with Java & Clojure
Analyze the optimization potential in various areas.
Paper: 20th IEEE ISCC 2015
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Raw Data Acquisition: Improved Method vs. Old Method
0 1000 2000 3000 4000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 25 50 75 100
[kpp
s]
Rel
. Std
. Dev
. [%
]
Packet Size [x100 byte]
Th.Pkt.Rt. 1 Gbps [kpps]Cap.Rt. (Dbl.Buf.) [kpps]CR Rel.SD (Dbl.Buf.) [%]
Th.Pkt.Rt. 10 Gbps [kpps]Cap.Rt. (Non-B.) [kpps]CR Rel.SD (Non-B.) [%]
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Example Results of Self-adaptive Performance-based Adjustment
0 200 400 600 800
1000 1200 1400
0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 0 2 4 6 8 10 12 14
[kpp
s]
# of
Rul
es
Time [s]
Cap. Rt. [kpps]Pkt. Rt. [kpps]
Drop Rt. [kpps]Min. # Rules
# Act. Rules
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Outline
1 Introduction
2 Thesis Overview
3 Two Highlights
4 Summary and Conclusion
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Summary
Computer Networks: Critical Importance
Assuring Operating Networks → Information
Network Analysis and Surveillance(Network Reconnaissance, Network Monitoring)
“Good” Information → challenging.
(Contradicting) Requirements and Properties
EDA and CEP to the Rescue
Related Work: Too Focused, Real World Applicability?
Thesis Aims: Overarching and Applicable NAaS
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Answered Research Questions (I)
1 What are important properties and requirements foroverarching and flexible NAaS?
→ Enumeration and Discussion of Properties andRequirements
2 Does our NAaS approach offer convergence of datasources and is it flexible?
→ Convergence works.→ The architecture is flexible.
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Answered Research Questions (I)
1 What are important properties and requirements foroverarching and flexible NAaS?
→ Enumeration and Discussion of Properties andRequirements
2 Does our NAaS approach offer convergence of datasources and is it flexible?
→ Convergence works.→ The architecture is flexible.
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Answered Research Questions (II)
3 What are the performance limits and what is the mostrelevant bottleneck?
→ Detailed Performance Analysis→ CEP and EDA for NAaS works.→ Most Important Bottleneck: Sensors
4 Can the most relevant performance bottleneck beaddressed by leveraging distributed approaches?
→ Cooperative Sensors
5 Can the increased complexity of distributedapproaches be addressed?
→ Self-adaptive On-demand Cooperation
0
20
40
60
80
100
100 150 200 250 300
Cap
ture
Rat
io [%
]
Packet Rate [kpps]
clj Pcap Sing.
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Answered Research Questions (II)
3 What are the performance limits and what is the mostrelevant bottleneck?
→ Detailed Performance Analysis→ CEP and EDA for NAaS works.→ Most Important Bottleneck: Sensors
4 Can the most relevant performance bottleneck beaddressed by leveraging distributed approaches?
→ Cooperative Sensors
5 Can the increased complexity of distributedapproaches be addressed?
→ Self-adaptive On-demand Cooperation
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Answered Research Questions (II)
3 What are the performance limits and what is the mostrelevant bottleneck?
→ Detailed Performance Analysis→ CEP and EDA for NAaS works.→ Most Important Bottleneck: Sensors
4 Can the most relevant performance bottleneck beaddressed by leveraging distributed approaches?
→ Cooperative Sensors
5 Can the increased complexity of distributedapproaches be addressed?
→ Self-adaptive On-demand Cooperation
0
100
200
300
400
500
0 10 20 30 40 50 60 70 80 90 100
[kpp
s]
Time [s]
Send 3Send 2Send 1Drp. 1Drp. 2Drp. 3Pkt. Rt.Recv.
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Answered Research Questions (III)
6 May the distributed nature of our approach causeadditional performance issues and how can these beaddressed?
→ Problem: Accumulating Data→ Hierarchical Event Patterns, Multi-tiered Setups
7 What is the most relevant performance limit of ourapproach in a non-distributed scenario and how can itbe addressed?
→ Sensor→ Example: Packet Capturing→ Various Improvements
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Answered Research Questions (III)
6 May the distributed nature of our approach causeadditional performance issues and how can these beaddressed?
→ Problem: Accumulating Data→ Hierarchical Event Patterns, Multi-tiered Setups
7 What is the most relevant performance limit of ourapproach in a non-distributed scenario and how can itbe addressed?
→ Sensor→ Example: Packet Capturing→ Various Improvements
0 200 400 600 800
1000 1200 1400
0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 0 2 4 6 8 10 12 14
[kpp
s]
# of
Rul
es
Time [s]
Cap. Rt. [kpps]Pkt. Rt. [kpps]
Drop Rt. [kpps]Min. # Rules
# Act. Rules
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Publications
Improving Network Traffic Acquisition and Processing with the Java Virtual Machine, R. Gad, M. Kappes,and I. Medina-Bulo, 20th IEEE ISCC 2015, in press
Monitoring Traffic in Computer Networks with Dynamic Distributed Remote Packet Capturing, R. Gad,M. Kappes, and I. Medina-Bulo, IEEE ICC 2015, in press
Analysis of the Feasibility to Combine CEP and EDA with Machine Learning using the Example ofNetwork Analysis and Surveillance, R. Gad, M. Kappes, and I. Medina-Bulo, JCIS – SISTEDES 2014
Bridging the Gap between Low-level Network Traffic Data Acquisition and Higher-level Frameworks, R.Gad, M. Kappes, and I. Medina-Bulo, IEEE COMPSACW 2014
Header Field Based Partitioning of Network Traffic for Distributed Packet Capturing and Processing, R.Gad, R. Mueller-Bady, M. Kappes, and I. Medina-Bulo, 28th IEEE AINA 2014
Employing the CEP Paradigm for Network Analysis and Surveillance, R. Gad, M. Kappes, J.Boubeta-Puig, and I. Medina-Bulo, AICT 2013
Leveraging EDA and CEP for Integrating Low-level Network Analysis Methods into Modern, DistributedIT Architectures, R. Gad, M. Kappes, J. Boubeta-Puig, and I. Medina-Bulo, JCIS – SISTEDES 2012
Hierarchical events for efficient distributed network analysis and surveillance, R. Gad, M. Kappes, J.Boubeta-Puig, and I. Medina-Bulo, WAS4FI 2012
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Open Source Software Contributions
Clojure and Java Packet Capturing Libraryhttps://github.com/ruedigergad/clj-net-pcap
Distributed Remote Packet Capturing (DRePCap)https://github.com/fg-netzwerksicherheit/drepcap
clj-jms-activemq-toolkithttps://github.com/fg-netzwerksicherheit/clj-jms-activemq-toolkit
drepcap-sensorhttps://github.com/fg-netzwerksicherheit/drepcap-sensor
drepcap-mergerhttps://github.com/fg-netzwerksicherheit/drepcap-merger
drepcap-frontendhttps://github.com/fg-netzwerksicherheit/drepcap-frontend
Patches for jNetPcap
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
Conclusion
EDA and CEP for Overarching NAaS
It works!
Improved the state of the art.
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems
Introduction Thesis Overview Two Highlights Summary and Conclusion
End
Thank you for your attention!
Questions?
Rudiger Gad
Rudiger Gad
Event-driven Principles and Complex Event Processing for Self-adaptive Network Analysis and Surveillance Systems