in-kernel analytics and tracing with ebpf for openstack clouds
TRANSCRIPT
In-kernel Analytics and Tracing with eBPF for
OpenStack CloudsOctober 2016
Brenden BlancoPLUMgrid
Ali KhayamPLUMgrid
Thank You to Sponsoring Members
2
IO Visor Project, What is in it?
• A set of development tools, IO Visor Dev Tools
• A set of IO Visor Tools for management and operations of the IO Visor Engine
• A set of Applications, Tools and open IO Modules build on top of the IO Visor framework
• A set of possible use cases & applications like Networking, Security, Tracing & others
3
The promise of Microservices: Better cloud app lifecycle …… but what about security?
4
Shared kernel Larger attack surface?
Self service Developer = Security Expert?
Shared Infrastructure Insider threats?
Fast Development & Iteration Compromised zero trust?
Where should microservices security be implemented?
All layers…. but from the app cloud provider’s perspective:best to trust what you build/operate/control
=> “Security-as-a-Service” in the cloud infrastructure
InfrastructureOperator
ApplicationDeveloper
An ideal Security-as-a-Service offeringTransparent: Application shouldn’t be aware of this layer
No new software installation/configuration
Generically applicable: Should be able to characterize microservice security profiles for diverse applications, without having visibility into service behavior
Efficient: No compromises on performance or scalability
What features can characterize a Microservice Security Profile?
APIAPI call, payload len
Trafficbytes tx/rx, packets rx/rx
Disk I/ODisk I/O rx/tx
Tenants# of active tenants
…. how to get these features without compromising transparency and efficiency
How to extract features for Microservice Security Profiles?Objectives: Transparency, Seamlessness,
Efficiency
IO Visor instrumented infra to extract features for service security profiles:
▪already present in Linux kernels▪capture API calls and resource usage▪system-call level insight
▪ real-time monitoring▪without efficiency degradation
8
Automation
Developers
IOVisor framework
Advanced MonitoringSecurityAutomation / OperationsMachine Learning
Infrastructure
Monitor
Ops/Automation
Maintain
Plugging features into an ML model to learn Microservice security profiles
9
Com
pute
Nod
e
Use
r Spa
ceK
erne
l Spa
ce
API / Traffic Data Disk/Memory Data
Microservice CollectorMachine
Learning
API Traffic(Ingress / Egress)
Microservice Security Profiles
IO Visor Code Snippet (Userspace)
IO Visor Code Snippet (Kernel)
www.iovisor.org
Preliminary Evaluation1) OpenStack Controller Services as
Microservices
12
OpenStack Controller Services as MicroservicesIO Visor instrumentation is used to build security profiles of all controller services
nova, neutron, keystone, cinder, etc.
API calls learned as they arrive on the services’ veth interfaceno pre-training of API calls
IO Visor hooks to monitor vfs_{read/write} accesses from each serviceseparated based on PIDs for each container
ML algorithm builds security profiles based on initial (training) datathen security profile deviations are used for attack detection on run-time data
Attack: Bruteforce password cracking on keystone
Lots of Background (benign) Traffic:Continuous CRUD APIs from a real-world app cloud use caseAll API calls (incl. service-to-service) must get auth_token from keystone
first
Attack Traffic:2-4 password attempts per secondAttack continued for a sustained period of time
Results of brute-force password attack on keystone
Attack Detection Rate False Positive Rate
97% 0%
• Results obtained from an ROC curve by tuning the detection threshold
• API and Traffic features are the main contributors to these results
Preliminary Evaluation2) Database container using MySQL
16
MySQL Microservice instrumentation
MySQL Docker image (MySQL version 5.7, docker 1.12 )
SQL queries (TCP packets) intercepted by IOVisor hooks on veth pairs
handshakes, teardown and acks ignored
IOVisor hooks for vfs_{read/write} for queries into a large DB (180Mb)
separated on PID and TID for docker
17
Attack: First order SQL injection
Benign traffic consisted of Simulated SQL queriesGenerated randomly and continuously
Attack results in extracting large segments of the DBSegment size varyingIn parallel to benign traffic on the microservice
18
Results of brute-force password attack on keystone
Attack Detection Rate False Positive Rate
93.5% 3.5%
• Results obtained from an ROC curve by tuning the detection threshold
• Correlating Traffic and disk access was essential for detection
Dashboard
Conclusion:Meeting the requirements of an ideal Security-as-a-Service offering
21
Transparency
Application shouldn’t be aware of this layer
IO Visor works on eBPF constructs that are present in >4.x upstream kernels
IO Visor instrumentation runs in kernel and is not visible to the developer
The only non-standard dependency is github.com/iovisor/bcc python library
Generic Applicability
Should be able to characterize microservice security profiles for diverse applications,
without having visibility into service behavior
Trained/Tested on SQL
Trained/Tested on OpenStack services
Future Work:
Train/Test for DNS attacks
Train/Test for ransomware attacks
EfficiencyNo compromises on performance or scalability
eBPF counting is done inside the kernel with little or no overhead
Main overhead is kernel to userspace interaction
Data polled by userspace every 1 minute
All data structures are reset after polling; data cannot grow
indefinitely
Data is exported by the userspace application to a collector node
Machine learning and classification is applied on the collector node
i.e. no impact to performance on computes
EfficiencyNo compromises on performance or scalabilityData structures have low overhead:vfs_read (BFP_HASH):
size at time ti = Ni x 3, where:Ni = # of read process at ti
the map has: {key: pid, value1: # of reads, value2: aggregate size of all reads
vfs_write (BFP_HASH): has the the same structure as vfs_read
traffic (BFP_HASH):size at time ti = Fi x 7, where:
Fi = # of active TCP flows at ti
the map’s key is a 5-tuple flow id, and values are the same as vfs_{read/write}
http_traffic (BPF_HISTOGRAM):size at time ti = Si x LSi x 7, where:
key is a 5-tuple flow id of http packetsSi = # of active HTTP session at ti
LSi = # of HTTP packets with unique lengths received on session S i
How to Contribute
github.com/akhayam/conmon (this presentation)
www.iovisor.org
github.com/iovisor
#iovisor at irc.oftc.net
lists.iovisor.org/mailman/listinfo/iovisor-dev
26
Questions?