apache eagle in action
TRANSCRIPT
![Page 2: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/2.jpg)
Who is the Guy
2
Co-creator, Committer and PMC @ Apache [email protected]
Hao Chen / 陈浩Sr. Software Engineer @ eBay Cloud [email protected]
Speaker @ Hadoop Summit (SJC, SHA, BJ) ...http://people.apache.org/~hao
![Page 3: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/3.jpg)
Agenda
3
•About Eagle•Architecture•Ecosystem•Q & A
![Page 4: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/4.jpg)
What’s Apache Eagle
4
Apache Eagle is a distributed real-time monitoring and alerting engine for hadoop from eBay
Open sourced as Apache Incubator Project on Oct 26th 2015
Secure Hadoop in Realtime a data activity monitoring solution to instantly identify access to sensitive data, recognize attacks/ malicious activity and block access in real time.
See http://eagle.incubator.apache.org or http://goeagle.io
![Page 5: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/5.jpg)
Apache Eagle History
Donated to Apache Software Foundation (ASF) from eBay at Oct 26th, 2015
5
Dec 2013 Oct 23 2015 Oct 26 2015
Hadoop Eagle
Project Initiative
Apache Incubatoreagle.incubator.apache.org
Github Open Source github.com/apache/incubator-eagle
Hadoop EagleProduction Release
May 2014
![Page 6: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/6.jpg)
Why build Apache Eagle
6
Eagle was initialized by end of 2013 for hadoop ecosystem monitoring as any existing tool like zabbix, ganglia can not handle the huge volume of metrics/logs generated by hadoop system in eBay.
2013/201410,000 nodes150,000+ cores170 PB2000+ user
3000+ nodes10,000+ cores50+ PB2012
20111000+ nodes10,000+ cores10+ PB
100+ nodes1000 + cores1 PB2010
200950+ nodes
20071-10 nodes
Hadoop Data • Security• ActivityHadoop Platform • Heath• Availability• Performance
Hadoop @ eBay Inc
![Page 7: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/7.jpg)
Apache Eagle @ eBay
7
7 CLUSTERS7427 NODES160 PB DATA
10 B+ EVENTS / DAY500+ METRIC TYPES50,000+ JOBS / DAY50,000,000+ TASKS / DAY
MONITOR
PROCESS
![Page 8: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/8.jpg)
Agenda
8
•About Eagle•Architecture•Ecosystem•Q & A
![Page 9: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/9.jpg)
Apache Eagle Architecture Overview
9
Scalable Scales to monitor thousands of policies and billions of access events
Machine Learning Create dynamic user profiles based on user behavior
Real-time Generates alerts in real time and blocks users with malicious intent
ExtensibleEagle can be easily extended to monitor other data sources
![Page 10: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/10.jpg)
Apache Eagle Architecture Overview
10
STREAM PROCESSINGENGINE
Data
Col
lect
orKa
fka
HDFS, Audit, Security
METADATA MANAGER
DATA STO
RES
REMEDIATION ENGINE
Apache Ranger
MACHINE LEARNING MODULE
Custom module
Actionable Alerts
Activities
Actionable Alerts
PolicyThresholdsUser properties
ML Thresholds
Real Time Alert Dashboard
Security Analyst
Admin Console
Security Engineer
Insights
Metadata Management
MACHINE LEARNING TRAINING MODULE
Policy Engine
![Page 11: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/11.jpg)
Apache Eagle Architecture Features
11
•Real-time Data Collection•Distributed Policy Engine•Stream Processing DSL•Scalable Data Storage & Query•Machine Learning Intergration
NOTE {NAME}-{NUMBER} like HDFS-6914 means open source project ticket id contributed by us
![Page 12: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/12.jpg)
Apache Eagle - Data Collection
12
Decoupling with Message Bus• Apache Kafka: high-throughput distributed
messaging• Partition: balance between logic and throughput
Cross-Platform Integration• Community Kafka Client (18+)
• Python/Go/C/C++/JAVA .. • Enhanced Log4j-kafka
• KAFKA-2041: Extensible Partition Key• KAFKA-2077: Advanced Topic Selector
![Page 13: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/13.jpg)
Apache Eagle - Data Collection
13
Availability: Filebeat + Logstash
Logstash-1
Logstash-2Logstash-
…Logstash-
Ligh-weight collector (golang) with daemon Logstash instances cluster
Shuffle Grouping KafkaField Grouping
Distributed Message Bus
Resource consumption balanceMessage throughput balance ( LOGSTASH-179)
![Page 14: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/14.jpg)
Storm Spout: Distributed crawling for hadoop job, node jmx and service logs, etc.Zookeeper: Centralized state management and distributed locking
Apache Eagle - Data Collection
14
Scalability: Distributed Real-time Ingestion
Zookeeper
METRIC
Centralized State Management
JOB EVENT LOG
![Page 15: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/15.jpg)
Apache Eagle - Distributed Real-time Policy Engine
15
METADATA MANAGER
Distributed Streaming Cluster Environment
AlertExecutor_{1}
AlertExecutor_{2}
…
AlertExecutor_{N}
Real Time Alerts
Alerts
Policy Management
Policy
Dynamical Policy Deployment
Real-time Event Stream
Stream_{1}
Stream_{*}
Dynamical Stream Schema
Stream Processing
Highlights
• Real-time• Usability• Scalability• Extensibility• Metadata-driven
![Page 16: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/16.jpg)
Apache Eagle - Distributed Real-time Policy Engine
16
METADATA MANAGER
Real Time Alerts
Alerts
Policy Management
Policy
Event Stream(Kafka)
Dynamical Stream SchemaDynamical Policy Deployment
Real-time
• Kafka-based Distributed Message Bus (Extensible)
• Storm-based Real-time Execution Environment (Extensible)
• Stream events are processed and alerts are evaluated during streaming
![Page 17: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/17.jpg)
Apache Eagle - Distributed Real-time Policy Engine
17
METADATA MANAGER
Distributed Streaming Cluster Environment
Real Time Alerts
Alerts
Policy Management
Policy
Dynamical Policy Deployment
Usability
• Powerful SQL-Like CEP CQL for Policy Definition
• Dynamical Poilcy Metadata Lifecycle Management (Deployment/Update)
• Easy-to-use Policy management and Alert analytics UI
from metricStream[(name == 'ReplLag') and (value > 1000)] select * insert into
outputStream;
![Page 18: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/18.jpg)
Apache Eagle - Distributed Real-time Policy Engine
18
Full-function Streaming CEP CQL: Siddhi on Storm by default
hdfsAuditLogEventStream[(src == '/tmp/private')]#window.externalTime(timestamp,10 min) select user, count(timestamp) as aggValue group by user having aggValue >= 5 insert into outputStream;
• Filter• Join• Aggregation: Avg, Sum , Min, Max, etc• Group by• Having• Stream handlers for window: TimeWindow, Batch Window, Length Window • Conditions and Expressions: and, or, not, ==,!=, >=, >, <=, <, and arithmetic operations• Pattern processing• Sequence processing• Event Tables: intergrate historical data in realtime processing• SQL-Like Query: Query, Stream Definition and Query Plan compilation
![Page 19: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/19.jpg)
Apache Eagle - Distributed Real-time Policy Engine
19
Distributed Streaming Cluster Environment
AlertExecutor_{1}
AlertExecutor_{2}
…
AlertExecutor_{N}
Stream_{1}
Stream_{*}
Stream Processing
Scalability: dynamic policy partition by {event} * {policy}
• N Users with 3 partitions, M policies with 2 partitions, then 3*2 physical tasks• Physical partition + policy-level partition
![Page 20: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/20.jpg)
Apache Eagle - Distributed Real-time Policy Engine
20
Distributed Streaming Partition Problem
https://en.wikipedia.org/wiki/Partition_problem
S = {3,1,1,2,2,1,1}S1 = {1,1,1,1,1}S2 = {2,2}S3 = {3}
![Page 21: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/21.jpg)
Apache Eagle - Distributed Real-time Policy Engine
21
Distributed Streaming Partition Strategy
groupBy[ GreedyStrategy ]((_.key1,_.key2 ))
HBase
Key Distribution Statistics (Online/Offline)
Realtime Partition Strategy
Key Statistics CacheAsync
Strategy• Greedy (Online/Offline)• PoTC• PKG• Hashing
![Page 22: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/22.jpg)
Apache Eagle - Distributed Real-time Policy Engine
22
Distributed Real-time Policy Engine
Siddhi CEP Policy
Evaluator
Machine Learning Policy
Evaluator
Extensibility
• Support WSO2 Siddhi CEP as first class• Extensible policy engine implementation
• Extensible policy lifecycle management
Extensible Policy Evaluator
public interface PolicyEvaluatorServiceProvider {public String getPolicyType(); // literal string to identify one type of policypublic Class getPolicyEvaluator(); // get policy evaluator implementationpublic List getBindingModules(); // policy text with json format to object mapping
}
public interface PolicyEvaluator {public void evaluate(ValuesArray input) throws Exception; // evaluate
input eventpublic void onPolicyUpdate(AlertDefinitionAPIEntity newAlertDef);//
policy updatepublic void onPolicyDelete(); // invoked when policy is deleted
}
METADATA MANAGER
Policy/Metadata
![Page 23: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/23.jpg)
Apache Eagle - Distributed Real-time Policy Engine
23
Metadata-Driven
• Stream Schema: AlertStreamSchemaEntity• Policy Definition: AlertDefinitionAPIEntity• Central metadata management• Dynamic metadata deployment
@Table("alertdef")@ColumnFamily("f")@Prefix("alertdef")@Service(AlertConstants.ALERT_DEFINITION_SERVICE_ENDPOINT_NAME)@JsonIgnoreProperties(ignoreUnknown = true)@TimeSeries(false)@Tags({"site", "dataSource", "alertExecutorId", "policyId", "policyType"})@Indexes({ @Index(name="Index_1_alertExecutorId", columns = { "alertExecutorID" }, unique = true),})public class AlertDefinitionAPIEntity extends TaggedLogAPIEntity{ @Column("a") private String desc; @Column("b") private String policyDef; @Column("c") private String dedupeDef;
METADATA MANAGER
Distributed Real-time Policy Engine
Dynamic Metadata Loading
![Page 24: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/24.jpg)
.flatMap(AuditLogTransformer) .groupBy(_.user) .flatMap(UserProfileAggregator);
Apache Eagle - Fluent Stream Processing DSL
24
env.fromKafka (KafkaConfig)
.alert.persistAndEmail
val env = ExecutionEnvironment.getStorm()
env.execute();
Distributed Streaming Cluster Environment
AlertExecutor_{1}
AlertExecutor_{2}
…
AlertExecutor_{N}
Alerts
Real-time Event Stream
Stream_{1}
Stream_{*}
Stream Processing
env.execute()
![Page 25: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/25.jpg)
Apache Eagle - Fluent Stream Processing DSL
25
• Physical execution platform independent• Easily assemble data transformation, filtering,
join and alerting DAG in fluent way• DAG rewrite and optimization
• StreamUnionExpansion• StreamGroupbyExpansion• StreamNameExpansion• StreamAlertExpansion• StreamParallelismConfigExpansion
trait StreamProducer{ filter flatMap map{1,2,3,4} groupBy streamUnion // stream join is hard, not implemented for storm alertWithConsumer}
StormExecutionEnvironment env = ExecutionEnvironmentFactory.getStorm(config); env.newSource(new KafkaSourcedSpoutProvider().getSpout(config)).renameOutputFields(1) .flatMap(new AuditLogTransformer()) .groupBy(0) .flatMap(new UserProfileAggregatorExecutor()); .alertWithConsumer(“userActivity“,”userProfileExecutor“)env.execute();
Optimizer
1. Development 2. Optimization 3. Compile to native app
![Page 26: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/26.jpg)
Apache Eagle - Scalable Data Storage and Query
26
• Entity Metadata on large-scale NoSQL storage like HBase
• Full-function SQL-Like REST Query • Optimized rowkey design for time-series
monitoring data• HBase Coprocessor• Secondary Index
@Table("alertdef")@ColumnFamily("f")@Prefix("alertdef")@Service(AlertConstants.ALERT_DEFINITION_SERVICE_ENDPOINT_NAME)@JsonIgnoreProperties(ignoreUnknown = true)@TimeSeries(false)@Tags({"site", "dataSource", "alertExecutorId", "policyId", "policyType"})@Indexes({ @Index(name="Index_1_alertExecutorId", columns = { "alertExecutorID" }, unique = true),})public class AlertDefinitionAPIEntity extends TaggedLogAPIEntity{ @Column("a") private String desc; @Column("b") private String policyDef; @Column("c") private String dedupeDef;
query=AlertDefinitionService[@dataSource="hiveQueryLog"]{@policyDef}
![Page 27: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/27.jpg)
Uniform rowkey design
• Metric
• Entity
• Log
Rowkey ::= Prefix | Partition Keys | timestamp | tagName | tagValue | …
Rowkey ::= Metric Name | Partition Keys | timestamp | tagName | tagValue | …
Rowkey ::= Default Prefix | Partition Keys | timestamp | tagName | tagValue | …
Rowkey ::= Log Type | Partition Keys | timestamp | tagName | tagValue | …Rowvalue ::= Log Content
27
Apache Eagle – Uniform HBase Rowkey Design
![Page 28: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/28.jpg)
Apache Eagle - Machine Learning Intergration
28
![Page 29: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/29.jpg)
29
User Activity Profiling
Offline: Determine bandwidth from training dataset the kernel density function parameters (KDE)Online: If a test data point lies outside the trained bandwidth, it is anomaly (Policy)
PCs(Principle Components) in EVD (Eigenvalue Value Decomposition)Kernel Density Function
Apache Eagle – User/System Activity Profiling
![Page 30: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/30.jpg)
30
Anomaly Metric Predictive Detection
Offline: Analyzing and combining 500+ metrics together for causal anomaly detections (IG -> PCA -> GMM -> MCC)Online: Predictively alert for anomaly metrics
Normal (Green) and Abnormal (Red) Data and Probability Distribution and Threshold Selection
PCA (Principal Component Analysis)
Apache Eagle - Anomaly Metric Predictive Detection
Anomaly Metric Predictive Detection Case Study
![Page 31: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/31.jpg)
Agenda
•About Apache Eagle•Architecture•Ecosystem•Q & A
31
![Page 32: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/32.jpg)
32
Apps Security Hadoop Cloud Database
Interface Web Portal REST Services Analytics Visualization
Integration Ambari Docker Ranger Dataguise
Eagle FrameworkDistributed real-time framework for efficiently developing highly scalable monitoring applications
Eagle AppsSecuriy / Hadoop / Cloud / Database
Eagle InterfaceREST Service / Management UI / Customizable Analytics Visualization
Eagle IntegrationAmbari / Docker / Ranger / Dataguise Open SourceCommunity-driven and Cross-community cooperation
Eagle Framework
Apache Eagle Ecosystem
![Page 33: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/33.jpg)
33
Apache Eagle Ecosystem - Security
How to Secure Hadoop in Realtime?
• Apache Eagle• Apache Ranger• Apache Knox• Dataguise
![Page 34: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/34.jpg)
34
Apache Eagle Ecosystem - Hadoop
Eagle in Apache Amabri: natively be part of hadoop ecosystem
![Page 35: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/35.jpg)
35
Apache Eagle Ecosystem - Docker
Eagle in Docker: natively fly on Cloud/Container
STORMKAFKAZOOKEEPERHBASEHADOOP…
Powered of git clone apache/incubator-eagle eagle-docker
15 + 1 = 1docker pull apacheeagle
![Page 36: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/36.jpg)
36
Apache Eagle Ecosystem - Open Source
If you want to go fast, go alone.If you want to go far, go together.
-- African Proverb
![Page 37: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/37.jpg)
Learn more about Apache Eagle
37
• EAGLE: USER PROFILE-BASED ANOMALY DETECTION IN HADOOP CLUSTER (IEEE)
• EAGLE: DISTRIBUTED REALTIME MONITORING FRAMEWORK FOR HADOOP CLUSTER
![Page 38: Apache Eagle in Action](https://reader035.vdocuments.site/reader035/viewer/2022081517/589b9f4b1a28abd63e8b5d79/html5/thumbnails/38.jpg)
Q & A
apache/incubator-eagle
@TheApacheEagle
@ApacheEagle
http://eagle.incubator.apache.org
The slide is licensed under Creative Commons Attribution 4.0 International license.