kafka eagle architecturepdf.kafka-eagle.org/file/kafka_eagle_architecture.pdf · 2019. 8. 27. ·...

KAFKA EAGLEA EASY AND HIGH-PERFORMANCE MONITORING SYSTEM

AUTHOR : SMARTLOLI UPDATE : 2019.07.14

01 Part IThe Background of Kafka Eagle

02 Part IIDesign Architecture of Kafka Eagle

03 Part IIIThe Advantages of Kafka Eagle

04 Part IVFollow-up Planning of Kafka Eagle

MENU

01Part I

The Background of Kafka Eagle

Background

01

02

03

04Convenience

It is difficult to configure and use the existing monitoring system.

Apache Kafka

Apache Kafka does not officially provide monitoring systems or pages.

Open Source

Open source Kafka monitoring system has too few functions or suspension of maintenance.

IM Support

Some monitoringsystem can not satisfy integration with existing IM, such as wechat ordingding.

02Part II

Design Architecture of Kafka Eagle

Architecture

02

01

0305

06

Visualization1. Topic list, rate, consumer2. Cluster heathy3. TBD

ACLs1. Topic create,detele etc2. User manage

Alert1. E-Mail2. WeChat3. DingDing

Collector

1. Zookeeper2. Kafka jmx & external topic3. Kafka api (2.x+)

Storage

1. MySQL2. SQLite

Metrics

1. Consumer application2. Cluster heathy

KafkaEagle

04

03Part III

The Advantages of Kafka Eagle

Advantages-Convenience

https://www.kafka-eagle.org/

Download

https://docs.kafka-eagle.org/

Install

ke.sh start

QuickStart

http://ip:port/ke

Visit

Step 1 Step 3Step 2 Step 4

Advantages-Extensive

DashboardKafka Eagle

Topic

Cluster

Alarm

System

The kafka eagle system has favorableuser interface, powerfulfunction modules.

Consumers Metrics

Dashboard

Show the scale of clusters, themes, consumers, etc.

Show kafka cluster survival nodes.

Topic

1. Create topic.

2. List topic.

3. SQL query.

4. Mock data.

5. Config topic.

Consumers

Display consumer group application details.

Display consumer active topics details.

Cluster

1. Show details.

2. Multi-cluster.

3. ZK client.

Metrics

1. Broker mbean.

2. Kafka metrics.

3. ZK metrics.

Alarm

1. Add lag alarm.

2. Modify lag.

3. Add cluster.

4. Modify cluster.

System

1. Add user.

2. Assign role.

3. Manage res.

04Part IV

Follow-up Planning of Kafka Eagle

Planning

Dimension 2Dimension 1 Dimension 4Dimension 3

Kafka Eagle follow-up plan is divided into four dimensions.

QAP

Continuous optimization of system modules of Kafka Eagle.

DSR

Collect Github issues problems and fix them.

PPL

Developing new functions of Kafka Eagle system for actual needs.

PHB

Organize and file the user manual of Kafka Eagle system.

About Me

Kafka Eagle

Blog

PublishBooks

ME

Hadoop: https://item.jd.com/12455361.htmlKafka: https://item.jd.com/12371763.html

https://www.cnblogs.com/smartloli/

http://www.kafka-eagle.org/https://github.com/smartloli/kafka-eagle/

THANK YOU FOR WATCHINGhttp://www.kafka-eagle.org/

2 0 1 9

AUTHOR : SMARTLOLI UPDATE : 2019.07.14