美团点评技术沙龙08 - 分布式监控系统实践
Post on 16-Apr-2017
376 views
TRANSCRIPT
![Page 1: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/1.jpg)
CAT (Central Application
Tracking)
1
![Page 2: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/2.jpg)
•
• 2010
• slb
2
![Page 3: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/3.jpg)
• CAT
• CAT
•
3
![Page 4: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/4.jpg)
CAT
• CAT(Central Application Tracking) Java
• CAT
4
![Page 5: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/5.jpg)
• 1
• 2
•
5
![Page 6: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/6.jpg)
CAT Logview•
• ID IP ID •
6
![Page 7: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/7.jpg)
Logview
7
![Page 8: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/8.jpg)
Logview
8
![Page 9: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/9.jpg)
APMTransac'on
EventProblem Hearbeat JVM Memory Thread Matrix Cross SOA RPC Cache
Dependency … …
9
![Page 10: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/10.jpg)
• CAT
• CAT
•
10
![Page 11: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/11.jpg)
CAT
•
•
•
11
![Page 12: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/12.jpg)
A
B
C
TCP
Logview Report
Consumers Consoles
12
![Page 13: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/13.jpg)
HttpThread
Call
SQL
Cach
......
ThreadContext
Call
SQL
Cache
.....
Sender
threads
Message Queue
SenderSender
ServerServer
Server
13
![Page 14: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/14.jpg)
•
•
• CPU
• 2%
•
• netty
14
![Page 15: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/15.jpg)
-IO• java message tree id java
MappedByteBuffer
•
• cpu io
15
![Page 16: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/16.jpg)
-Memory
• MessageTreemessageTree
•
16
![Page 17: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/17.jpg)
Receiver
threads
File A
B
C
Receiver
Receiver
threads
Analyzer
Analyzer
Analyzer
17
![Page 18: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/18.jpg)
Analyzer
Receiver
Receiver
ReceiverAnalyzer2
Analyzer3
Analyzer1
Analyzer1domain hash
18
![Page 19: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/19.jpg)
•
•
• CPU
•
•
•
19
![Page 20: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/20.jpg)
•
•
20
![Page 21: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/21.jpg)
url/sql
Exception
Cpu,IO
Transaction
Event
Heartbeat
Metric
21
![Page 22: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/22.jpg)
KeyValue
•
•
•
22
![Page 23: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/23.jpg)
• Transaction
• Event
• Problem
• Heartbeat
• ……
23
![Page 24: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/24.jpg)
•
•
•
• XML, JSON, Binary...
•
24
![Page 25: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/25.jpg)
25
![Page 26: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/26.jpg)
26
![Page 27: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/27.jpg)
cpu
27
![Page 28: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/28.jpg)
•
•
28
![Page 29: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/29.jpg)
• ID ShopWeb-0a010680-375030-2
•
• /2012/10/13/14/ShopSerivce-ShopWeb-10.1.6.1
• /2012/10/13/14/ShopService-ShopWeb-10.1.6.2
• 375030 => 2012-10-13 14:00:00
• ShopService => domain
• 10.1.6.1/2 => IP
• 0a010680 => 10.1.6.128 ID
29
![Page 30: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/30.jpg)
• 1 transaction report event report name
• 2 swap off
• 3 numactl --interleave=all
• 4 jvm
30
![Page 31: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/31.jpg)
• CAT
• CAT
•
31
![Page 32: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/32.jpg)
CAT• 2011-11
• 2012-3 MVP
• 2012-6
• 2012-12 150+ 500+
• 2013-12 400+ 1500+
• 2014-12 800+ 3000+
• 2015-9 1500+ 7000+
• 2016-6 2600+ 12000+
32
![Page 33: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/33.jpg)
MVP
• Demo 1
• MVP 3
•
33
![Page 34: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/34.jpg)
•
• vip
34
![Page 35: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/35.jpg)
•
•
• ***
35
![Page 36: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/36.jpg)
•
•
• mat
36
![Page 37: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/37.jpg)
code
•
•
•
37
![Page 38: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/38.jpg)
•
• sql rpc web
•
38
![Page 39: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/39.jpg)
• jetty server
• hdfs
• mysql
39
![Page 40: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/40.jpg)
•
• 2-3
•
•
40
![Page 41: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/41.jpg)
• scope
•
•
41
![Page 44: 美团点评技术沙龙08 - 分布式监控系统实践](https://reader034.vdocuments.site/reader034/viewer/2022050712/5873a65f1a28aba3548b4f4d/html5/thumbnails/44.jpg)
QA
• Thanks
44