let's reconsider about collecting logs. plus, visiting elastic@moutain view!

Let’s reconsider about collecting logs.Plus, visiting elastic@Mountain View!

Shin Tanimoto Acroquest Technology Co., LTD.

Copyright © Acroquest Technology Co., Ltd. All rights reserved.

Who am I?

2

• 谷本心 (Shin Tanimoto)

- Acroquest Technology Co., LTD.(Sales partner of elastic)

- Java Troubleshooter- Board member of JJUG

(Japan Java User Group)- Twitter : @cero_t- Facebook : shin.tainmoto

Copyright © Acroquest Technology Co., Ltd. All rights reserved.Copyright © Acroquest Technology Co., Ltd. All rights reserved.

Quiz🙋

3


What is the originof the word “log”?

4


1. Ancient Greece people record the “date” using

branches of the tree.

6


2. In medieval Europe, people measured “speed” of ship with log (round wood).

7


3. In the early 20th century United States, engineers used a

logarithm table for “usage history” of computers.

8


1. Ancient Greece people’s “date” record.

2. Medieval Europe sailors’ “speed” record.

3. American engineers’ “usage” record.

9


1.

2. Medieval Europe sailors’ “speed” record.

3.

10


Common sense:Log is important

11


True common sense:Watching log is painful!

12


Then log should bewatched and processedby machine (ordinary)

13


#1 Ordinal Log Processing


1. Ordinal Log Processing

ELK stack

15

send logs

search



Access counts (upper) / response time (lower)

16



Access counts (upper) / response time (lower)

17

10/sec

100/sec

30sec

20sec

10sec



Huge performance issue

18



Huge performance issue

19

3000sec

2000sec

1000sec



Slow query log of MySQL

20

same shape!



Slow query log of MySQL

21

2000sec

1000sec

same shape! same scale!



But where do these shapes come?

22



23

But where do these shapes come?1. Lock tables?2. Up to maximum size of connection pool?3. CPU bottle neck?4. Disk I/O bottle neck?



24

Confirm the stored procedure in detail1. Found 100,000 times of insert into “temporary table” query

– (even in the search function … )– causing high CPU and Disk I/O usage

2. Optimized the stored procedure removing wasting process– Only a drop in the bucket 😩

3. Modify the create temporary table state in the stored procedure to create that temporary table “on memory”– with memory tunings ( tmp_table_size etc. )– resulted in …



Performance issue was resolved!

25



Performance issue was resolved!

26

500sec

100sec

Never mind, some heavy batch



Disk I/O improved!!!

27

Disk I/O on MySQL server

before ← →after



I/O wait had gone!

28

before ← →after

CPU usage on MySQL server


#2 Reconsider Log

Processing


Watching logs to detect errors is a responsibility of

developers, isn’t it?


Watching logs is important but painful


Let’s think about painless log

processing system


#2 Reconsider Log Processing

Logs can be used in various waysVisualizing - as chartWatching - and notifying by e-mailViewing - by human’s eyesKeeping - backup just in case

33



Logs can be used in various purposeVisualizing - To find “unknown” issuesWatching - To find “known” issuesViewing - To find the cause of issuesKeeping - To use as necessary

34



Logs retention period are also variousVisualizing - last 2 or 4 weeksWatching - last 24 hoursViewing - last 2 or 4 weeksKeeping - entire period

35



Tools for processing logs are differentVisualizing - ElasticsearchWatching - Zabbix or some custom batchViewing - Text editorKeeping - File server

36


Log is not necessarily files.


Log can be regarded as events.


Log streaming hub



Log Streaming Hub

40

Application

AgentStreaming

Hub

Viewer

Watcher

Visualizer

Storage

Application

Agent



Ordinal case

41

Application

fluentd

Text Editor

Zabbix

Elasticsearch+ Kibana

NAS

Application

fluentd



Using fluentd

42

Application

fluentd fluentdZabbix


Application

fluentd

Text Editor

NAS


#2 Processing Logs

Using Redis?

43

Application

Logstash RedisZabbix


Application

Logstash

Text Editor

NAS



An example on AWS

44

Application

Kinesis Agent

Kinesis Cloudwatch Logs


Application

Kinesis Agent

S3

Lambda


#2 Processing Logs

An example on AWS

45

Application

awslogsCloudwatch

Logs


Application

awslogsS3

Lambda


#2 Processing Logs

An example on AWS

46

Application

logstash S3


Application

logstash

Cloudwatch Logs

Lambda


#2 Processing Logs

Anyway…

47

Application

AgentStreaming

Hub

Viewer

Watcher

Visualizer

Storage

Application

Agent


すんなり入る話ですよね？


By the way


I had visited elastic@Mountain View!!


写真はブログでhttp://acro-engineer.hatenablog.com/

entry/2015/11/08/150942

http://acro-engineer.hatenablog.com/entry/2015/11/08/150942


elastic stack .Next


#3 elastic stack .Next

ElasticsearchTask management APIReindex API

https://www.elastic.co/elasticon/2015/sf/whats-next-for-elasticsearch-2x-and-beyond

LogstashClusteringPersistent

https://www.elastic.co/guide/en/logstash-roadmap/current/index.html

53

https://www.elastic.co/elasticon/2015/sf/whats-next-for-elasticsearch-2x-and-beyond

https://www.elastic.co/guide/en/logstash-roadmap/current/index.html



KibanaCustom Apps / plugins

https://www.elastic.co/elasticon/2015/sf/whats-cookin-in-kibana-4

BeatsPacketbeatFilebeatTopbeat

https://www.elastic.co/products/beats

54

https://www.elastic.co/elasticon/2015/sf/whats-cookin-in-kibana-4

https://www.elastic.co/products/beats



Commercial pluginCross-stack monitoring / managementCross-stack securityPDF reportingOrchestration / Automation

55


Using elastic stack .Next



Using next ELK stack + AWS

57

Application

Filebeat Topbeat

Logstash


Application

Filebeat Topbeat

S3

Cloudwatch Logs


Beatsシリーズ調べなくちゃって気に

なりました