let's reconsider about collecting logs. plus, visiting elastic@moutain view!

Let’s reconsider about collecting logs.Plus, visiting elastic@Mountain View!

Shin Tanimoto Acroquest Technology Co., LTD.

Who am I?

• 谷本心 (Shin Tanimoto)

- Acroquest Technology Co., LTD.(Sales partner of elastic)

- Java Troubleshooter- Board member of JJUG

(Japan Java User Group)- Twitter : @cero_t- Facebook : shin.tainmoto

Quiz🙋

What is the originof the word “log”?

1. Ancient Greece people record the “date” using

branches of the tree.

2. In medieval Europe, people measured “speed” of ship with log (round wood).

3. In the early 20th century United States, engineers used a

logarithm table for “usage history” of computers.

1. Ancient Greece people’s “date” record.

2. Medieval Europe sailors’ “speed” record.

3. American engineers’ “usage” record.

2. Medieval Europe sailors’ “speed” record.

Common sense:Log is important

True common sense:Watching log is painful!

Then log should bewatched and processedby machine (ordinary)

#1 Ordinal Log Processing

1. Ordinal Log Processing

ELK stack

send logs

search

Access counts (upper) / response time (lower)

10/sec

100/sec

Huge performance issue

3000sec

2000sec

1000sec

Slow query log of MySQL

same shape!

Slow query log of MySQL

2000sec

1000sec

same shape! same scale!

But where do these shapes come?

But where do these shapes come?1. Lock tables?2. Up to maximum size of connection pool?3. CPU bottle neck?4. Disk I/O bottle neck?

Confirm the stored procedure in detail1. Found 100,000 times of insert into “temporary table” query

– (even in the search function … )– causing high CPU and Disk I/O usage

2. Optimized the stored procedure removing wasting process– Only a drop in the bucket 😩

3. Modify the create temporary table state in the stored procedure to create that temporary table “on memory”– with memory tunings ( tmp_table_size etc. )– resulted in …

Performance issue was resolved!

500sec

100sec

Never mind, some heavy batch

Disk I/O improved!!!

Disk I/O on MySQL server

before ← →after

I/O wait had gone!

before ← →after

CPU usage on MySQL server

#2 Reconsider Log

Processing

Watching logs to detect errors is a responsibility of

developers, isn’t it?

Watching logs is important but painful

Let’s think about painless log

processing system

#2 Reconsider Log Processing

Logs can be used in various waysVisualizing - as chartWatching - and notifying by e-mailViewing - by human’s eyesKeeping - backup just in case

Logs can be used in various purposeVisualizing - To find “unknown” issuesWatching - To find “known” issuesViewing - To find the cause of issuesKeeping - To use as necessary

Logs retention period are also variousVisualizing - last 2 or 4 weeksWatching - last 24 hoursViewing - last 2 or 4 weeksKeeping - entire period

Tools for processing logs are differentVisualizing - ElasticsearchWatching - Zabbix or some custom batchViewing - Text editorKeeping - File server

Log is not necessarily files.

Log can be regarded as events.

Log streaming hub

Log Streaming Hub

Application

AgentStreaming

Viewer

Watcher

Visualizer

Storage

Application

Ordinal case

Application

fluentd

Text Editor

Zabbix

Elasticsearch+ Kibana

Application

fluentd

Using fluentd

Application

fluentd fluentdZabbix

Application

fluentd

Text Editor

#2 Processing Logs

Using Redis?

Application

Logstash RedisZabbix

Application

Logstash

Text Editor

An example on AWS

Application

Kinesis Agent

Kinesis Cloudwatch Logs

Application

Kinesis Agent

Lambda

#2 Processing Logs

An example on AWS

Application

awslogsCloudwatch

Application

awslogsS3

Lambda

#2 Processing Logs

An example on AWS

Application

logstash S3

Application

logstash

Cloudwatch Logs

Lambda

#2 Processing Logs

Anyway…

Application

AgentStreaming

Viewer

Watcher

Visualizer

Storage

Application

すんなり入る話ですよね？

By the way

I had visited elastic@Mountain View!!

写真はブログでhttp://acro-engineer.hatenablog.com/

entry/2015/11/08/150942

elastic stack .Next

#3 elastic stack .Next

ElasticsearchTask management APIReindex API

https://www.elastic.co/elasticon/2015/sf/whats-next-for-elasticsearch-2x-and-beyond

LogstashClusteringPersistent

https://www.elastic.co/guide/en/logstash-roadmap/current/index.html

KibanaCustom Apps / plugins

https://www.elastic.co/elasticon/2015/sf/whats-cookin-in-kibana-4

BeatsPacketbeatFilebeatTopbeat

https://www.elastic.co/products/beats

Commercial pluginCross-stack monitoring / managementCross-stack securityPDF reportingOrchestration / Automation

Using elastic stack .Next

Using next ELK stack + AWS

Application

Filebeat Topbeat

Logstash

Application

Filebeat Topbeat

Cloudwatch Logs

Beatsシリーズ調べなくちゃって気に

なりました

Enjoy processing logs using ELK!

let's reconsider about collecting logs. plus, visiting elastic@moutain view!

Technology

moutain bike in valled'aosta - 61 itinerari ai piedi dei...

moutain living - 2011.01-02

moutain living 2009 08

moutain living - 2011.05-06

logs – solve using logs method

jack ryger: brighton moutain, utah

sd reconsider

jack mccullough motion to reconsider

logs...............the back logs

salt moutain

edward said and to reconsider his

moutain living - 2009.05-06

vaude moutain sports winter 2010/11

logs, logs, logs, from natural element homes

43 motion to reconsider

reconsider our built environment

c:\fakepath\reconsider baby

moutain land & homes jan 2013

moutain bike

dossier moutain wilderness