elasticsearch(java)fluentd(ruby) kibana(javascript)
TRANSCRIPT
![Page 1: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/1.jpg)
Elasticsearch(java)Fluentd(ruby)
Kibana(javascript)Система сбора логов
Alexander Sigachev for orel-rb.ru
![Page 2: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/2.jpg)
Какую проблему мы решаем?
Растущее количество серверов усложняет задачу локализации проблемы
Модные микросервисы увеличивают количество logs файлов личный пример: lk, api, billing, admin, pushmsg, websocker-server, thumb-img
docker контейнеры - не имеют состояния, но генерирую логи
![Page 3: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/3.jpg)
Как было раньше?
nginx -> access.log -> logrotate -> access.log.1.gz
grep для поиска
Аналитика zcat | cat | sort | uniq -c | sort -nr
![Page 4: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/4.jpg)
Поиск инструментов
rsyslog
Свой велосипед - на mysql / other db
Набирающая популярность ELK (elasticsearch+logstash+kibana)
что то еще о чем мы не слышали…
![Page 5: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/5.jpg)
Наш выбор - fluend
http://docs.fluentd.org/
он на ruby +1 в карму
Упоминался в докладе на HighLoad (https://goo.gl/JB5fys)
![Page 6: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/6.jpg)
![Page 7: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/7.jpg)
![Page 8: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/8.jpg)
Архитектура fluend
OUTPUTPARSERINPUT FORMATTEDBUFFER
![Page 9: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/9.jpg)
Возможности опробованные нами:
in_tail - чтение access.log nginx
in_forward - прием логов от других агентов для центрально обработки
in_http - http интерфейс
![Page 10: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/10.jpg)
IN_HTTP
<source> @type http port 8888 bind 0.0.0.0 body_size_limit 32m keepalive_timeout 10s</source>
![Page 11: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/11.jpg)
IN_TAIL
<source> @type tail path /var/log/nginx/access.log pos_file /var/log/td-agent/nginx-access.log.pos tag nginx.access format nginx</source>
![Page 12: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/12.jpg)
IN_FORWARD
<source> @type forward port 24224 bind 0.0.0.0</source>
![Page 13: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/13.jpg)
Наш пример match
<match geoip.es.*.*> buffer_chunk_limit 100m buffer_queue_limit 100 @type geoip geoip_lookup_key remote <record> country ${country_code["remote"]} city ${city["remote"]} </record> remove_tag_prefix geoip.</match>
![Page 14: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/14.jpg)
<match es.*.**> @type elasticsearch_dynamic buffer_type memory buffer_chunk_limit 100m buffer_queue_limit 256 host 127.0.0.1 port 9200 resurrect_after 5 reconnect_on_error true logstash_format true logstash_prefix ${tag_parts[1]} include_tag_key true tag_key @log_name time_key_format %Y-%m-%dT%H:%M:%S.%N%z flush_interval 30s</match>
![Page 15: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/15.jpg)
Пример filter
<filter geoip.**.*> @type record_transformer enable_ruby auto_typecast true <record> hostname ${hostname} speed_kbps ${record["size"] * 8 / (record["resptime"] + 0.001) / 1000 } </record></filter>
![Page 16: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/16.jpg)
Схема
source - tail
filter - record
match - forward
source - tail
filter - record
match - forward
source - forward
match - geo
match - elasctic
Elasticsearch kibana
![Page 17: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/17.jpg)
Пример из жизни: что случилось?
![Page 18: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/18.jpg)
query string -> json -> POST in_http
![Page 19: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/19.jpg)
Ресурсы:
Для теста использовался Cloud Hosting https://goo.gl/KfytA8
4Gb 1core - 1000rps RAM + CPU
8Gb 2core - 2000rps RAM + CPU = ~ 31Gb за сутки
16Gb 4core - 3000rps xxxx
![Page 20: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/20.jpg)
Как быстро попробовать?
Берем сервер 8Gb 2Core ~ 80Gb
docker + docker-compose
Настраиваем td-agent (он же fluentd)
Наслаждаемся результатом
Установка elasticsearch + kibana
curl -L https://goo.gl/usglpi | sh -
![Page 22: Elasticsearch(java)fluentd(ruby) kibana(javascript)](https://reader033.vdocuments.site/reader033/viewer/2022061608/5876baaf1a28abad1a8b6b85/html5/thumbnails/22.jpg)
Идеи: fluent-plugin-clickhouse
https://github.com/vladimirich/fluent-plugin-clickhouse