know more about your ceph cluster with elk stack · logstash provides methods to parse logs simple...
TRANSCRIPT
![Page 1: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/1.jpg)
Know more about your Ceph Cluster with ELK Stack
Cameron Seader
Technology Strategist
![Page 2: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/2.jpg)
2
![Page 3: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/3.jpg)
3
Ceph and Logging
● rsyslog, syslog-ng to forward logs to (Logstash / Fluentbit)● Filebeat● Ceph has Graylog (GELF) support● store logs for later use● analyze logs for Alerting● analyze data with Machine Learning
● X-Pack machine learning● R client
● trouble shooting/postmortem analyze
![Page 4: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/4.jpg)
4
Forwarding logs
● you can format your logs before forwarding● there is a tutorial for rsyslog how to reformat to GELF● Logstash has a lot of pipeline input modules
● syslog { }● graylog { }
● Filebeat
![Page 5: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/5.jpg)
5
Ceph GELF
● to forward logs in GELF, update ceph.conf● log_to_graylog = true● log_graylog_host = 127.0.0.1● log_graylog_port = 12201
● restart ceph services● or use DeepSea
● has support for custom Ceph config options● salt-run state.orch ceph.stage.3● that also would restart services correctly
![Page 6: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/6.jpg)
6
Parse and Manage
● Logstash provides methods to parse logs● simple alerting could be done with Logstash● use grok { } and other Filters of Logstash
● to add fields● better indexing and managing your data
● Ceph Logstash example● Supportconfig Analyzer
![Page 7: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/7.jpg)
7
Logstash Pipeline filter example
filter {
if [type] == "cephlog" {
grok {
# https://github.com/ceph/ceph/blob/master/src/log/Entry.h
match => { "message" => "(?m)%{TIMESTAMP_ISO8601:stamp}\s%{NOTSPACE:thread}\s*%{INT:prio}\s(%{WORD:subsys}|):?\s%{GREEDYDATA:msg}" }
# https://github.com/ceph/ceph/blob/master/src/common/LogEntry.h
match => { "message" => "%{TIMESTAMP_ISO8601:stamp}\s%{NOTSPACE:name}\s%{NOTSPACE:who_type}\s%{NOTSPACE:who_addr}\s%{INT:seq}\s:\s%{PROG:channel}\s\[%{WORD:prio}\]\s%{GREEDYDATA:msg}" }
}
date { match => [ "stamp", "yyyy-MM-dd HH:mm:ss.SSSSSS", "ISO8601" ] }
}
}
![Page 8: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/8.jpg)
8
The Tools
ELK
EFK
Graylog
![Page 9: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/9.jpg)
9
How to get it
● ELK● docker-compose is simple for dev needs
● EFK● Helm charts for each ● install individually in a cluster
● Graylog● Separate installed cluster
● Future● Ceph in containers (Rook) converged with these tools on K8s
![Page 10: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/10.jpg)
10
![Page 11: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/11.jpg)
11
![Page 12: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/12.jpg)
12
![Page 13: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/13.jpg)
13
![Page 14: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/14.jpg)
14
Kibana Search
![Page 15: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/15.jpg)
15
RGW
Object storage client to the ceph cluster, exposes a RESTFUL S3 & Swift API
![Page 16: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/16.jpg)
16
RGW
● A RESTful API access to object storage, a la S3● implements user accounts, acls, buckets● heavy ecosystem of s3/swift client tooling can be leveraged against
RGW
![Page 17: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/17.jpg)
17
RGW
● Supports a lot of S3 like features● Multipart uploads● Object Versioning● torrents● lifecycle● encryption● compression● static websites● metadata search...
● From Jewel we support multisite which allows geographical redundancy
![Page 18: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/18.jpg)
18
ElasticSearch
You know, for search
● distributed● horizontally scalable● schemaless● speaks REST● Easy Configuration without setting your hair on fire
![Page 19: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/19.jpg)
19
RGW Metadata search with ES Motivation
● Objects have metadata associated with them that is often interesting to analyze
● Since it is an “object storage” you don’t have any traditional filesystems tool at your disposal
● No du, df & friends, and either way these are hard on a distributed storage system
![Page 20: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/20.jpg)
20
Motivation
● Some existing support with admin API, however the problems with this:● returns specific metadata, not ideal for aggregation● no notifications when new objects/buckets/accounts are created● also permissions for users to access the admin API is tricky, since admin API
was meant for administering● As an storage administrator you'd be interested in finding out for
eg. the top 10 users, average object size etc., no of objects held on a user account etc.
![Page 21: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/21.jpg)
21
Design
● Built atop of the multisite architecture, where data & metadata is forwarded to multiple zones
● From Kraken, we have sync plugins● Allows for data & metadata to be forwarded to external tiers, allows
for building of:● Interesting solutions analyzing bucket/object/user metadata (ES for starts)● Backup solutions (S3/cloud sync plugin for Mimic)
![Page 22: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/22.jpg)
22
Elastic Sync Plugin
● Forwards metadata from other zones onto a ES instance● Requires a read only zone that doesn't cater to user requests &
only forwards to ES● No off the shelf authentication module that can work with RGW● Recommendation to not expose ES endpoint to public
![Page 23: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/23.jpg)
23
Elastic Sync Plugin: User Requests
For normal user requests, RGW itself can authenticate the user; ensures users don't see other's data
We have an attribute mentioning owners for an object and this is used to service user requests
Also allows for custom metadata fields to be set up per user
Elastic queries to analyze common system faults
Integration into Ceph dashboard
Analysis of meta and/or log data with Machine Learning
Contribute
https://github.com/denisok/elk_supportconfig Github Repo for ongoing ELK work
https://ceph.com/IRC/ - Ceph upstream community mailing lists and IRC channels
http://lists.suse.com/mailman/listinfo/deepsea-users - DeepSea upstream mailing list.
https://groups.google.com/forum/#!forum/openattic-users - openATTIC upstream mailing list.
https://github.com/ceph/ceph - upstream Ceph sources
Questions?
![Page 24: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/24.jpg)
24
metadata
RGW Primary RGW secondary ES
Ceph
S3
![Page 25: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/25.jpg)
25
Example Metadata
{
_index" : "rgw-default-6cb1f916",
"_type" : "object",
"_id" : "86740559-297e-4487-b770-d3106b900a97.34125.1:american-gods:null",
"_score" : 0.2876821,
"_source" : {
"bucket" : "s3bucket",
"name" : "american-gods",
"instance" : "null",
"versioned_epoch" : 0,
"owner" : {
"id" : "jane.doe",
"display_name" : "Jane Doe"
},
"permissions" : [
"jane.doe"
],
"meta" : {
"size" : 1387924,
"mtime" : "2018-02-01T22:27:36.490Z",
"etag" : "d41d8cd98f00b204e9800998ecf8427e",
"tail_tag" : "86740559-297e-4487-b770-d3106b900a97.34125.7336",
"x-amz-content-sha256" : "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"x-amz-date" : "20180201T222736Z"
},
"tagging" : [
{
"key" : "author",
"value" : "gaiman"
}]}}
![Page 26: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/26.jpg)
26
Example Query
curl -XPOST 'localhost:9200/rgw-gold/_search?size=0&pretty' -d
{
"aggs" : {
"avg_size" : { "avg" : { "field" : "meta.size" } }
}
}
![Page 27: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/27.jpg)
27
Response
{
"took" : 22,
"timed_out" : false,
"_shards" : {
"total" : 10,
"successful" : 10,
"failed" : 0
},
"hits" : {
"total" : 22,
"max_score" : 0.0,
"hits" : [ ]
},
"aggregations" : {
"avg_size" : {
"value" : 177.72727272727272
}
}
}
![Page 28: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/28.jpg)
28
Interesting queries possible
● Object storage PUT requests on a specific time range.● Stats on objects with specific metadata content
● It is possible to index metadata to non string fields on a per bucket basis
![Page 29: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/29.jpg)
29
Future Work
● Support for ES 6 for RGW● Custom metadata fields for object tagging● Elastic queries to analyze common system faults● Integration into Ceph dashboard● Analysis of meta and/or log data with Machine Learning
![Page 30: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/30.jpg)
30
Contribute
● https://github.com/denisok/elk_supportconfig Github Repo for ongoing ELK work
● https://ceph.com/IRC/ - Ceph upstream community mailing lists and IRC channels
● http://lists.suse.com/mailman/listinfo/deepsea-users - DeepSea upstream mailing list.
● https://groups.google.com/forum/#!forum/openattic-users - openATTIC upstream mailing list.
● https://github.com/ceph/ceph - upstream Ceph sources
![Page 31: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/31.jpg)
31
Questions?
![Page 32: Know more about your Ceph Cluster with ELK Stack · Logstash provides methods to parse logs simple alerting could be done with Logstash use grok { } and other Filters of Logstash](https://reader034.vdocuments.site/reader034/viewer/2022043004/5f87a816aa8e7f29b530a50b/html5/thumbnails/32.jpg)