kelk stack on aws
TRANSCRIPT
![Page 1: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/1.jpg)
![Page 2: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/2.jpg)
SteamhausKELK ON AWS
Who am I?Sean Clerkin Senior Site Reliability Engineer
![Page 3: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/3.jpg)
Logging is difficult
![Page 4: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/4.jpg)
No centralised loggingUser needs OS
knowledgeDistribution
Of keys
Enemy of autoscaling
Log rotation
Users download logs unnecessarily
Doesn’t scale To many servers
Slow to find issues
Alertingis hard
Sshing to servers :(
![Page 5: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/5.jpg)
SteamhausKELK ON AWS
ELK is awesomE
![Page 6: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/6.jpg)
SteamhausKELK ON AWS
ELK on ec2
![Page 7: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/7.jpg)
SteamhausKELK ON AWS
KELK on AWS
• Low maintenance - No ec2, Uses entirely AWS serverless technologies and services
• ALB, Cloudfront and Cloudtrail logs are ingested as well as EC2 logs
• Logs are archived in S3 for long term storage, and indexed in Elasticsearch for short term analytics
• Automated with Terraform
• Open source
Kinesis: buffering and delivering instance logs Elasticsearch: Indexing and log storage Lambda: processing and delivering S3 logs Kibana: Search and analytics
![Page 8: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/8.jpg)
SteamhausKELK ON AWS
How does it work?
![Page 9: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/9.jpg)
SteamhausKELK ON AWS
![Page 10: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/10.jpg)
SteamhausKELK ON AWS
![Page 11: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/11.jpg)
SteamhausKELK ON AWS
![Page 12: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/12.jpg)
SteamhausKELK ON AWS
![Page 13: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/13.jpg)
SteamhausKELK ON AWS
![Page 14: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/14.jpg)
SteamhausKELK ON AWS
![Page 15: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/15.jpg)
SteamhausKELK ON AWS
![Page 16: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/16.jpg)
SteamhausKELK ON AWS
![Page 17: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/17.jpg)
SteamhausKELK ON AWS
![Page 18: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/18.jpg)
SteamhausKELK ON AWS
![Page 19: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/19.jpg)
SteamhausKELK ON AWS
![Page 20: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/20.jpg)
SteamhausKELK ON AWS
![Page 21: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/21.jpg)
SteamhausKELK ON AWS
Automationcode
Sample Web Stack
VPC
ALB
EC2
LoggingStack
Kinesis
ElasticsearchService
Lambda
S3
CloudfrontPython
Terraform
Do try this at home!github.com/steamhaus/kelk-example
![Page 22: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/22.jpg)
SteamhausKELK ON AWS
Callouts from the build• It’s not production ready, built for readability • Nailing iam and bucket policies can take a while! • Testing lambda - create a test event in the UI • Use Terraform, rinse and repeat
![Page 23: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/23.jpg)
SteamhausKELK ON AWS
Any Questions..?
![Page 24: KELK Stack on AWS](https://reader034.vdocuments.site/reader034/viewer/2022050722/5a65ac537f8b9aa4758b592f/html5/thumbnails/24.jpg)
Thank you :)Contact [email protected] 820 2020@steamhausmcr
Locate us
Fourways House57 Hilton Street Manchester M1 2EJ