logging for containers
TRANSCRIPT
Logging for Containers
Eduardo Silva @edsiper
Logging
Logging
● Monitoring
● Troubleshooting
● Statistics
It matters
● Operating System
● Applications Unit
● Services
Logging Scenarios
Distribution Channels
Containers
Containers
Everything is about Isolation
● File System
● Standard I/O interfaces (stdout / stderr)
● Over Network
Containers & Log Handlers
Distribution Channels
● Json-file● Syslog● Journald● Fluentd● ...
Docker Implement drivers for different formats and distribution channels:
Docker Logging Drivers
Output / Destinations
Structured Logs
● Often based in Key-Value pairs
● Two minimum keys: time and message
Structured logging makes data processing easier
Structured Logs
Distribution Channels
● JSON: readable format for structured data
● MessagePack: Binary serialization (json-like)
Structured logging makes data processing easier
Structured Logs
Common Formats
Original Log Message
This is a test message
Structured Log Message
{ "container_id":"bfdd5b9...", "container_name":"/infallible_mayer", "source":"stdout", "log": "This is a test message"}
Structured Logs
Docker log example
Microservices
Monolithic● A service produces all
data about users access
Microservices● Many services produce
data about users access● Log needs to be collected
from many services.
Microservices
● How to deal with different input formats ?
● Parse plain text is really expensive.
● Not all containers have permanent storage.
● Where to write the logs ?
Microservices
Logging Challenges
Distributed Logging
Distributed Logging
Architecture
Collector● Retrieve raw logs: file system / network.● Parse log content.
Aggregator● Get data from multiple sources.● Convert incoming data into Streams.
Destination● Retrieve data streams from Aggregator.● Store formatted logs (records).
Distributed Logging
Workflow
● Network Traffic
● CPU Load: parsing and formatting is expensive
● High Availability / Redundant aggregators
Scaling Logging
Items to consider
Aggregation Patterns
w/o source aggregation with source aggregation
Source Aggregation Patterns
Pros● Simple Configuration
Cons● Fixed Aggregator endpoint address● Many network connections● High load on Aggregator
Aggregation Patterns
Without Source Aggregation
Pros● Less connections● Lower load in aggregator● Less config in Containers
Cons● Need more resources (1 aggregate container
per host.
Aggregation Patterns
With Source Aggregation
w/o destination aggregation With destination aggregation
Destination Aggregation Patterns
Pros● Less Nodes● Simpler configuration
Cons● Storage side changes affects collector side● Worse performance: many small write requests
on storage
Aggregation Patterns
Without Destination Aggregation
Pros● Collector side configuration is
free from storage side changes.● Better performance with fine
tune on destination side aggregator.
Cons● More Nodes.● More complex configuration.
Aggregation Patterns
With Destination Aggregation
● High Performance
● Built-in Reliability
● Structured Logs
● Pluggable Architecture
● More than 300 plugins! (input/filtering/output)
Open Source Data/Log Collector
Architecture / Workflow
● Docker Interoperability Native Docker logging driver to use Fluentd
● KubernetesFluentd as main aggregator (notes)
● OpenShiftFluentd as main aggregator
Full Collector/Aggregator for Containers
Docker use case
Docker use case
Kubernetes use case
Fluentd in the Real World
Microsoft use case
We collect1.8M events per second !
It’s a proud member of:
NEWS!
Fluentd is joining CNCF!
Let’s back to Logging
● Written in C
● High Performance
● Pluggable Architecture
● Built-in CPU / Memory metrics / Network TLS support
● Event-Driven
● Fluentd Compatible!
Fluent Bit
Lightweight log aggregator
Fluent Bit
Architecture
Messages Forwarding Performance
500K messages per second! (1 CPU core)
Architecture
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
● Write plugins in Golang!
● Functional and upcomingfor v0.10 release.
● Do you want to contribute ?
Fluent Bit - Announcement
Golang Support!
Fluent Bit - Announcement
Golang Support!
● Write Go plugins!
● Built-in HTTP Monitoring
● Variables
● Filtering
Fluent Bit
Roadmap v0.10
Eduardo Silva @[email protected]
http://fluentd.org http://fluentbit.io
Thanks!
Logging and Open Source matters!