Download - Containers and Logging
Containers and Logging Challenges and Solutions
Eduardo Silva (@edsiper) LinuxCon Europe 2016
Logging
Logging
● Monitoring
● Troubleshooting
● Statistics
It matters
Logging Scenarios
● Operating System
● Applications Unit
● Services
Distribution Channels
Containers
ContainersEverything is about Isolation
Containers & Log Handlers
● File System
● Standard I/O interfaces (stdout / stderr)
● Over Network
Distribution Channels
Docker Logging Drivers
● Json-file● Syslog● Journald● Fluentd● ...
Outputs
Docker Implement drivers for different formats and distribution channels:
Structured Logs
Structured Logs
● Often based in Key-Value pairs
● Two minimum keys: time and message
Distribution Channels
Structured logging makes data processing easier
Structured Logs
● JSON: readable format for structured data
● MessagePack: Binary serialization (json-like)
Common format
Structured logging makes data processing easier
Structured LogsDocker log example
Original Log Message
This is a test message
Structured Log Message
{ "container_id":"bfdd5b9...", "container_name":"/infallible_mayer", "source":"stdout", "log": "This is a test message"}
Microservices
MicroservicesMonolithic
● A service produces alldata about users access
Microservices● Many services produce
data about users access● Log needs to be collected
from many services.
Microservices
● How to deal with different input formats ?
● Parse plain text is really expensive.
● Not all containers have permanent storage.
● Where to write the logs ?
Logging Challenges
Distributed Logging
Distributed LoggingArchitecture
Distributed LoggingWorkflowCollector
● Retrieve raw logs: file system / network.● Parse log content.
Aggregator● Get data from multiple sources.● Convert incoming data into Streams.
Destination● Retrieve data streams from Aggregator.● Store formatted logs (records) .
Scaling Logging
● Network Traffic
● CPU Load: parsing and formatting is expensive
● High Availability / Redundant aggregators
Topics to consider
Aggregation Patterns
Source Aggregation Patterns
w/o source aggregation with source aggregation
Aggregation PatternsWithout Source Aggregation
Pros● Simple Configuration
Cons● Fixed Aggregator endpoint address● Many network connections● High load on Aggregator
Aggregation PatternsWith Source Aggregation
Pros● Less connections● Lower load in aggregator● Less config in Containers
Cons● Need more resources (1 aggregate container
per host.
DestinationAggregation Patterns
w/o destination aggregation with destination aggregation
Aggregation PatternsWithout Destination Aggregation
Pros● Less Nodes● Simpler configuration
Cons● Storage side changes affects collector side● Worse performance: many small write requests
on storage
Aggregation PatternsWith Destination Aggregation
Pros● Collector side configuration is
free from storage side changes.● Better performance with fine
tune on destination side aggregator.
Cons● More Nodes.● More complex configuration.
Open Source Data/Log Collector
● High Performance
● Built-in Reliability
● Structured Logs
● Pluggable Architecture
● More than 300 plugins! (input/filtering/output)
Architecture / Workflow
Full Collector/Aggregator for Containers
● Docker Interoperability Native Docker logging driver to use Fluentd
● KubernetesFluentd as main aggregator (notes)
● OpenShiftFluentd as main aggregator
Docker use case
FluentdDocker use case
FluentdKubernetes use case
Fluentd in the Real World
Microsoft Use Case
We collect1.6M events per second !
It’s a proud member of:
Fluentd + CNCF ?Logging
Fluentd + CNCF ?Application: Work in Process
Fluent BitLightweight log aggregator
● Written in C
● High Performance
● Pluggable Architecture
● Built-in CPU / Memory metrics / Network TLS support
● Event-Driven
● Fluentd Compatible
Architecture
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
Library Mode
Messages Forwarding Performance
500K messages per second! (1 CPU core)
Fluent BitRoadmap
● Built-in HTTP Monitoring
● Lua scripting support
