logging for containers

Logging for Containers

Eduardo Silva @edsiper

Logging

Logging

● Monitoring

● Troubleshooting

● Statistics

It matters

● Operating System

● Applications Unit

● Services

Logging Scenarios

Distribution Channels

Containers

Containers

Everything is about Isolation

● File System

● Standard I/O interfaces (stdout / stderr)

● Over Network

Containers & Log Handlers

Distribution Channels

● Json-file● Syslog● Journald● Fluentd● ...

Docker Implement drivers for different formats and distribution channels:

Docker Logging Drivers

Output / Destinations

Structured Logs

● Often based in Key-Value pairs

● Two minimum keys: time and message

Structured logging makes data processing easier

Structured Logs

Distribution Channels

● JSON: readable format for structured data

● MessagePack: Binary serialization (json-like)

Structured logging makes data processing easier

Structured Logs

Common Formats

Original Log Message

This is a test message

Structured Log Message

{ "container_id":"bfdd5b9...", "container_name":"/infallible_mayer", "source":"stdout", "log": "This is a test message"}

Structured Logs

Docker log example

Microservices

Monolithic● A service produces all

data about users access

Microservices● Many services produce

data about users access● Log needs to be collected

from many services.

Microservices

● How to deal with different input formats ?

● Parse plain text is really expensive.

● Not all containers have permanent storage.

● Where to write the logs ?

Microservices

Logging Challenges

Distributed Logging

Distributed Logging

Architecture

Collector● Retrieve raw logs: file system / network.● Parse log content.

Aggregator● Get data from multiple sources.● Convert incoming data into Streams.

Destination● Retrieve data streams from Aggregator.● Store formatted logs (records).

Distributed Logging

Workflow

● Network Traffic

● CPU Load: parsing and formatting is expensive

● High Availability / Redundant aggregators

Scaling Logging

Items to consider

Aggregation Patterns

w/o source aggregation with source aggregation

Source Aggregation Patterns

Pros● Simple Configuration

Cons● Fixed Aggregator endpoint address● Many network connections● High load on Aggregator

Aggregation Patterns

Without Source Aggregation

Pros● Less connections● Lower load in aggregator● Less config in Containers

Cons● Need more resources (1 aggregate container

per host.

Aggregation Patterns

With Source Aggregation

w/o destination aggregation With destination aggregation

Destination Aggregation Patterns

Pros● Less Nodes● Simpler configuration

Cons● Storage side changes affects collector side● Worse performance: many small write requests

on storage

Aggregation Patterns

Without Destination Aggregation

Pros● Collector side configuration is

free from storage side changes.● Better performance with fine

tune on destination side aggregator.

Cons● More Nodes.● More complex configuration.

Aggregation Patterns

With Destination Aggregation

● High Performance

● Built-in Reliability

● Structured Logs

● Pluggable Architecture

● More than 300 plugins! (input/filtering/output)

Open Source Data/Log Collector

Architecture / Workflow

● Docker Interoperability Native Docker logging driver to use Fluentd

● KubernetesFluentd as main aggregator (notes)

● OpenShiftFluentd as main aggregator

Full Collector/Aggregator for Containers

Docker use case

Docker use case

Kubernetes use case

Fluentd in the Real World

Microsoft use case

We collect1.8M events per second !

It’s a proud member of:

NEWS!

Fluentd is joining CNCF!

Let’s back to Logging

● Written in C

● High Performance

● Pluggable Architecture

● Built-in CPU / Memory metrics / Network TLS support

● Event-Driven

● Fluentd Compatible!

Fluent Bit

Lightweight log aggregator

Fluent Bit

Architecture

Messages Forwarding Performance

500K messages per second! (1 CPU core)

Architecture

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

● Write plugins in Golang!

● Functional and upcomingfor v0.10 release.

● Do you want to contribute ?

Fluent Bit - Announcement

Golang Support!

Fluent Bit - Announcement

Golang Support!

● Write Go plugins!

● Built-in HTTP Monitoring

● Variables

● Filtering

Fluent Bit

Roadmap v0.10

Eduardo Silva @edsipereduardo@treasure-data.com

http://fluentd.org http://fluentbit.io

Thanks!

Logging and Open Source matters!