containers and logging - events.static.linuxfound.org · containers and logging challenges and...

Containers and Logging Challenges and Solutions

Eduardo Silva (@edsiper) LinuxCon Europe 2016

Logging

Logging

● Monitoring

● Troubleshooting

● Statistics

It matters

Logging Scenarios

● Operating System

● Applications Unit

● Services

Distribution Channels

Containers

ContainersEverything is about Isolation

Containers & Log Handlers

● File System

● Standard I/O interfaces (stdout / stderr)

● Over Network

Distribution Channels

Docker Logging Drivers

● Json-file● Syslog● Journald● Fluentd● ...

Outputs

Docker Implement drivers for different formats and distribution channels:

Structured Logs

Structured Logs

● Often based in Key-Value pairs

● Two minimum keys: time and message

Distribution Channels

Structured logging makes data processing easier

Structured Logs

● JSON: readable format for structured data

● MessagePack: Binary serialization (json-like)

Common format

Structured logging makes data processing easier

Structured LogsDocker log example

Original Log Message

This is a test message

Structured Log Message

{ "container_id":"bfdd5b9...", "container_name":"/infallible_mayer", "source":"stdout", "log": "This is a test message"}

Microservices

MicroservicesMonolithic

● A service produces alldata about users access

Microservices● Many services produce

data about users access● Log needs to be collected

from many services.

Microservices

● How to deal with different input formats ?

● Parse plain text is really expensive.

● Not all containers have permanent storage.

● Where to write the logs ?

Logging Challenges

Distributed Logging

Distributed LoggingArchitecture

Distributed LoggingWorkflowCollector

● Retrieve raw logs: file system / network.● Parse log content.

Aggregator● Get data from multiple sources.● Convert incoming data into Streams.

Destination● Retrieve data streams from Aggregator.● Store formatted logs (records) .

Scaling Logging

● Network Traffic

● CPU Load: parsing and formatting is expensive

● High Availability / Redundant aggregators

Topics to consider

Aggregation Patterns

Source Aggregation Patterns

w/o source aggregation with source aggregation

Aggregation PatternsWithout Source Aggregation

Pros● Simple Configuration

Cons● Fixed Aggregator endpoint address● Many network connections● High load on Aggregator

Aggregation PatternsWith Source Aggregation

Pros● Less connections● Lower load in aggregator● Less config in Containers

Cons● Need more resources (1 aggregate container

per host.

DestinationAggregation Patterns

w/o destination aggregation with destination aggregation

Aggregation PatternsWithout Destination Aggregation

Pros● Less Nodes● Simpler configuration

Cons● Storage side changes affects collector side● Worse performance: many small write requests

on storage

Aggregation PatternsWith Destination Aggregation

Pros● Collector side configuration is

free from storage side changes.● Better performance with fine

tune on destination side aggregator.

Cons● More Nodes.● More complex configuration.

Open Source Data/Log Collector

● High Performance

● Built-in Reliability

● Structured Logs

● Pluggable Architecture

● More than 300 plugins! (input/filtering/output)

Architecture / Workflow

Full Collector/Aggregator for Containers

● Docker Interoperability Native Docker logging driver to use Fluentd

● KubernetesFluentd as main aggregator (notes)

● OpenShiftFluentd as main aggregator

Docker use case

FluentdDocker use case

FluentdKubernetes use case

Fluentd in the Real World

Microsoft Use Case

We collect1.6M events per second !

It’s a proud member of:

Fluentd + CNCF ?Logging

Fluentd + CNCF ?Application: Work in Process

Fluent BitLightweight log aggregator

● Written in C

● High Performance

● Pluggable Architecture

● Built-in CPU / Memory metrics / Network TLS support

● Event-Driven

● Fluentd Compatible

Architecture

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

Library Mode

Messages Forwarding Performance

500K messages per second! (1 CPU core)

Fluent BitRoadmap

● Built-in HTTP Monitoring

● Lua scripting support

Thank You!

Eduardo Silva / @edsipereduardo@treasure-data.com

http://fluentd.org http://fluentbit.io