containers and logging

Download Containers and Logging

Post on 18-Feb-2017




0 download

Embed Size (px)


  • Containers and Logging Challenges and Solutions

    Eduardo Silva (@edsiper) LinuxCon Europe 2016

  • Logging

  • Logging




    It matters

  • Logging Scenarios

    Operating System

    Applications Unit


    Distribution Channels

  • Containers

  • ContainersEverything is about Isolation

  • Containers & Log Handlers

    File System

    Standard I/O interfaces (stdout / stderr)

    Over Network

    Distribution Channels

  • Docker Logging Drivers

    Json-file Syslog Journald Fluentd ...


    Docker Implement drivers for different formats and distribution channels:

  • Structured Logs

  • Structured Logs

    Often based in Key-Value pairs

    Two minimum keys: time and message

    Distribution Channels

    Structured logging makes data processing easier

  • Structured Logs

    JSON: readable format for structured data

    MessagePack: Binary serialization (json-like)

    Common format

    Structured logging makes data processing easier

  • Structured LogsDocker log example

    Original Log Message

    This is a test message

    Structured Log Message

    { "container_id":"bfdd5b9...", "container_name":"/infallible_mayer", "source":"stdout", "log": "This is a test message"}

  • Microservices

  • MicroservicesMonolithic

    A service produces alldata about users access

    Microservices Many services produce

    data about users access Log needs to be collected

    from many services.

  • Microservices

    How to deal with different input formats ?

    Parse plain text is really expensive.

    Not all containers have permanent storage.

    Where to write the logs ?

    Logging Challenges

  • Distributed Logging

  • Distributed LoggingArchitecture

  • Distributed LoggingWorkflowCollector

    Retrieve raw logs: file system / network. Parse log content.

    Aggregator Get data from multiple sources. Convert incoming data into Streams.

    Destination Retrieve data streams from Aggregator. Store formatted logs (records) .

  • Scaling Logging

    Network Traffic

    CPU Load: parsing and formatting is expensive

    High Availability / Redundant aggregators

    Topics to consider

  • Aggregation Patterns

  • Source Aggregation Patterns

    w/o source aggregation with source aggregation

  • Aggregation PatternsWithout Source Aggregation

    Pros Simple Configuration

    Cons Fixed Aggregator endpoint address Many network connections High load on Aggregator

  • Aggregation PatternsWith Source Aggregation

    Pros Less connections Lower load in aggregator Less config in Containers

    Cons Need more resources (1 aggregate container

    per host.

  • DestinationAggregation Patterns

    w/o destination aggregation with destination aggregation

  • Aggregation PatternsWithout Destination Aggregation

    Pros Less Nodes Simpler configuration

    Cons Storage side changes affects collector side Worse performance: many small write requests

    on storage

  • Aggregation PatternsWith Destination Aggregation

    Pros Collector side configuration is

    free from storage side changes. Better performance with fine

    tune on destination side aggregator.

    Cons More Nodes. More complex configuration.

  • Open Source Data/Log Collector

    High Performance

    Built-in Reliability

    Structured Logs

    Pluggable Architecture

    More than 300 plugins! (input/filtering/output)

  • Architecture / Workflow

  • Full Collector/Aggregator for Containers

    Docker Interoperability Native Docker logging driver to use Fluentd

    KubernetesFluentd as main aggregator (notes)

    OpenShiftFluentd as main aggregator

  • Docker use case

  • FluentdDocker use case

  • FluentdKubernetes use case

  • Fluentd in the Real World

  • Microsoft Use Case

  • We collect1.6M events per second !

  • Its a proud member of:

  • Fluentd + CNCF ?Logging

  • Fluentd + CNCF ?Application: Work in Process

  • Fluent BitLightweight log aggregator Written in C

    High Performance

    Pluggable Architecture

    Built-in CPU / Memory metrics / Network TLS support


    Fluentd Compatible

  • Architecture

  • Library Mode

  • Library Mode

  • Library Mode

  • Library Mode

  • Library Mode

  • Library Mode

  • Library Mode

  • Library Mode

  • Library Mode

  • Library Mode

  • Library Mode

  • Messages Forwarding Performance

    500K messages per second! (1 CPU core)

  • Fluent BitRoadmap

    Built-in HTTP Monitoring

    Lua scripting support

  • Thank You!

    Eduardo Silva /

    Slide 1Slide 2Slide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Slide 21Slide 22Slide 23Slide 24Slide 25Slide 26Slide 27Slide 28Slide 29Slide 30Slide 31Slide 32Slide 33Slide 34Slide 35Slide 36Slide 37Slide 38Slide 39Slide 40Slide 41Slide 42Slide 43Slide 44Slide 45Slide 46Slide 47Slide 48Slide 49Slide 50Slide 51Slide 52Slide 53Slide 54Slide 55Slide 56Slide 57Slide 58