centralized logging

Centralized Logging

Logging provides value.

How’s that now?

How does logging provide value?

We can gather information on the system while it’s in operation. We can use this information to provide more value to the business.

More Value?Information gathered from logs allows you to proactively take actions that will keep systems providing the most value to the business.

Talk in circles much?

Centralized logging provides visibility into the operational efficiency of the system and process.

A story on operational sleep efficiency

How is it done?

Logs for most/all systems are shipped, parsed and stored in a central location for monitoring and decision making.

Um…sounds like a bunch of work.

IIS LogsSQL LogsRouter LogsEtc.

Initial focus is on application logging for in-house applications.

Where we are…

Where many are…we’re not alone.

System logs file to directory

System logs file to database

?System

logs file to directory

Where we want to go.

A system that has the ability to store and analyze log data from any system that provides it.

Shipper

App

ViewerParsing, Indexing,

Storage

Logger

Systems

Logging Issuesto resolve

Access To LogGetting access to production servers is not always easy.

Standardizing Logging format

While enabling extensibility…

Log(“Error Occurred, 45, 64”)

Log({Error : “Doh!”,Exeception : ex})

Mobile Client Log

Service Log

Database Log

MVC Application LogDNN Log

Web Client(js) Log

Access to all log data

DB

Service/Logic

Data Tier

Web ClientMobile Client

Logs

Logs

Logs

Logs

Logs

Use log data for…

• Troubleshoot errors and design• Monitoring and notification• Auditing

Business decisions (Analytics)

Upcoming version of Kibana!

The Good

& Bad

of existing solutions

Windows Event Log

• Low Cost – Built in.• Built in UI• Can forward logs to a central server

Good

Windows Event Log

• Event entries stored in memory.• Have to configure logging per server.• Service interface for other clients to

hook into not available.

Bad

Log to central SQL database

• Low Cost – SQL in house, create an database/table.

Good

Log to central SQL database

• Relational (Schema based) format challenging to use for unstructured data.

• logging straight to database could degrade application performance.

Bad

Commercial Hosted Tools

• Handles multiple log formats• Robust charting/searching Features

Good

• Turned down previously• $$$ - pricey for amount of

space allotted

Bad

Commercial Hosted Tools

And the winner is….

A Hybrid Approach.

Application

Redis Queue

Log Indexer

Kibana

Application

EL

Application

Elastic Search

Log

Lo

g

Log

ELEL

Rest APIArchival

Low(No) Cost

OSS FTW!

Extensible

Using a Pub/Sub pattern from the queue will allow other components to subscribe to it.

Redis Queue

Log Indexer(Publisher)

Statistics(Subscriber)

Notifications(Subscriber)

Runs on Windows

Java/Ruby based tools run on windows with some setup.

Blocking

Asynchronous

Mes

sage

Queu

e

No

Why ElasticSearch?

You mean aside from the fact that StackOverflow founder uses it?

Inverted Index“In computer science, an inverted index (also referred to as postings file or inverted file) is anindex data structure storing a mapping from content, such as words or numbers, to its locations in adatabase file, or in a document or a set of documents. The purpose of an inverted index is to allow fast  full text searches, at a cost of

increased processing when a document is added to the database.” via http://en.wikipedia.org/wiki/Inverted_index

Page

WordWordWord

Word

PagePagePage

Regular Index Inverted Index

Documented Oriented• No need to define schema upfront• Can store, index and quickly search unstructured data.• Schema can be defined per type for customization of

the indexing process.

Big Community

• VIA http://www.elasticsearch.org/community/

Easy to scale

• Distributed by nature.• Indexes broken down by shards with 0 or more

replicas.

Easy Level 1 Rest API

• Well documented and straight forward api, which makes it easy to build a client for it.

What’s Needed recap• Server for Elastic Search (Windows/Linux)• Redis Server• Windows service to index logs from queue.• Modify Enterprise Library Logger to send to

queue.

In closingGathering the data is one part. Logging enough information and knowing what questions your are looking to answer is another part.

References