application logging for large systems

©2010 Improving Enterprises, Inc.

Logging For Fun and Profit

Jane Prusakova

[email protected]://softwareandotherthings.blogspot.com

Improving Enterprises

Houston JUG

2014

mailto:[email protected]


Jane Prusakova

Hard-core developer

Data science geek

Dancer and photographer


Why

How-to

Using the results

So lets talk about logging…


Peek into the soul

Real setup

Real data

Real users

Real workDev environment

Happy path

Minimum load

Limited data


Inside the cloud

No programming-by-coincidence

Replaces debugging

Works on server


Logging [vs Debugging]

Set up once, always ready to use

Minimum disruption to the flow

Works locally and in the cloud

Faster data collection

Collected data is persisted


Logging helps with…

Integration

Multithreading

High load


What else?

Trace what the users do

Features

Sequences

Data flow


When to use logging?

New systems

New features

Bug fixing


Are logs forever?

Permanent

Errors

Extraordinary situations


Some are seasonal

Very temporary

Debugging info

Trace bug fixes


Others last awhile

Medium term

Data flow

Resource utilization

Optimization points


How-to

Using the results


Yes, I want to setup logging!

Pick a framework

Configure logging

Storage and access

Getting value from logs


Log4J SLF4J

Apache MIT license

Logging framework Logging facade

Configuration, log levels and categories,

rotate log files, thread-safe logging.

Support: online documentation, tutorials,

online forums.


Database Files

Easy to parse,

hard to evolve

Very hard to parse,

easy to change

Uses DB

connections

Requires FS

access

Can cause app

crash, loss of logs,

slow down app

Can fill up space


Where to write logs?

Console

File system

DB

Queue mechanisms

Combination


Logging configuration

Java properties-style

XML

Programmatically

Hard-coded

Takes preference


Log Message

What to log with the message

Importance level

Timestamp

Where in the code

Data being processed

System state info


Example layout

log4j.appender.stdout.layout.ConversionPattern=

%d %5p [%t] (%F:%L) - %m%n

- Date

- Log Level

- Thread name

- File and line (slow to retrieve)


Good log messages

Support code

Correctness

Readability

Performance


Better log messages

Relate to generating point

Code

Data


Best log messages

Are formatted for

Readability

Aggregation


Logging objects

.toString()

org.apache.commons.lang.builder.ToStringBuilder

log.info(myDataObject);


What to log

Input and output

Errors and exceptions

Computation results

Integration points

Thread rendezvous


Input and output

public int calculateCost(String user, Flight flight)

{

log.debug(“calculateCost: “ + user + “: “ +

flight);

int cost = … ;

… // cost calculation

log.debug((“calculateCost: cost for “ + user

+ “: “ + flight + “ = “ + cost);

return cost;

}



// do dangerous work

result = DoDangerousWork(user, flight);}

catch (Exception e)

{

log.error(“methodName: exception in

DoDangerousWork for “ + user + “: “ + flight, e);

// recover or re-throw

}


More

Log destination, parameters, outcomesDB calls

SOAP calls

Wait time

Thread joins

Available memory

Memory-intensive operations

Intermediate calculation results

For complicated calculations


Avoid

Little information

… // step 1

logger.info(“did something - XXXX”);

… // step 2

logger.info(“did more work - YYYY”);

foreach (…) {

… // useful work

logger.info(“working hard”);

}


Avoid

Code cluttering

if (logger.isDebugEnabled()) {

logger.debug(“Show Views");

logger.debug("Address:\n" +

view.getAddress());

logger.debug("Email:\n" +

view.getEmail());

}


Watch for logging-caused bugs

Side effects

Logger.info(count++ + “: did something”);


Logger.info(m.GetValue());


DRY principle applies


Growing, growing…

Naming conventions

Rotation

by size

by time


You’ve got logs!

Real-time access

Deal with space limitations

Warning: large datasets


Security

Absolutely

No Sensitive Data


Using the results


Learning from the logs

Never rely on eye-balling

WYS is not WYG

Distinguish common and

rare


Code and logs


Process log

Line by line

No need to load entire file

Aggregate events

Relate to time


Tools

Utilities

grep

sort

uniq

Scripting

Perl

Awk

sed


Learning from the logs

> grep userName myApplication-MMDDYYYY.log

> grep methodName myApplication-MMDDYYYY.log

> grep FATAL myApplication-MMDDYYYY.log

> tail -2000 myApplication-MMDDYYYY.log | grep Exception


Querying logs using Awk

GNU Awk www.gnu.org/software/gawk

Create histograms

Wait times

Number of calls

Exceptions per time period or per user

Compare and relate events

Exception stacks traces

Outcomes by caller or thread


AWK demo

Logs for the demo generously

provided by

http://42graphy.org/

http://42graphy.org/


Jane.Prusakova at ImprovingEnterprises.com

SoftwareAndOtherThings.blogspot.com

@jprusakova

Jane Prusakova

http://www.slideshare.net/jprusakova


http://www.slideshare.net/jprusakova


Logging For Fun and Profit

Jane Prusakova

[email protected]://softwareandotherthings.blogspot.com

Improving Enterprises

Houston JUG

2014


application logging for large systems

Technology