thread dump analysis

OverviewReal life situations

Thread dump format

Taking thread dumps

TDA tools

Reading material

Questions

Real life situations

Problems with I/OThreads maybe blocked or perform slowly during access to I/O resources:

File system

Network

Database

Single or multiple thread dumps may help understading the problem

Problems withalgorithms

Algorithm complexity is not suited well for the data set size

Algorithm is overusing expensive resources or memory

Algorithm is overusing locks, while it can be implemented in a lock-free manner

Single or multiple thread dumps may help noticing the problem

Problems withlocks and

syncronizationNo locking (not thread-safe implementations)

Deadlocks (overly syncronised)

Lock contention (overly syncronised)

Lock overhead (too many locks)

Single or multiple thread dumps may explain the situation

Example:HashMap gives100% CPU load

Usage of not syncronised in a multi-threaded applicationHashMap

In certain situations it goes into infinite loop

This causes 100% loadCPU

The probabilty of entering infinite loop depends on the size of the HashMap

Single thread dump reveals the problem

DeadlocksFrom Wikipedia:

“A deadlock is a situation in which two or more competing actions are each waitingfor the other to finish, and thus neither ever does.

DeadlocksIn real life, deadlocks are pretty rare

CPU usage is usally close to 0

Single thread dump can reveal a deadlock situation

Lock contentionFrom Wikipedia:

“This occurs whenever one process or thread attempts to acquire a lock held byanother process or thread. The more granular the available locks, the less likely oneprocess/thread will request a lock held by the other. (For example, locking a rowrather than the entire table, or locking a cell rather than the entire row.)

Lock contentionThe most common problem with multi-threaded applications related to lockhandling

Sometimes easy, sometimes very hard to detect

You can only see one layer at a time

Multiple thread dumps may help to detect lock contention situations

Sporadicapplicationbehaviour

May be related to not thread safe code

May be related to load patterns

May be related to specific data

Threads dumps may or may not help

Thread dumpformat

FormatsFormats differ for various JVM and application server implementations

There is no tool that supports them all

Though format is usually very readble and follows same principles

HotSpot format

Thread states Ihttp://docs.oracle.com/javase/1.5.0/docs/api/java/lang/Thread.State.html

“A thread can be in one of the following states:

NEW - A thread that has not yet started is in this state.

RUNNABLE - A thread executing in the virtual machine is in this state.Java

BLOCKED - A thread that is blocked waiting for a monitor lock is in this state.

Thread states II“

WAITING - A thread that is waiting indefinitely for another thread to perform aparticular action is in this state.

TIMED_WAITING - A thread that is waiting for another thread to perform anaction for up to a specified waiting time is in this state.

TERMINATED - A thread that has exited is in this state.

Taking threaddumps

How?jstack

jconsole

jvisualvm

write your own clientJMX

use your favorite profiler

Demo

Thread dumpanalysis tools

TDA toolsIBM Thread and Monitor Dump analysis

Java Thread Dump Analyzer

Samurai

Thread Dump Analyzer

Thread Dump Viewer

Demo

Book

Questions?

JVM InternalsTraining

Aestas/IT organizes "JVM Internals" training on March 25/26, 2013.

This 2–day course takes the developer on an in-depth tour of the Virtual Machine.JavaIntended for experienced JVM-based programmers, who want to get deeper with theplatform, this course will introduce the major subsystems of the JVM and practical waysto apply this knowledge in real applications.

Topics IHistory of and JVMJava

Specifications

Bytecode

Base components

Detailed architecture

JIT compilation

Garbage collection

Threads

Locks

Classloading

Topics IISecurity

Monitoring tools

Java agents

Debugging

Memory analysis

Tuning principles

GC tuning

Responsiveness and throughput

Tuning Options

RegistrationTraining registration at:

http://aestasit.com/jvm-training-riga-2013

Or by sending e-mail to:

training@aestasit.com

Thanks!