Leak, Lock and a Long Pause

Troubleshooting JVM issues

You will learn that

• There are memory leaks in Java• Garbage collector can be a bottleneck• Thread locks are more common than you

think• You can automatically detect these problems

How can JVM leak memory?

• Static fields are not gc’d• Thread-local + thread pools• Application scope• Complex bi-directional refs• OS resource handles: files, sockets, db

Static fields

class Forever { // Unless you explicitly unload this class // GC will not touch it static final ArrayList list = new ArrayList(1000);}

Thread-local

public void doFilter(ServletRequest req, ServletResponse res){ myThreadLocal.set(new BigLeak()); // thread goes back to the thread pool // with the big leak planted chain.doFilter(req, res);}

Application scope

getServletContext().setAttribute(”leak", bigObj);// unless you call removeAttribute somewhere// this will live as long as Tomcat is up

Bi-directional relationships

org.w3c.dom.Document doc = readXmlDocument();org.w3c.dom.Node child = doc.getDocumentElement().getFirstChild();doc.removeNode(child);doc = null;// child still has reference to doc. No GC for the doc.

OS resources

try { Connection conn = ConnectionFactory.getConnection(); ... conn.close();} catch (Exception e) { // oops, connection is still open}

GC Pause

source: oracle.com

GC Pause

• Full garbage collection stops the world!• Heavily utilized heap = frequent pauses• Complex dependency graph = long pause• Lots of long living objects = long pauses

Heap size vs pause

source: performanceterracotta.blospot.com

Is it really our problem?

source: plumbr.eu

Is it really our problem?

QUESTION:Does it impact our business if the system regularly suspends itself for several seconds?BETTER QUESTION:What is the level of unresponsiveness we allow in our system?As low as possible = bullshit answer

No measurement = no answer

pauses for about half a second 2-3 times a minute

GC friendly programming

• Avoid long living objects – minor gc is faster• Avoid mutable objects – questionable…• Design your classes small – faster compaction• Avoid complex dependency graphs• NEVER call System.gc() – disturbs optimal plan

Thread locking

“Plain Threads are the GOTO of Today’s Computing”--Dr Hartmut Kaiser

Thread locking

“Locking is not evil, lock contention is”Vladimir Šor

Contention

Thread contention

source: blogs.mulesoft.org

Thread contention - causes

• Too broad sync scope• Overusing same locking object• Shared mutable state abuse

What’s wrong here?

public boolean isAdmin(String name) { synchronized (privileges) { String privs= privileges.get("usr.”+name+".prv"); if (privs== null) return false; else return (privs.indexOf(”administrator") >= 0); }}

Too broad sync scope// “Protect data, not code” – Michael Suesspublic boolean isAdmin(String name) { synchronized (privileges) { String privs= privileges.get("usr.”+name+".prv"); if (privs== null) return false; else return (privs.indexOf(”administrator") >= 0); }}

What’s wrong here?public class Attributes { private HashMap usersStore; private HashMap servicesStore;

public synchronized void setUserAttributes(String usr, UserAttrs attrs) { usersStore.put(usr, attrs); }

public synchronized void setSvcAttributes(String svc, ServiceAttrs attrs) { servicesStore.put(svc, attrs); }

Overusing a lockpublic class Attributes { private HashMap usersStore; private HashMap servicesStore;

public synchronized void setUserAttributes(String usr, UserAttrs attrs) { usersStore.put(usr, attrs); }

public synchronized void setSvcAttributes(String svc, ServiceAttrs attrs) { servicesStore.put(svc, attrs); }

// both methods sync on same Attributes instance

Shared mutable state

“Every piece of mutable state you have is a liability”Martin Odersky

Is there really no other way?

balance.deposit(amount)

Is there really no other way?

newBalance = oldBalance.deposit(amount)// no sync needed

Share immutable objects

public class Address { public final String streetName; public final int aptNo; public final String zipCode; public Address (String name, int apt, String zip){ // initialize all the fields }}

Jhiccup

JVM AppJVM App

jhiccup thread

hiccup.hloghiccup.hlogmeasure thread sleepdelta(expected – actual)

src: azulsystems.com

What is a JVM Hiccup?

• When your application is stalled• GC stop-the-world• Virtualization• CPU contention• OS Scheduler

Plumbr

System running on JVMSystem running on JVM

plumbr agentplumbr agent

plumbr serverplumbr server

usage data

DeMOtime for

Sources

• plumbr.eu/blog/how-do-leak-detectors-work-plumbr-case-study

• java.dzone.com/articles/how-tame-java-gc-pausesthinkingparallel.com/2007/07/31/10-ways-to-reduce-lock-contention-in-threaded-programs/

• blogs.mulesoft.org/chasing-the-bottleneck-true-story-about-fighting-thread-contention-in-your-code

• ibm.com/developerworks/library/j-threads2/• http://www.azulsystems.com/giltene/how-java-got-

the-hiccups