java gc - pause tuning

Everything you ever wanted to know about GC pauses**but were afraid to ask

Death by pauses

Tuesday, July 10, 12

Agenda

1. Introduction

2. Crime Scene Investigation

3. JVM Memory management systems and tools

4. Putting it together

2Tuesday, July 10, 12

The Crime ScenePG 13*

* Parents strongly cautioned: typed language, dead objects and verbose logs may not be suitable to scripting language fans

B2Ce-commerce platform

•12+ Servers•10 different Webapps•50+ JVMs (Oracle JDK6)

•> 30000 sessions•250-400 Req/s•Variance is high

... an unusual victim...

Product catalog modeled as a Graph100% custom implementation

100% on-heap (no SQL except for initial load)in-place update by AtomicReference.set()

... an unusual victim...

Product catalog modeled as a Graph100% custom implementation

100% on-heap (no SQL except for initial load)in-place update by AtomicReference.set()

Caching aggressively is not possibleLarge number of request-scoped objects

Many WS into backoffice systems = latency

Throughput Latency

Interactive e-commerce app:Low latency is the top

priority!

The Crime Scene

JDBC Connections

The Crime Scene

JDBC Connections

Requests/s

The Crime Scene

JDBC Connections

Requests/s

Active threads

The Crime Scene

JDBC Connections

Requests/s

Active threads

HTTP Executor Queue Size

The evidence

1 hour

The evidence

1 hour

Can’t see anything: let’s zoom out!

The evidence

24 hours

The evidence

24 hours

The evidence

24 hours

1 hour

The evidence

Time spent in GC (%)

1 hour

The usual suspects...

• OutOfMemory Heap

• OutOfMemory PermGen

• Long GC pauses

➡ under high load = immediate death

• Long GC pauses

Death by

pauses

Why do we need this GC thing again ?

“Many concurrent algorithms are very easy to write with a GC and totally hard (to down right impossible) using explicit free.”

Cliff Click

Fine, we just need to tune the JVM, right?...

POP QUIZZ!Number of command-line flags*?

* Oracle JVM 1.6.0_31 x86_64 server

less than 100 flags

less than 100 flags100 <= X< 200

less than 100 flags100 <= X< 200200 <= X< 300

less than 100 flags100 <= X< 200200 <= X< 300300 <= X< 400

less than 100 flags100 <= X< 200200 <= X< 300300 <= X< 400400 <= X< 500

less than 100 flags100 <= X< 200200 <= X< 300300 <= X< 400400 <= X< 500500 <= X< 600

less than 100 flags100 <= X< 200200 <= X< 300300 <= X< 400400 <= X< 500500 <= X< 600600 <= X< 700

less than 100 flags100 <= X< 200200 <= X< 300300 <= X< 400400 <= X< 500500 <= X< 600600 <= X< 700 664 Flags!

Memory in the JVM

Permanent (PermGen) Class metadatainterned Strings, etc.

Permanent (PermGen)

Application Objects

Class metadatainterned Strings, etc.

Young / New

Old / Tenured

Permanent (PermGen)

Old / Tenured

Eden S0 S1

Permanent (PermGen)

The Garbage Collector is generational

Eden Sur

vivor 0

Surviv

Eden Sur

vivor 0

Surviv

Allocation

Eden Sur

vivor 0

Surviv

Eden Sur

vivor 0

Surviv

100% = GC!

Eden Sur

vivor 0

Surviv

LiveUnreferenced

Eden Sur

vivor 0

Surviv

Reset...

Eden Sur

vivor 0

Surviv

Eden Sur

vivor 0

Surviv

Allocation

Eden Sur

vivor 0

Surviv

100% = GC !

Eden Sur

vivor 0

Surviv

Eden Sur

vivor 0

Surviv

Eden Sur

vivor 0

Surviv

Eden Sur

vivor 0

Surviv

Génération 1

Génération 2

Eden Sur

vivor 0

Surviv

Allocation

Eden Sur

vivor 0

Surviv

100% = GC !

Eden Sur

vivor 0

Surviv

Eden Sur

vivor 0

Surviv

Promotion

Old“Almost full” = GC !

Compaction(optional)

Garbage Collectors

•Générational

• Stop the world!

• Throughput or Concurrent

GC characteristics

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent

GC characteristics

YoungYoung

OldOldOld

Serial Parallel

Serial Default

Parallel N/A

Concurrent

GC characteristics

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent

GC characteristics

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent

Serial

GC characteristics

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent

Serial Parallel

GC characteristics

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent

Serial Parallel

ParallelOld

GC characteristics

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent

Serial Parallel

ParallelOld

GC characteristics

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent

Serial Parallel

ParallelOld

CMSCMS Serial

GC characteristics

YoungYoung

OldOldOld

Serial Parallel

Serial Serial Parallel

Parallel ParallelOld

Concurrent CMS Serial CMS

Parallel implementation actually differ for each variant

GC characteristics

CMS is the right choice

Serial

Parallel

ParallelOld

CMS Serial

0 250 500 750 1000

Average test duration (s)

Tools: CLI

jps, jhat, jmap, jstack, jstat

$ jstat -gcutil PID S0 S1 E O P YGC YGCT FGC FGCT GCT 0.00 40.88 58.41 18.34 66.65 2729 316.538 46 6.820 323.358

Tools: GUIs

Tools: GUIs (2)

•Any profiler

•During development

• For autopsies!

Tools: GUIs (2)

•Any profiler

•During development

• For autopsies!

HeapDumpOnOutOfMemoryErrorHeapDumpPath

verbose:gc

Stop the world!

verbose:gc

Stop the world!

MBeans

OK, so we can measure... temperature!!

OK, so we can measure... temperature !

58Credit: http://www.lhup.edu/mkhalequ/fieldtrip/geos253.htm

But...a single temperature measure is not enough to diagnose anything!

We must archive all measurementsto know the baseline!

Therefore we must persist all measurements!

• JMX + jmxtrans

• RRD

• Graphite

• etc.

Operating the (many) switches only makes sense...

60Credit: http://www.our-energy.com

...if we can measure/compare the effects!

Before

Putting it together

We want to minimize the GC pauses

Young (ParNew)Old (CMS-initial-mark + CMS-remark)

Tomcat

Application(code)

1. Code

• Tuning the JVM cannot compensate for bad code

• Rules of thumb

• Immutability = object reuse = less allocations *

• Move code invariants out of tight loops

• Know the characteristics of your data structures & frameworks (java.util, Guava, Hibernate, etc.)

• Mind the gap: data structure overhead can kill you!

65* But...pooling can be counter-productive!

Example : HashMap

HashMap

Entry[16]

Example : HashMap

HashMap

Entry[16]

32Overhead = 160 Bytes!

Example : HashMap

HashMap

Entry[16]

32Overhead = 160 Bytes!

•SingletonMap (40 Bytes)•initialCapacity + loadFactor

Less allocations...

... saves CPU

2. Tomcat

• Pooling

• JSP tags: enablePooling in web/webdefault.xml

• -Dorg.apache.jasper.runtime.JspFactoryImpl.USE_POOL=false

• Careful with buffers and their reuse

• -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true

• JSP usage is a factor in PermGen requirements

• Test & Measure, always!

2. Tomcat

• Pooling

• JSP tags: enablePooling in web/webdefault.xml

• -Dorg.apache.jasper.runtime.JspFactoryImpl.USE_POOL=false

• Careful with buffers and their reuse

• -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true

• JSP usage is a factor in PermGen requirements

• Test & Measure, always!

!Pooling may lead

to Old fragmentation!

3. The JVM

pause > 1s !

3. The JVM

Frequent GC

pause > 1s !

The heap

Heap-Xms : start size-Xmx : max size

Young vs Old

-XX:NewSize -XX:MaxNewSize-XX:NewRatio

Young vs Old

Old•“Working Set”•Caches, object pools•HttpSession, average lifespan objects

Objects < RequestScope

First mistake: setting the Young too small

Young fills up quickly = many GC Young

Objects promoted to Tenured too fast = many GC Old

Second mistake: setting Young too large

GC Young pauses increase

Tuning Young

Default NewRatio=8 with -server on Intel

=Too small for a webapp with non-

trivial load!

Tuning Young

Increase Young slowly and measure the effects!

Old: Mind the Gaps (fragmentation)!

OldJDK6 < u22:

ParNew (prom

otion failur

e size = 155

95) (promoti

on failed)

Old generation : ideal shape

Old generation : real life

Old generation : ideal vs. real

Rate increases

Things to watch for

• Traffic/Load variance

• Traffic increases => Memory pressure increase

• CMS requires some headroom to operate properly

• Several phases are concurrent, i.e. at the same time as new objects are allocated

Things to watch for

• Traffic/Load variance

• Traffic increases => Memory pressure increase

• CMS requires some headroom to operate properly

• Several phases are concurrent, i.e. at the same time as new objects are allocated

(concurrent mode failure): 2165740K->1284261K(2228224K), 8.9411250 secs

Giving CMS some room to operate

CMSInitiatingOccupancyFraction = 92%

This is the default....

We really need 75-80%

UseCMSInitiatingOccupancyOnly to force the JVM to only consider this criteria

CMS initial-mark

CMS initial-mark (cumulative)

Median: -83%

Top 99%: -79%

CMS remark

CMS remark (cumulative)

Top 90%: -56%

But...I still see pauses !

• RMI triggers explicit GC regularly

• Invokes System.gc()

• Explicit GC = Full GC (Serial) = 4-8s stop-the-world pause !

• DisableExplicitGC + CMSClassUnloadingEnabled

• ExplicitGCInvokesConcurrentAndUnloadsClasses

Complete GC comparison

• Survivors tuning (S0 & S1)

• Size, ratio vs. Eden, max generation

• G1

• Principles and operations are radically different!

• Other JVMs : JRockit, Azul, IBM

• Check tuning validity after every code change!

• Measure, measure, measure!

What’s next?

Questions ?

java gc - pause tuning

oracle jvm

high load

outofmemory heap12tuesday

gc thing

evidence heapsize

crime scene pg

crime scene investigation3

initial load

Technology

nonstop java & middleware update · 2013-12-12 · •...

gc nirvana - software development conference · gc nirvana...

shenandoah gc - part i: the garbage collector that could ›...

safety pause

jvm performance tuning with respect to garbage collection...

model: gc-269s gc-309b gc-269sa gc-309ba gc-269sv gc …

czechoslovakia: the dubcek pause...title czechoslovakia: the...

low pause gc in hotspot

gc-l207wvq gc-l207wtq gc-l207wlq

the art of garbage collection tuning - angelika langer ·...

shenandoah gc - part ii: i see you have your fancy...

stadtteil bad neuenahr dienstleistung | handwerk einkaufen...

aktive pause

understanding java garbage collection€¦ · gc behavior...

sophie humphreys chief executive pause. pause in action

gc performance tuning - workable...gc performance tuning...

tuning ibms generational gc

become a gc hero · 2017. 7. 19. · key performance...

do your gc logs speak to you63.170: [gc pause (young)...

jvm performance tuning gc tuning parameters -...