the (not so) dark art of atlassian performance tuning

The (Not So) Dark Art of Atlassian Performance Tuning

Dan Hardiker • CTO • @adaptavist

Standard horizontal view logo.

Purposes: Website, Blogs, External PRs, External websites, Event websites.

Smaller lettering.

Purpose: Business cards.

Logo centralised.

Purpose: Events, stall headers, handouts.

Logo centralised.




Smaller lettering.


Logo centralised.


Logo centralised.


Manage



Smaller lettering.


Logo centralised.


Logo centralised.


•  Users

•  Application Administrators

•  System Administrators

Who do I think you are?!



Smaller lettering.


Logo centralised.


Logo centralised.


•  Performance Testing

•  Load / Capacity

•  Tools in Action

•  List of Solutions

... and lots more!

Out of Scope



Smaller lettering.


Logo centralised.


Logo centralised.


Dark Arts

C A S E S T U D I E S

J F D I

M E T H O D O L O G Y

P E R F O R M A N C E

What is Performance?



Smaller lettering.


Logo centralised.


Logo centralised.




Smaller lettering.


Logo centralised.


Logo centralised.


•  Performance sucks!

•  How do you know?

•  What’s causing it?

•  Are you sure??

So it’s slow …

What affects Performance?



Smaller lettering.


Logo centralised.


Logo centralised.




Smaller lettering.


Logo centralised.


Logo centralised.


Usage Profile



Smaller lettering.


Logo centralised.


Logo centralised.


•  Build a Usage Profile

•  Identify power & automated users

•  Understand the business processes that the applications support

In Summary



Smaller lettering.


Logo centralised.


Logo centralised.




Smaller lettering.


Logo centralised.


Logo centralised.


•  Understand the User Experience

•  Change Control for Configuration

•  Manage the Schemes

In Summary



Smaller lettering.


Logo centralised.


Logo centralised.




Smaller lettering.


Logo centralised.


Logo centralised.


•  Use the Latest Supported JVM

•  Integrate with the OS

•  Monitor all the things

•  Virtualize with Care

•  Define the Scale Required

In Summary



Smaller lettering.


Logo centralised.


Logo centralised.


Dark Arts



J F D I


Infrastructure



Smaller lettering.


Logo centralised.


Logo centralised.


Java



Smaller lettering.


Logo centralised.


Logo centralised.


Java

-XX:+PrintGC -XX:+PrintGCDetails

-XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps

-XX:+PrintTenuringDistribution

--Xloggc:$APP_HOME/logs/gc-$(date "+%m%d%H%M%Y.%S").log



Smaller lettering.


Logo centralised.


Logo centralised.


Atlassian



Smaller lettering.


Logo centralised.


Logo centralised.




Smaller lettering.


Logo centralised.


Logo centralised.


Atlassian

Atlassian



Smaller lettering.


Logo centralised.


Logo centralised.


Dark Arts



J F D I


Request Time



Smaller lettering.


Logo centralised.


Logo centralised.




Smaller lettering.


Logo centralised.


Logo centralised.


•  Caching

•  Encryption

•  Compression

•  Transport

•  Page Generation

•  Connection Queuing

•  Ordering

•  Browser Rendering

Where is the time going?



Smaller lettering.


Logo centralised.


Logo centralised.


•  Filters

•  Authentication & Authorisation

•  Content Rendering

•  Searching

•  Database Queries

•  Plugins

•  External System Dependencies

Factors Affecting Page Generation



Smaller lettering.


Logo centralised.


Logo centralised.


•  Obtain from Access Logs

•  Average over a time window

•  Resolve into a Usage Profile

•  Define use cases, measure APGT for each

•  Use Usage Profile scale for load test setup

Averaging Page Generation Time



Smaller lettering.


Logo centralised.


Logo centralised.




Smaller lettering.


Logo centralised.


Logo centralised.


Time Boxing

Time Boxing

1020ms



Smaller lettering.


Logo centralised.


Logo centralised.


Tooling: Locating the Bottleneck



Smaller lettering.


Logo centralised.


Logo centralised.


•  Access Logging

•  Apache & Application

•  Time-To-First-Byte timing

•  Database Logging

•  Log4JDBC

•  Database Server Tooling

Logging



Smaller lettering.


Logo centralised.


Logo centralised.


•  Application Execution

•  Atlassian Built-in Profiler

•  Targetted Timing

•  AOP or Plugins with Perf4J

•  Sub-millisecond is usually unreliable

Profiling



Smaller lettering.


Logo centralised.


Logo centralised.


CPU



Smaller lettering.


Logo centralised.


Logo centralised.


CPU



Smaller lettering.


Logo centralised.


Logo centralised.


No

CPU



Smaller lettering.


Logo centralised.


Logo centralised.


Yes

No

CPU Operating System



Smaller lettering.


Logo centralised.


Logo centralised.


Yes

No




Smaller lettering.


Logo centralised.


Logo centralised.


Yes JVM

No


Application



Smaller lettering.


Logo centralised.


Logo centralised.


Yes JVM

No

Dark Arts



J F D I


Problem Statement

“It’s slow” erratically

•  Every so often the system becomes unresponsive.

•  The system recovers given some time.

Better Problem Statement



Smaller lettering.


Logo centralised.


Logo centralised.


•  System shows kernel times are low

•  jstat -gccause shows STW GC is active at pauses

System Analysis



Smaller lettering.


Logo centralised.


Logo centralised.


•  GREP / SED / AWK

•  Analysers

•  HP JMeter

•  GCViewer (https://github.com/chewiebug)

•  jClarity Censum

Garbage Collection Analysis



Smaller lettering.


Logo centralised.


Logo centralised.


X Full GC � Minor GC



Smaller lettering.


Logo centralised.


Logo centralised.




Smaller lettering.


Logo centralised.


Logo centralised.

Purpose: Events, stall headers, handouts.X Full GC � Minor GC

•  Use the latest supported Oracle JVM

•  Understand & Size the Generations

•  Use appropriate collectors

•  Use NUMA & TLAB, if possible

•  LargePages/HugePages stop swapping

•  Except when they don’t!

Findings



Smaller lettering.


Logo centralised.


Logo centralised.


•  Upgrade OS to gain NUMA support

•  Repeat testing to measure performance change

•  Roll out one thing at a time

Next Steps



Smaller lettering.


Logo centralised.


Logo centralised.


Problem Statement

“It doesn’t respond”

•  Stress is Stressful

•  Read: why zebra’s don’t get ulcers

•  Actions Taken Prematurely

•  “Doing something is better than nothing”

Anti-patterns



Smaller lettering.


Logo centralised.


Logo centralised.


•  Conclusions drawn from partial data

•  Changes implemented rashly

Shot in the Dark



Smaller lettering.


Logo centralised.


Logo centralised.


•  OutOfMemoryError seen in log

•  System is GCing heavily

•  It needs more memory!

•  Action: double the JVM heap

•  Outcome: no change - problem recurred as before

Knee Jerk Analysis



Smaller lettering.


Logo centralised.


Logo centralised.


•  System had been stable for 2 weeks

•  Usage Profile had not varied significantly

•  All HTTP threads waiting on the database pool

•  All above threads had a common code path

What was missed?



Smaller lettering.


Logo centralised.


Logo centralised.


•  Database connection pool monitoring

•  DBCP

•  C3P0

•  Long running queries will tie up connections

•  Unreturned / Unused connections can be leaks

A Better Approach



Smaller lettering.


Logo centralised.


Logo centralised.


Database Connection Pool



Smaller lettering.


Logo centralised.


Logo centralised.


•  Team Calendars had been upgraded

•  New JIRA integration increased database activity

•  System Requirements changed

•  Larger DB pool size needed

•  Pool increased from 30 to 250

•  We didn’t just “make it bigger”!

Results



Smaller lettering.


Logo centralised.


Logo centralised.


Dark Arts


J F D I



The Bigger Picture



Smaller lettering.


Logo centralised.


Logo centralised.


KEEP C.A.L.M. and

DELIVER SOFTWARE

•  Nobody is available

•  Time is too short

•  Downtime is costing money

•  You need an external opinion

•  You need a framework to work by

•  The problem is too big

•  You just need a helping hand

Sometimes …



Smaller lettering.


Logo centralised.


Logo centralised.


Sometimes …



Smaller lettering.


Logo centralised.


Logo centralised.

Purpose: Events, stall headers, handouts.Standard horizontal view logo.


Smaller lettering.


Logo centralised.


Logo centralised.


Sometimes …



Smaller lettering.


Logo centralised.


Logo centralised.


•  Ensure you’ve JFDI

•  Be Methodical

If you only remember 1 thing:



Smaller lettering.


Logo centralised.


Logo centralised.




•  Measure, Don’t Guess

If you only remember 2 things:



Smaller lettering.


Logo centralised.


Logo centralised.




•  Measure, Don’t Guess

Amongst the things to remember



Smaller lettering.


Logo centralised.


Logo centralised.


“If I had an hour to solve a problem, I’d spend 55 minutes

defining the problem, and 5 minutes thinking about the

solution”

Einstein said something like:



Smaller lettering.


Logo centralised.


Logo centralised.


Further Study



Smaller lettering.


Logo centralised.


Logo centralised.


If you knew all this and we bored you…

…or if you’re just really, really good…

We’re hiring in the USA, UK and Germany.

Visit www.adaptavist.com

One Last Thing



Smaller lettering.


Logo centralised.


Logo centralised.


Thank you!

Dan Hardiker • CTO • @adaptavist



Smaller lettering.


Logo centralised.


Logo centralised.


the (not so) dark art of atlassian performance tuning

Technology

javalogo centralispurpose

stall heade

schemeslogo centralispurpose

loglogo centralispurpose

managelogo centralispurpose

slow performance sucks

event websites

ex external websites