bartow 302 java perf tuning

8/14/2019 Bartow 302 Java Perf Tuning

1/27


2/27

About Dan Responsible for Turbotax Online performance and New

Business Initiatives

Previously responsible for turbotax.com, intuit.com,quickbooks.com, quicken.com, back-end order systems,content services, and many others

The Intuit eStore cluster is the largest stateful JBoss cluster in the world

Best Buy, American Airlines, Sony Online Entertainment, MTV, Rubbermaid, Oprah, Plantronics, JCrew,Neiman Marcus, FAA, US Army, Kodak, American Eagle Outfitters, Cabelas, Finish Line, Lexmark, Alcatel,Deutsche Post, B&Q, Abbott Labs, United Health Group, Sun Microsystems, GlaxoSmithKline, AT&T Wireless,Yellowpages, and many others


3/27

Agenda

Intros and Expectations Architecture Components - Web Applications Observing and Tuning

Targeting each layer of the architecture Discussion


4/27


5/27

Architecture ComponentsWeb Applications

Network Infrastructure

Operating System

Java

Database

Your Application


6/27

Overview: 3 Tier Architecture

Load Balancer

Web Server 1 Web Server 2

Database

App Server 1 App Server 2

F5 BigIP/Cisco CSS

Apache

Tomcat/JBoss/Websphere

Oracle/SQLServer/MySQL


7/27

Observing and Tuning:Performance Analysis

Load (Real or Virtual)

What are your performance targets?How do you dene them?

Your Application/System

Concurrent users

Transactions/second


8/27

Observing and Tuning:A Capacity Model


9/27

Observing and Tuning:Performance Analysis

A number of things can happen on the way to your targets...or in every day operation

Response time gets slower

Response time is ne and then it

stops responding

Response time is ne, but hasintermittent bursts of slowness

Load (Real or Virtual)

Your Application/System


10/27

Observing and Tuning:Tools of the Trade

GCViewer - Free - Garbage Collection Analyzer

Some things youll need along the way

Samurai - Free - Thread Dump Analyzer

A Java Application Proler

Good sys admins :)

Good network engineers :)


11/27


12/27

Observing and Tuning:Web Server - Apache

Threading Model

Features and Focus Areas

Content gzip Compression

SSL CompressionCPU/Memory

Concurrent Threads in Use

Bound by:

OS Level Network ContentionOS Level Network Contention


13/27


PreFork

Worker

Threading Models PreFork forks off an apache process forevery request. This is the original modeland works great.

Worker model is multithreaded model. Itsmuch easier on memory because there areless processes in use.

The really important question is: Do you have enough threads?


14/27


Apache mod_status

A simple way to nd out how manythreads are in use!


15/27

Observing and Tuning:Operating System

CPU

Memory

Bound by:

TCP Stack

Cong (File Descriptors)


16/27


Theres a HARD and SOFT limit on le descriptors

To nd out the HARD: ulimit -H

This is a system wide maximum limit that cannever be exceeded by a user or process

Changing this is a root level cong changerequiring a system reboot

File Descriptors: In Unix based operating systems, a network socket is actually just aspecial type of a le on the lesystem

To nd out the SOFT limit: ulimit -n

This is the limit of le descriptors a particular shell canhave available to it

This can be changed without a reboot but you will needto put it in a shell script or prole to get set every time

the shell is started


17/27


Its the set of congs and behaviors that your OS uses for network communication

TCP stack?? - what the heck is that?

tcp_time_wait_interval

Default = 240,000ms/4 MinutesRecommended = 60,000ms/1 Minute

(sometimes lower)

http://www.sean.de/Solaris/soltune.htmltcp_conn_req_max_q

tcp_conn_req_max_q0

Default = 1024 and 128
http://www.sean.de/Solaris/soltune.htmlhttp://www.sean.de/Solaris/soltune.html


18/27

Observing and Tuning:Application Server

ThreadHandler Pool - Each thread handles a page request (JSP)

Defaults are often too low or too high - JBoss = 100, WebSphere = 20Find out how many are in use with the web admin browsers (JMX-Console...)


19/27

Observing and Tuning:Java

Virtual Machine Tuning


20/27


21/27


Garbage Collection

A few different ways to clean up your garbage

Single threaded (default - yikes)

Parallel

Concurrent Mark Sweep

The NEW and OLD generations use different collectors - set independently


22/27


GCViewer!

Add these JVM startup args:

-verbose:gc-XX:+PrintGCDetails-xloggc:/tmp/myGCFile.log

First - how do I see what my heap is doing under load?


23/27


Single or Parallel Threaded Collection

Full GCs get closer and closertogether as load increases until

throughput starts to drop


24/27


ConcurrentMarkSweepCollector

Heap usage stays in a tightband between 30-60%


25/27


A Java Thread Dump - Powerful!

kill -QUIT


26/27

Discussion - Q&A

Real performance problems?

Architecture challenges?

Organizational issues?


27/27

Thank you!For more information:

Dan BartowSr. Manager - Performance Engineering

Cons umer Groupdan_b [email protected] [email protected]

www.turbotax.com
http://www.turbotax.com/mailto:[email protected]:[email protected]://www.turbotax.com/http://www.turbotax.com/mailto:[email protected]:[email protected]:[email protected]:[email protected]

bartow 302 java perf tuning

Documents