Download - Bartow 302 Java Perf Tuning
-
8/14/2019 Bartow 302 Java Perf Tuning
1/27
-
8/14/2019 Bartow 302 Java Perf Tuning
2/27
About Dan Responsible for Turbotax Online performance and New
Business Initiatives
Previously responsible for turbotax.com, intuit.com,quickbooks.com, quicken.com, back-end order systems,content services, and many others
The Intuit eStore cluster is the largest stateful JBoss cluster in the world
Best Buy, American Airlines, Sony Online Entertainment, MTV, Rubbermaid, Oprah, Plantronics, JCrew,Neiman Marcus, FAA, US Army, Kodak, American Eagle Outfitters, Cabelas, Finish Line, Lexmark, Alcatel,Deutsche Post, B&Q, Abbott Labs, United Health Group, Sun Microsystems, GlaxoSmithKline, AT&T Wireless,Yellowpages, and many others
-
8/14/2019 Bartow 302 Java Perf Tuning
3/27
Agenda
Intros and Expectations Architecture Components - Web Applications Observing and Tuning
Targeting each layer of the architecture Discussion
-
8/14/2019 Bartow 302 Java Perf Tuning
4/27
-
8/14/2019 Bartow 302 Java Perf Tuning
5/27
Architecture ComponentsWeb Applications
Network Infrastructure
Operating System
Java
Database
Your Application
-
8/14/2019 Bartow 302 Java Perf Tuning
6/27
Overview: 3 Tier Architecture
Load Balancer
Web Server 1 Web Server 2
Database
App Server 1 App Server 2
F5 BigIP/Cisco CSS
Apache
Tomcat/JBoss/Websphere
Oracle/SQLServer/MySQL
-
8/14/2019 Bartow 302 Java Perf Tuning
7/27
Observing and Tuning:Performance Analysis
Load (Real or Virtual)
What are your performance targets?How do you dene them?
Your Application/System
Concurrent users
Transactions/second
-
8/14/2019 Bartow 302 Java Perf Tuning
8/27
Observing and Tuning:A Capacity Model
-
8/14/2019 Bartow 302 Java Perf Tuning
9/27
Observing and Tuning:Performance Analysis
A number of things can happen on the way to your targets...or in every day operation
Response time gets slower
Response time is ne and then it
stops responding
Response time is ne, but hasintermittent bursts of slowness
Load (Real or Virtual)
Your Application/System
-
8/14/2019 Bartow 302 Java Perf Tuning
10/27
Observing and Tuning:Tools of the Trade
GCViewer - Free - Garbage Collection Analyzer
Some things youll need along the way
Samurai - Free - Thread Dump Analyzer
A Java Application Proler
Good sys admins :)
Good network engineers :)
-
8/14/2019 Bartow 302 Java Perf Tuning
11/27
-
8/14/2019 Bartow 302 Java Perf Tuning
12/27
Observing and Tuning:Web Server - Apache
Threading Model
Features and Focus Areas
Content gzip Compression
SSL CompressionCPU/Memory
Concurrent Threads in Use
Bound by:
OS Level Network ContentionOS Level Network Contention
-
8/14/2019 Bartow 302 Java Perf Tuning
13/27
Observing and Tuning:Web Server - Apache
PreFork
Worker
Threading Models PreFork forks off an apache process forevery request. This is the original modeland works great.
Worker model is multithreaded model. Itsmuch easier on memory because there areless processes in use.
The really important question is: Do you have enough threads?
-
8/14/2019 Bartow 302 Java Perf Tuning
14/27
Observing and Tuning:Web Server - Apache
Apache mod_status
A simple way to nd out how manythreads are in use!
-
8/14/2019 Bartow 302 Java Perf Tuning
15/27
Observing and Tuning:Operating System
CPU
Memory
Bound by:
TCP Stack
Cong (File Descriptors)
-
8/14/2019 Bartow 302 Java Perf Tuning
16/27
Observing and Tuning:Operating System
Theres a HARD and SOFT limit on le descriptors
To nd out the HARD: ulimit -H
This is a system wide maximum limit that cannever be exceeded by a user or process
Changing this is a root level cong changerequiring a system reboot
File Descriptors: In Unix based operating systems, a network socket is actually just aspecial type of a le on the lesystem
To nd out the SOFT limit: ulimit -n
This is the limit of le descriptors a particular shell canhave available to it
This can be changed without a reboot but you will needto put it in a shell script or prole to get set every time
the shell is started
-
8/14/2019 Bartow 302 Java Perf Tuning
17/27
Observing and Tuning:Operating System
Its the set of congs and behaviors that your OS uses for network communication
TCP stack?? - what the heck is that?
tcp_time_wait_interval
Default = 240,000ms/4 MinutesRecommended = 60,000ms/1 Minute
(sometimes lower)
http://www.sean.de/Solaris/soltune.htmltcp_conn_req_max_q
tcp_conn_req_max_q0
Default = 1024 and 128
http://www.sean.de/Solaris/soltune.htmlhttp://www.sean.de/Solaris/soltune.html -
8/14/2019 Bartow 302 Java Perf Tuning
18/27
Observing and Tuning:Application Server
ThreadHandler Pool - Each thread handles a page request (JSP)
Defaults are often too low or too high - JBoss = 100, WebSphere = 20Find out how many are in use with the web admin browsers (JMX-Console...)
-
8/14/2019 Bartow 302 Java Perf Tuning
19/27
Observing and Tuning:Java
Virtual Machine Tuning
-
8/14/2019 Bartow 302 Java Perf Tuning
20/27
-
8/14/2019 Bartow 302 Java Perf Tuning
21/27
Observing and Tuning:Java
Garbage Collection
A few different ways to clean up your garbage
Single threaded (default - yikes)
Parallel
Concurrent Mark Sweep
The NEW and OLD generations use different collectors - set independently
-
8/14/2019 Bartow 302 Java Perf Tuning
22/27
Observing and Tuning:Java
GCViewer!
Add these JVM startup args:
-verbose:gc-XX:+PrintGCDetails-xloggc:/tmp/myGCFile.log
First - how do I see what my heap is doing under load?
-
8/14/2019 Bartow 302 Java Perf Tuning
23/27
Observing and Tuning:Java
Single or Parallel Threaded Collection
Full GCs get closer and closertogether as load increases until
throughput starts to drop
-
8/14/2019 Bartow 302 Java Perf Tuning
24/27
Observing and Tuning:Java
ConcurrentMarkSweepCollector
Heap usage stays in a tightband between 30-60%
-
8/14/2019 Bartow 302 Java Perf Tuning
25/27
Observing and Tuning:Java
A Java Thread Dump - Powerful!
kill -QUIT
-
8/14/2019 Bartow 302 Java Perf Tuning
26/27
Discussion - Q&A
Real performance problems?
Architecture challenges?
Organizational issues?
-
8/14/2019 Bartow 302 Java Perf Tuning
27/27
Thank you!For more information:
Dan BartowSr. Manager - Performance Engineering
Cons umer Groupdan_b [email protected] [email protected]
www.turbotax.com
http://www.turbotax.com/mailto:[email protected]:[email protected]://www.turbotax.com/http://www.turbotax.com/mailto:[email protected]:[email protected]:[email protected]:[email protected]