08 j2ee perf best practices

08_J2EE_Perf_BestPractices.ppt of 58

J2EE Performance Best Practices


WebSphere Application ServerKey Tuning Areas

• Web site applications

• Threading Issues

• EJBs

• JVM

• HttpSessions

• Connection pools

• External Performance Issues

• Dynacache


Web Site ApplicationApps and Performance

• Application tuning often yields the largest performance gains

• Code profilers help you reduce path length

• Other techniques to resolve runtime performance issues

• Thread dumps

• Java heap profiles

• HttpSession sizing


Web Site ApplicationCommon Pitfalls

• Synchronization

• Mishandling of pooled objects

• Indefinite delay

• Multi-threading problems

• JSP Issues

• Common EJB Issues


Web Site ApplicationSynchronization

• Synchronization is required in some cases• Accessing shared resources (ex: threadpools)

• Problems arise when• Overused to compensate for weak concurrency skills• Synchronized code block is too large• Synchronized code block improperly coded

• “Indefinite wait” inside the block• Permits starvation

• Globally search Web site applications for synchronization• Understand and authorize each use


Web Site ApplicationThread Pools

• Thread pools a good idea for:• Avoiding thread creation overhead• Managing number of threads in the system

• Rampant thread creation may crash a server

• Evaluate use of threads carefully• Would the app run more safely synchronously?• Are failures handled properly?

• Consider async beans in WAS EE• TOC for customer may be less in the long run


Web Site ApplicationProper Thread Pool Management

• Obtaining and Returning pooled threads must be synchronized• Keep sync’d code extremely small

• Min/Max thread pool size recommended• “Grow-by” parameter also desirable• Place parameters in an external source for tuning

• XML file, resource bundle• Read at initialization• Fancy pools provide servlet admin interface

• DO NOT WAIT INSIDE THE POOL!• For example, if the pool needs to add new connections• Better to throw soft failure and retry


Web Site ApplicationPooled Object Etiquette

• Return pooled resources quickly!• Do not make the return part of cleanup process• After finished, return the object• Relieves requester queue at peak times

• Cleanup the resource appropriately• close() the database statement• Do not return running threads to the threadpool


Web Site ApplicationIndefinite Delay

• Avoid indefinite waits inside the web application

• True for • Database requests• Launched threads• Remote requests of any kind (HTTP sockets, etc.)

• Bound wait with timeouts• Threads: join(100) instead of join()• JDBC calls: stmt.setQueryTimeout() if supported

• Place all timeout values in an external properties file• Essential for site tuning later


Web Site ApplicationIndefinite Delay Exceptions

• What if the servlet really needed the data?

• If the request times out, return a specific exception• Ex: “Non_Responsive_Exception”

• Code servlet to handle this error path• Select appropriate error JSP or similar action

• Better to return empty-handed than stall the site

• Prepare applications for eventuality of a remote outage!


Web Site ApplicationOther Notable Performance Issues

• Avoid excessive logging• Avoid servlet.log() • Build log Strings after confirming logging is enabled

• Avoid bean.instantiate() • Initiates disk I/O to check for bean contents• new the bean instead

• Avoid “+=“ to build Strings• Use JSPs; don’t build HTML in servlets

• Use J2EE datasources whenever possible


Web Site ApplicationOther notable performance issues

Turn off auto reloading for production and test systems

Avoid the “SingleThreadModel” for servlets

Cache sharable values in instance variablesExample: Datsources, EJB Home InterfacesObtain within init() method; faster than built-in caching


Web Site ApplicationJSP Issues

• Avoid creating unnecessary HTTPSession objects• <%@ page session=“false”%>

• Other servlet best practices apply to JSP scriptlets

• Keep scriptlet use to a minimum• Taglibs where appropriate• Better for maintainability

• Also:• Consider changing JSP reload intervals


Web Site ApplicationCanonical Servlet

• Avoid canonical servlets• One large servlet controlling the entire website

• For each servlet• Handle logic to select between 2 or 3 JSPs• Pull together dynamic data to feed the JSP• Handle common error conditions

• Also avoid• Fine-grained servlet modeling (opposite problem)• JSP-only solution

• Difficult to maintain dispersed logic


Web Site ApplicationFlow Examples

Servlet StockQuote.jsp

SymbolResult.jsp

NotFound.jspUnKnownError.jsp

Servlet

StockQuote.jsp

SymbolResult.jsp

MyAccount.jsp

TradeHistory.jsp

MyStatement.jsp

Errors.jsp

• Small servlets control a few pages• Work with group of servlets• Easier to extend website

• Add servlets

• Large servlet controls many pages• No logical relationship between pages

• Code path is very long• Performance issue

• Difficult to extend site• Must carefully modify huge servlet





• EJBs

• JVM

• HTTPSessions



• Dynacache


ThreadsHow Many?

• Each container controls its own thread pool• Default size is 10 threads minimum, 50 maximum• Only WAS interacts with these threads

• Threads launched within servlets do not use this pool!

• More is often less• More threads means more overhead• JVM time wasted on context switching

• Avoid unbounded growth• EJB container settings?


ThreadsFunnel Queue

• The application server contains a series of pools• Each pool contains a wait queue• Queuing reaches from the HTTP Server to backend DBs

• An example of a 150 > 20 > 10 Funnel Queue

150Listeners Q

ueue

20Threads Q

ueue

10Connections

HTTP Server Application Server Datasource


ThreadsFunnel Queue (cont’d)

• Processing at each level reduces pool demand downstream• Example: Some HTTP listeners return static content• Example: Servlets require significant String processing

• Also expedient to have some demand queued for quick entry

150Listeners Q

ueue

20Threads Q

ueue

10Connections



ThreadsFrozen Web site

• Threads may stall if a remote resource goes down• Downstream queues begin to fill up

• Servlet engine, HTTP Server• Eventually Web site cannot respond to static HTML requests

150Listeners Q

ueue

20Threads Q

ueue

10Connections






• EJBs

• JVM

• HttpSessions



• Dynacache


EJB 2.0 Performance668

625536

0

100

200

300

400

500

600

700

Thro

ughp

ut-R

eque

st/se

c

Local Interfaces

EJB 2.0 Local InterfaceCall-by-ReferenceEJB 2.0 Remote

SServlet

EJBsession

EJBentity

982 975

2687 2675

0

500

1000

1500

2000

2500

3000

EJB Caching Options

Option 'C'Option 'B'Option 'A'Life-time-in-Cache


EJBsUse a Façade Bean

• Avoid calling multiple Entity Beans within a servlet• Txn boundary created at each call for remote calls• Performance expensive

• Use a façade bean instead• Stateless session bean interacts with multiple EJBs• Local txn boundary spans all calls

• Within the Stateless session bean• Assumes SSB and Entity Beans in same container

• Return data as a regular databean to calling servlet

• Performance and architectural benefits

• Co-locate EJB and Web containers for best performance


EJBsGranularity

• Avoid extremely fine-grained EJB models

• Potential to create dozens/hundreds of objects on each call

• Creates issues with JVM memory, container sizing, etc.

• Better to use a more granular model• Create a few objects per call

• Reduces burden on memory, CPU

• More maintainable architecture


EJBsCache Sizing

• Container cache holds bean in use by the application

• Sizing the cache important for best performance

• Tuning guide recommendations• Entity Beans per txn * expected concurrent txns• Add active session bean instances

• Remember, EJBs sometimes long-lived• Entity bean Option A, Lifetime-in-cache caching• Stateful Session beans• Stateless Session bean pool


EJBsPassivation

• Passivation writes bean contents to hard disk

• Hard disk I/O very expensive• Inefficient at high volumes

• Size the bean container cache properly

• Remove Stateful Session beans as soon as possible• ejb.remove() method





• EJBs

• JVM

• HttpSessions



• Dynacache


JVMJVMs and Performance

• JVM sizing important for performance• Size vs. Garbage Collection time tradeoffs

• Application quality impacts JVM efficiency

• Key JVM issues• Heap size• Garbage collection• Memory leaks• Memory usage


JVMHeap Size

• Moderate heap sizes work best• -Xmx=512M for WAS• -Xmx=768M for WP• 1 GB heaps supportable on most multiprocessor boxes

• Minimum heapsize• Production systems set –Xms lower than –Xmx

• Gives headroom for emergencies• In theory, results in more efficient object table

• “Burst” testing requires –Xms=-Xmx

• Larger heaps feasible on extremely fast machines• Also indicates application inefficiencies


JVMGarbage Collection

• The larger the heap, the longer the GC cycle• Mitigates benefits of a large heap

• JDK 1.3.1 GC• Interim gc’s free memory (multi-threaded)• Compaction cycle for defragmentation

• Single-thread; stops application work

• Frequent GC caused by:• Inefficiencies in the application

• Excessive object creation (ex: Strings)• Sometimes occurs at –Xms threshold


JVMMemory Leaks

• Java applications may still “leak”• GC only picks up objects without references

• Does the heapsize continue to grow under steady load?• “Sawtooth” gc pattern has irregular shape• GC cycles become more frequent• “Java out of memory” errors in logs

• Some common “leak” culprits• HttpSession timeout is too long• HttpSessions contain extremely large objects• Servlets caching objects in another variable


JVMMemory Leak Pattern





• EJBs

• JVM

• HttpSession



• Dynacache


HttpSessionCommon Performance Pitfalls

• HttpSession misused by many Web applications• Impacts performance, scalability, and failover

• Common HttpSession pitfalls:

• Large HttpSession objects

• Poor timeout strategies

• Poor persistence preparation


HttpSessionLarge HttpSession Objects

• HttpSession objects live in a memory cache

• Cached shared by every user visiting the server

• Optimal HttpSession size is a few KB or less (approx. 2KB)

• Failure to keep session size small results in:

• Memory overhead

• Application “leaks” under large user volumes

• Poor session persistence performance


HttpSessionPoor Timeout Strategies

• Default HttpSession timeout is 30 minutes• No end user activity on the site for 30 minutes

• Depending on the site, this time maybe reduced• Frequent, short visits with little reading time on pages• Avoid setting too low

• Reaper thread impacts performance

• Give the user the opportunity to logout• Destroy the HttpSession on logout• Do not rely on the customer logging out

• Most do not• Keep this in mind in your testcases as well


HttpSessionPoor Persistence Preparation

• Persistence Session Management transparent to applications• No special APIs required• But the code may need special prep for persistence

• Session contents should be• Small for easy transfer and retrieval• Serializable

• No thread handles, binaries, etc.• Meaningful on failover• Broken into smaller object graphs

• Do not use persistent storage as an extended cache• Working set of HttpSessions should always be in memory


HttpSessionPersistence (cont’d)

• Large HttpSessions problematic for persistence performance• Longer network transfer time• More serialization overhead• More database storage required

• See InfoCenter for large session storage strategies

• Best strategy is reducing the size of the session data

• Persistent Session Management required for• HttpSession failover• Affinity routing in the 3.5.x product only





• EJBs

• JVM

• HttpSessions



• Dynacache


DatasourceConnection Pool

• Datasource manages the connection pool

• Most servlets spend most of their time munching data• Datasource pool sized much smaller than web thread pool• “Funnel” approach

• Keep in mind other users of the datasource• EJB containers, other servlet containers

• Monitor connection usage via Tivoli Performance Viewer

• Also monitor via database usage statistics


DatasourceExcessive Pool Utilization

• Web containers usually require few datasource connections• If you see significantly higher utilization• If you see “waiters” for database connections• You may have database issues

• Before increasing the connection pool• Check the query response time • Get the DBA to check for excessive table scans

• Database may need tuning for Web application

• Pool overutilization usually points to a slow database• Also may point to network between app server and db





• EJBs

• JVM

• HTTPSessions



• Dynacache


External Performance Issues

• Tuning doesn’t stop with WebSphere

• Remember systems outside of WAS• Http Server

• Remote content providers

• Infrastructure

• Server OS

• Network


Http Server• Http Server needs listeners to serve

• Static HTML, gif, jpegs, etc.• Dynamic page requests• Site requires sufficient listeners for these requests

• Listeners are CPU intensive• Drives up the “System” CPU utilization on Unix• Tailor listeners to your system’s needs

• Too few listeners results in “failed to connect” messages• Each user browser may make up to 5 simultaneous connections

• SSL more CPU intensive• Consider HTTP Server on separate machine; SSL accelerator


Http ServerIHS Key Parms for Unix Systems

• MaxClients• Maximum number of listeners Http Server will start• 150 – 250 works well for most sites

• StartServers• Listeners started at Http Server startup

• MaxSpareServers• Maximum unused listeners Http Server will allow to live• Set to MaxClients when in doubt

• MinSpareServers• Http Server tries to keep this many unused listeners alive• Set to StartServers when in doubt (will not exceed MaxClients)


Http ServerOther IHS Unix Parms

• HostnameLookups• Http Server can do a DNS lookup on incoming address• Extremely time intensive – do not use!• Recommend: HostnameLookups off

• LogLevel• Keep logging at a minimum to reduce overhead• Do not insert tags or processing during runtime

• Use post-processing tool• Recommend: LogLevel error (or lower)

• KeepAlive• Useful for sites with lots of static content• Maintains connection for extended time to reduce setup overhead• KeepAliveTimeout should be set reasonably low


Http ServerWindows

• IHS works differently on Windows platforms• “Threads per child”

• IIS more commonly used• Very different operation model• Key parms

• “Number of Hits Expected per Day”• Set to “More than 100000”

• “Listen Backlog”• A waiting queue inside IIS• Sometimes results in dropped connections in NT• Can go as high as 200

• See WAS tuning guide• Latest parms• Recommended settings


Http ServerLinux Considerations

• IHS Listeners

• Sometimes hang after handling large number of requests

• Require recycling for Linux

• Set the MaxRequestsPerChild parameter in httpd.conf

• Disabled by default (“0” value)

• Try 5,000 instead

• Set high for other Unix platforms

• Listener recycling not required for these platforms


Databases

• Databases often tuned for other types of applications

• Require new indexes to support Web applications

• Tuning for increased request volumes

• Memory buffers

• Maximum connections

• Sum of datasource connection pool max sizes

• May result in licensing issue

• Consider hardware changes/upgrades


DatabasesHardware

• Databases• Tend to be I/O rather than CPU bound• CPU I/O Wait time higher than WAS systems• Will existing platform support increased traffic?

• Improve performance by speeding up DB I/O• Add disk platters to reduce disk latency• Database “striping” across the platters

• Logs, data• DB2 “native” file writing

• Bypasses OS file system• Use where appropriate

• Disk buffering at the hardware level


DatabasesPrepared Statement Cache

• Whenever possible, use prepared statements• Dynamic SQL calls• Prepared Statement calls

• Configure WAS prepared statement cache• PrepStmtCache > diffPrepStmts * min(concurrent user threads, connectionPoolSize)

• See documentation for how to set various WAS versions

• 20% performance savings in some cases

• Monitor via Tivoli Performance Viewer and adjust as needed• If “cache ejections” value > 0, increase cache size


ServerOS Settings

• Check settings for • TCP/IP

• Solaris boxes always require TCP/IP adjustment• System file handles

• Refer to the Performance Redbook for details• Settings are OS specific


ServerWhen to Clone?

• Large n-way server with plenty of memory

• Good response time and relative throughput• Do not add clones if infrastructure under strain

• Example: High Database utilization• Example: Network bandwidth full utilized

• One JVM doesn’t drive the CPU to saturation

• One JVM has insufficient memory to support the site• Example: Large HttpSessions

• Best determined by experimentation• Add clone, add load, measure results• Throughput increases, CPU utilization increases• Response time remains constant


Network

• Easy to clog, hard to monitor

• Best to plan network capacity up front• Validate plan with actual page sizes

• Use network protocol analyzers, file transfers to monitor

• Much more on network capacity planning coming up





• EJBs

• JVM

• HttpSessions



• Dynacache


Dynacache• WAS 4.x/5.x feature

• Allows administrator to define cacheable dynamic content• Cache

• JSP/servlet ouput• Commands• WebServices

• Also flush cache to refresh, as required

• Cache resides inside WAS or • Pushed out to EdgeServer

• Cache keys include:• URL only • URL + parameter values• URL + parameter values + session id• others


Summary

• Many potential tuning points in the WAS system• Application tuning usually produces best results• But not always

• Important thing is to find the bottlenecks in order of severity

• Your system is only as fast as the slowest component


Shameless Plug

ISBN: 0201844540

Available at fine eTailers,

Brick and Mortar stores,

and PubOrder (IBM)!

08 j2ee perf best practices

Documents