jb jeap5 performance tuning wp web

8/3/2019 JB JEAP5 Performance Tuning Wp Web

1/20

www.jb.

2 In search of hIgh-performance

applIcatIons

2 performance tunIng prIncIples

2 Why tune for performance?

2 Performance priorities

3 A word about performance benchmarks

5 JBoss eap 5.0 tunIng

5 Connection pooling

6 Thread pooling

8 Object and component pools

9 Logging

9 Caching

11 JBoss EAP performance tuning summary

11 Linux-specictuning:Largememorypages

12 Linux-specictuning:Tuningthevirtual

memory manager12 Database tuning update

12 Case study

17 applIcatIon performance tunIng:

a contInuous process

18 appendIx: usIng large-page memory

(lInux-specIfIc InstructIons)

Best practices forperformance tuningJBoss enterprIse

applIcatIon platform 5tIps and trIcks for optImIzIng your applIcatIons performance


2/20

2 www.jb.

in search of high-performance applications

Overall,performancetuningisaveryimportantpartofcreating,maintaining,anddeployingasuccessful

businessapplication.Whetheryouarebuildingcustomapplicationsordeployingcommercial,off-the-shelf

solutions,youwilllikelyneedtotunetheapplication,thedatabase,themiddleware,orallthree.Infact,75%

of performance issues originate with the application itself.

Whenorganizationsselectapplicationmiddleware,performanceisalwaysoneoftheirmostimportantselec-

tioncriteria,ifnotthemostimportant.InmanycasesourcustomerstellustheychoseJBossEnterprise

Application Platform (EAP) because of its superior performance. They know that many users of JBoss EAP

areachievingsuperiorapplicationperformancedayafterday.

Togetthemostfromyourcompanysinvestmentinmiddleware,developersandarchitectsneedtoknow

thespecicwaystheycanachievesuperiorperformancewithJBossEAP.Whilewewouldallliketothink

thatanapplicationcouldperformwellstraightoutofthebox,thisisnotusuallythecase.Applicationscanhavewidelyvaryingcharacteristics,andwhilesomeapplicationsmightperformwellwithdefault

middlewaresettings,otherswillnot.

IfyouarenewtoJBossEAPorperformancetuning,thispaperwillintroduceyoutobestpracticesthatcan

helpyouavoidcommonperformancepitfallsasyouprepareyourapplicationforproduction.Ifyoureanold

handatapplicationperformanceissues,youknowthattechnologyisconstantlychanging.Youmaybenet

from an update on best practices in JBoss EAP performance tuning.

performance tuning principles

Why tune for performance?

Performance was once considered just another feature of an application. Today it is frequently consid-eredthemostimportantcharacteristicoftheapplicationonethatcanhaveasignicantimpactonyour

businessandyourproductivity.Consideryourreactiontoaslowwebsite.Ifyourelikemostpeople,you

becomefrustrated,losepatience,andgoelsewhere.IfthatcompanyiscountingonrevenuefromWebsales,

ithasnotonlylostyourattentionithaslostbusiness.Evenforinternalapplications,poorperformance

canaffectproductivityifusershavetowaitordealwithunpredictablebehavior.Atbest,theymightfeel

annoyed,losealittletime,orformanegativeopinionoftheirITdepartments.Atworst,businesstrans-

actionsmaybelost,orcustomersmaygowithoutimportantserviceifusersmustworkaroundapoorly

performing application.

But user experience is not the only reason to tune for performance. A well-performing application will

generallyusefewerhardwareandsoftwareresources.Acompanycanoptimizeitsinvestmentinhardware

whenapplicationsaretunedappropriately,whetherthatmeansusingoldersystemslonger,purchasing

newsystemsthataremoremodestlysized,orusingfewersystemsoverall.Onthesoftwareside,awell-

performingapplicationwillgenerallyneedtousefewerCPUcountsorsoftwarelicenses,nomatterwhat

typeofsoftwareisinvolved.Reducingsoftwarecostscansavethecompanysignicantmoneyovertime.

performance prIorItIes

Superiorperformancecomesfrommanylayersoftheapplicationstack,notjusttheapplicationserver.In

manycases,thewaytheapplicationisdesignedandhowitconnectstothedatabaseandothersoftware

componentscanhavealargeimpactonoverallapplicationperformance.Manyorganizationsspendmore

BestpracticesforperformancetuningJBossEnterpriseApplicationPlatform5.0


3/20

www.jb. 3

timetuningtheircustom-builtapplicationsanddatabasesthantheirunderlyingapplicationservers.Sokeep

inmindthatasuperiorlyperformingapplicationservermayhaveonlyaminorimpactontheoverallperfor-mance of your applications.

a Word aBout performance Benchmarks

Some organizations rely on industry-standard performance benchmarks when selecting middleware. While

benchmarkscanhelpvendors,theycandeceiveyouasanevaluatorbecausebenchmarkapplicationsare

usuallyverydifferentfromtheapplicationsyouwillruninproduction.Thecurrentbenchmarkforapplica-

tionservers,forexample,SPECjAppServer2004,isa2004applicationthatdoesnttakeadvantageofmany

recentdevelopmentsinJava.Butasidefromthat,everyapplicationisdifferentandthesystemsonwhich

benchmarkapplicationsarerunmaybeverydifferentfromyours.

Understandinghowagivenbenchmarkrunsmaytellyouverylittleabouthowyourapplicationwillrunon

yourhardwarewithyoursettings.Beforeyourelyonabenchmark,youneedtoinvestigatethesoftwareand

hardwarecongurationsused,theserverandnetworkcongurations,thesettingsused,thearchitectureoftheapplication,andhowallofthesecomparewithyourownenvironment.Werecommendthatbefore

selectingasystem,youtestitwithanapplicationandhardwarecongurationasclosetoyoursaspossible.1

performance tunIng prIncIple #1:

understand your performance requIrements

Therststepintuningyourapplicationforperformanceistounderstandtheconditionsunderwhichitwill

needtoperform.Ifyourapplicationisareplacementforanexistingsolution,thenyourorganizationalready

hassignicantexperiencewiththatsolution.Chancesareyouhavemetricsavailablesuchasthenumberof

users,thenumberoftransactionsperday,thevariationsintransactionloadortypeoverthecourseofaday,

week,month,oryear,andsoon.

Ifyouaredeployingacompletelynewsolution,youwillneedtostudythatapplicationsbusinesscontext

verycarefully.Themoreyouunderstandexactlyhowyourapplicationwillbeused,themoresuccessfulyourperformancetuningwillbe.Sometimesassumptionsdonotreectreality.Inonecase,adevelopmentteam

createdandtesteditsapplicationbasedonanassumedworkloadof60,000transactionsperday.Therst

dayinproduction,sixmilliontransactionsoccurred.


plan for peaks, not averages

Asyouexamineperformancerequirements,oneofyourgoalsshouldbetodevelopaproleofyourapplica-

tionsworkloadwithspecialattentiontothepeaks.Forexample,manybusinessapplicationsexperiencedaily

peaksinthemorningandafternoonwithavalleyduringthemiddleofthedaywhenemployeesareeating

lunch.Someapplicationsexperiencepeaksattheendofthemonth,quarter,orseason.Yourapplications

workloadprolewilldependonthespecicsofyourbusiness,butyoushouldalwayspayparticularattention

toperiodsofpeakworkload.Oneofthebiggestmistakesdevelopersmakeistorelyonanaverage(average

dailyworkload,forexample).Averagesarenotsufcienttoensurethatyourapplicationwillperformduring

periods of peak load.

1 Foramorein-depthexplanationoftheprosandconsofperformancebenchmarks,seeDoPerformance

Benchmarks&ComparisonsMatter?AGuidetoAssessingApplicationServerPerformanceResults.



4/20

4 www.jb.


alWays Instrument your applIcatIon

Allapplicationsshouldbeinstrumentedtoprovideinformationforperformanceanalysis.Businesscondi-

tions,includingcustomerbehaviorandworkloadcurves,canchangedramaticallyovertime,soevenifan

applicationonceperformedwell,itmaynotperformwellundertodaysconditions.Ifyourunintotrouble,

andyourapplicationhasnotbeeninstrumentedforperformance,thenyouhavenoeasywaytoknowwhere

yourproblemsare.Butifyourapplicationisappropriatelyinstrumented,thenyoullbeabletomonitor

changes in business conditions easily and tune your applications to match before problems occur.

Inthepast,adeveloperneededtoembedcodewithintheapplicationitselfinordertoinstrumentanappli-

cation.Todaymanysolutionsprovideinformationwithoutrequiringdeveloperstocode.WithJBossEAP,

thecallstatisticsinthecontainerprovideyouthenumberofcalls,concurrentcalls,mincalltime,maxcall

time,averagecalltime,andthestandarddeviationofcalltimes.Hibernatestatisticsprovidequeryexecu-

tiontimes.ThenewJBossEAP5AdminConsole,aswellasandJBossOperationsNetwork(JBossON)

Monitoringcandisplayallofthisinformationandgraphtheresultsinrealtime.Additionally,numerousthird-partyapplicationperformancemanagement(APM)productsareavailablefromcertiedRedHat

partnerstoprovideapplicationperformanceinformationaswell.

Finally,forperformance-criticalsituationswhereoneoftheothertoolsdoesntprovidetheinformationyou

need,youcanwriteyourowninstrumentationusingtheJBossAOPframework.Thisframeworkoffersquite

a few features that enable you to see exactly what is happening at runtime. Whether you choose to write

yourowninstrumentationwilldependontheapplication,yourskillsandavailabletime,andtheimportance

oftheapplicationsperformancecharacteristicstoitsoverallsuccess.


understand Where your applIcatIon spends Its tIme

One important reason to instrument your application is to understand where time is being spent. While you

wanttoknowabouttimeacrosseachlayerofyourapplicationstack,thetoolswithinJBossEAPwillhelpyouunderstandonlypartofthisequation.Ifyourapplicationisspendingtoomuchtimeinthedatabase,

forexample,thenyoumayneedtofocusonyourdatabasestatisticstopindowntheproblem.

Byunderstandingwhereyourapplicationspendsitstime,youwillbeabletoavoidtheshotgunmethod

of performance tuning trying multiple solutions to common problems without knowing whether any of

themarerelevanttoyourproblem.Youmightsolveyourproblemthisway,buttheprobabilityislow.Many

developerswhoassumetheyknowwhatiscausingtheirperformanceproblemsaremistaken,resultingin

problemsthatpersistformonthsorsometimesyears.Thatiswhyhavinganobjectivewaytoknowwhere

yourapplicationisspendingtimewillhelpyouavoidspendingtoomuchtimesolvingperformanceproblems.



5/20

www.jb. 5

performance tunIng prIncIple #5

replIcate or model your productIon envIronment

Asyouworktounderstandexactlyhowyourapplicationwillperforminproduction,youwillneedtorunit

inatestenvironment.Somecompaniesmakeitstandardpracticetoimplementtestenvironmentsthatare

exactreplicasoftheirproductionenvironments.Thisisanidealapproachinmanyrespects,becausedevel-

opersdonthavetowonderhowtheirapplicationswillscaleforlargersystems.

Formanyorganizations,though,thecostofreplicatingtheproductionenvironmentisasignicant

obstacle.Theirtestenvironmentstypicallyemploysmallersystems.Ifthisisyoursituation,thenyouwill

needtocreateamodelthatdenestherelationshipbetweenperformanceinyourtestenvironmentand

performanceintheproductionenvironment.Thatis,youneedtomodelhowtheapplicationwillscale.As

youdoso,keepthefollowinginmind:

d not i ii. Performancedoesntoftenscalelinearly.Whilevendors

maytellyoutheyexperiencelinearresults,thisisrarelythecaseforrealproductionapplications.

B i i i , ii w ib. Ifyou

canbaseyourmodelondatafrompastexperiencewithotherapplications,youwillbeabletorene

itovertimeandincreaseyourcondenceinitsaccuracy.

JBoss eap 5 tuning

While75%ofallperformanceproblemsaretheresultoftheapplication,notthemiddlewareorthe

operatingsystem,youstillneedtoknowhowtotunesettingsinthemiddlewaretoimproveperformance

andthroughput.Dependingonyourapplication,oneormoreofthefollowingmayneedsomeattention:

Connection pooling

Thread pooling

Object and component pools

Logging

Caching

Thefollowingsectionsprovideanoverviewofeachoftheseareas.

connectIon poolIng

Connection pooling and thread pooling are the most important areas to consider when you want to maximize

throughputonmodernhardware.Intermsofsystemresources,databaseconnectionsareexpensiveto

setupandteardown.Butinspiteofthis,someapplicationscreateanewconnectiontothedatabasewith

everyqueryortransactionandthenclosethatconnectionimmediately.Thispracticeaddsagreatdealof

overheadtotransactionprocessingandcanleadtopoorperformance.

TotakeadvantageoftherobustconnectionpoolinginJBossEAP,startbyadjustingconnectionpoolsettings

onthedatasourcedenitionsthatyoucansetupinthedeploydirectory.Settheminimumpoolsizetothe

levelyouwanttotunefor,thensetthemaximumatleast25-30%higher.Dontbeconcernedaboutsetting

themaximumtoohigh,becauseifyoudontneedthatmanyconnections,thepoolwillshrinkautomatically.



6/20

6 www.jb.

Todeterminethepropersizing,youcanmonitoryourconnectionusage.Apoolthatistoosmallwillthrottle

theapplication,asJBossEAPwillqueuetherequestforadefaultof30,000milliseconds(or30seconds)beforegivingupandthrowinganexception.Ifyoustartseeingalotof30-secondtimeouts,thatisastrong

cluethatyouneedtolookatyourconnectionpooling.Youcanmonitortheconnectionpoolutilizationfrom

theEAPJMXconsole,fromJBossON,orfromdatabase-specictools.

Hereisanexampleofconnectionpoolsettingsforadatasource:

MySQLDS

jdbc:mysql://[host]:3306/[database]

com.mysql.jdbc.Driver

someuser

somepassword

org.jboss.resource.adapter.jdbc.vendor.MySQLExceptionSorter

75

100

TRANSACTION_READ_COMMITTED

100

true

thread poolIng

Thread pooling is the next most important area to consider as you tune your application for performance.

JBossEAPhasrobustthreadpooling,butbeforeyoucansizethethreadpoolsappropriately,youneedto

know how they are used and which ones might be affecting your applications performance. The character-

isticsofyourspecicapplicationwilldeterminewhichthreadpoolsareusedandwhichonesmightbecome

bottlenecks.Thiscanvarysignicantlyfromapplicationtoapplication.Thetablebelowprovidesasummary

of how each thread pool is used.



7/20

www.jb. 7

removIng connectors

Ifyouarecertainthataparticularconnectorwillnotbeusedinyourapplication,considerremovingit.

Forexample,manyapplicationsuseeitherthehttpdthreadpoolorthemod_jkthreadpoolbutnotboth.

Soyoucanremovetheoneyoudontneed.monItorIng and tunIng thread pools

MonitoryourthreadpoolsviatheEAP5AdminConsole,whichdisplaysnotonlythenumberofactive

threadsforeachpoolbutalsothequeuesize.InboththeEAP5AdminConsoleandJBossON,youcan

adjustanddenethreadpoolsettingsthatpersist.SettingsadjustedviatheJMXconsolewillnotsurvive

areboot.IfyouusetheJMXconsole,remembertogobackandedittheleifyouwanttheadjustments

to be permanent.

thread pool Where is it defined? hoW is it used?

System thread pool Injboss-service.xmlintheconfdirectory ForJNDInamingthedefaultsettingisneformost cases

HTTPdthreadpool

in JBoss Web

Intheserver.xmlleunder /deploy/

jboss-web-sar.

WhenmakingHTTPrequestsdirectlytoEAP

AJP thread pool Intheconnectorsectionofserver.xml WhenmakingHTTPrequeststhroughmod_jk

ormod_cluster

JCA thread pool (also called the

WorkManagerthreadpool)

Denedin/deploy/

jca-jboss-beans.xml

InconjunctionwithJMS,asJBossMessaging

usesJCAinowastheintegrationintoEAP

JBossMessagingthreadpool

(for remote clients)

Denedin/deploy/messaging/

remoting-bisocket-service.xml

Pools the TCP sockets

JBossMessagingthreadpool

(inJVMclients)

All the processing will occur on the JCA thread

pool(WorkManager)

Notethatifyouhaveamessagedrivenbean,

thatinvokesotherbeans,suchasstateless

sessionbeans,thosebeanswillalsorunonthe

JCA thread pool.

EJB3(sameJVM) ClientsinthesameJVMwillrunonwhatever

thread pool they are already using

Forexample,awebrequestcomesinthrough

theAJPconnector.WhenitcallsanEJB3bean,

it will continue executing on the AJP connector

thread pool.

EJB (remote clients) /ejb3/deployer/META-INF/

jboss-service.xml



8/20

8 www.jb.

example: http thread pool

example: mod_Jk or aJp thread pool

example: Jca thread pool

@org.jboss.aop.microcontainer.aspects.jmx.JMX(name=jboss.jca:

service=WorkManagerThreadPool, exposedInterface=org.jboss.util.threadpool.

BasicThreadPoolMBean.class)

WorkManager

1024

100

60000

oBJect and component pools

Object pools and component pools are essentially the same thing. Their settings represent the number of

objectinstances.ForEJB3,twotypesofpoolsaredenedin/deploy/ejb3-interceptors-

aop.xml.ThesearetheThreadLocalPoolandtheStrictMaxPool.Bydefault,StatelessSessionandStateful

SessionBeansusetheThreadLocalPool,whichisbackedbyanInnitePoolwithnomaximumsize.Therefore,

itgrowsaccordingtovolumeinyourapplication.Thishasthedistinctadvantageofnotneedingtobetuned.

Bydefault,MessageDrivenBeans(MDBs)usetheStrictMaxPool.Thispoolactuallyobeysamaximum,willqueueuprequestswhenthatmaximumhasbeenreached,andwilltimeoutanythinginthequeueifthereis

notanavailablereferencefromthepool.Inthiscase,thesystemwillthrowanexceptionandiftheproblem

occurredinmid-transaction,youwillexperienceatransactionrollback.Giventheimpactfailedtransactions

canhaveonyourbusiness,youshouldmonitortheStrictMaxPoolcloselyviatheJMXconsole.



9/20

www.jb. 9

loggIng

Developersshouldtakefulladvantageoflogginginthedevelopmentandtestingphasesoftheapplicationlifecycle.Inproduction,however,loggingcancausebottlenecks.Youwanttobesurethatloggingprovides

you with useful information without hurting application throughput. Consider making the following changes

asyoupromoteyourapplicationintoproduction:

t i i i. InJBossEAPsdefaultconguration,consoleloggingisenabled,

whichmeansyouhavetheopportunitytoseeallthelogsfromyourIDE.Inproduction,thisisanexpen-

siveprocesswithunbufferedI/O.Whilesomeapplicationsmaybenewithconsolelogging,high-volume

applicationsbenetfromturningitoff.JBossEAP5providesacongurationset,namedProduction,

whichgivesdevelopersabetterstartingpointforcreatingaproductionenvironment.IntheProduction

conguration,consoleloggingisturnedoff.

t w i bi. Thelessyoulog,thelessI/Owilloccur,andthebetteryouroverallappli-

cationthroughputwillbe.Loggingisalwaysatradeoff,sothinkcarefullyabouthowmuchloggingyou

really need in production.

u i. This can make a big difference for high-throughput applications. With

asynchronouslogging,logmessageswillgointoaqueueandcontrolreturnstotheapplicationasif

the logging had been completed. Then a separate thread executes the log operations from the queue.

W b wi If(debugEnabled()). This simple practice can make a huge differ-

enceifyourapplicationcontainsalotofdebuglogstatements.Withoutthisconditionset,yourapplica-

tioncreatesallofthestringobjectsforeachofthelogstatements,andLog4jcreatestheLoggingEvent

objectforeachlogstatementregardlessoftheloglevelthatissetbecausetheloglevelischeckedonly

afteralloftheseobjectshavebeencreated.Insomecasesthiscanleadtocreationofthousandsand

thousandsoftemporaryStringandLoggingEventobjects,resultinginmemoryandgarbagecollection

issues and reducing throughput dramatically. By placing a conditional wrapper around your debug log

statements,youcanensurethatunnecessarylogprocessingdoesnotaffectyourthroughput.

cachIng

Caching,whileoftenveryhelpful,maybeoneofthetrickiestareastotunecorrectly.JBossCacheisan

integralpartoftheJBossEAPinfrastructure,andyourapplicationcanuseittocacheanythingyoulike.

Cachingisespeciallyvaluableforapplicationsthatperformalotofreadoperationsagainstdatathatis

eithercompletelystaticordoesntchangefrequently,suchasreferencedata.Forapplicationswithheavy

useofwriteoperations,cachingmaysimplyaddoverheadwithoutprovidinganyrealbenet.Ifnotcong-

uredproperly,cachingmayactuallyreducethroughput.

OneoftheeasiestpotentialperformanceenhancementsyoucanmakeistocacheEJB3entities.Todene

whichentitiesyouwantcached,modifythelepersistence.xml that you deploy with your EJB 3 applica-

tion(anexampleisshownbelow)[email protected]

the@Cachedannotation,youspecifytheusageasbeingoneofthefollowing:

CacheConncurrencyStrategy.READ_ONLY, READ_WRITE, NONSTRICT_READ_WRITE,

or TRANSACTIONAL.



10/20

10 www.jb.

Youdenethecachesizeandevictionpolicyinthele

/deploy/cluster/jboss-cache-manager.sar/META-INF/jboss-

cache-manager-jboss-beans.xml.

Thesedenitionsmayrequiresometrialanderrortogetright.Rememberthatyouareworkingwithlimited

heapspace,andthatamisguidedcachingstrategycanbeworsethannoneatall.

Evictionpolicywilldependonthespecicsofyourapplication.Youcandenethecacheastransactional

orread-write.Withverylargedatasets,notethatcachingmaynotprovidenoticeableperformancebenets.

Inthiscase,shrinkingthecachescanimproveperformance.Also,ifyourapplicationisverywrite-heavy,it

maynotbenetfromcaching.Testingvariouscachingandnon-cachingcongurationswillhelpyoudeter-

mine this.

example: settIngs for cachIng In persIstence.xml

org.hibernate.ejb.HibernatePersistence

java:/MySQLDS

...


11/20

www.jb. 11

MySQLDS

jdbc:mysql://[host]:3306/[database]

com.mysql.jdbc.Driver

someuser

somepassword

org.jboss.resource.adapter.jdbc.

vendor.MySQLExceptionSorter

75

100

TRANSACTION_READ_COMMITTED

100

true

Asalways,itisrecommendedthatanalysisbeconductedwithinyourenvironmenttodeterminehowlarge

tomakethepoolsizeasthiswilldependonthepreparedstatementsthatarespecictoyourapplication

and data source.

JBoss eap performance tunIng summary

Tosummarize,keepthefollowingperformancerecommendationsinmindwhentuningJBossEAP5.0.

DenedatasourcesinthedeploydirectoryandtakeadvantageofEAPsrobustconnectionpooling.

Understandwhichthreadpoolsareactuallyusedbyyourapplication,removepoolsthatarenot

needed,monitorthenumberofactivethreadsandthequeuesize,andadjustpoolsizeifneeded.

Ifyouareusingmessage-drivenbeans,monitortheStrictMaxPoolcloselytoensurethatthemaximum

object pool size is not a bottleneck.

Employdifferentloggingstrategiesfordevelopmentandproduction.Especiallyforhigh-volumeappli-

cations,turnconsoleloggingoff,reduceloggingverbosity,useasynchronouslogging,andalwayswrap

debug log statements with If (debugEnabled ()).

Takeadvantageofcachingifyourapplicationisread-heavyormakessignicantuseofreferencedata.

Avoidcachingforverylargedatasetsorwrite-heavyapplications.

lInux-specIfIc tunIng: large memory pages2

Fortodays64-bitsystems,useoflargememorypagescanimproveperformancesignicantly.Thedefault

memorypagesizeistypically4kB.Whenyouareaddressinglargeamountsofmemory,thisquicklyadds

uptoalargenumberofmemorypagesjustonegigabyterequires262,144memorypages.Thatsalotforasystemtokeeptrackof,whichtranslatestoalotofsystemoverhead.

2 WhilethissectionprovidesLinux-specicsettings,theprinciplesoflargememorypageuseareapplicabletoany64-bitsystem.



12/20

12 www.jb.

Asidefromhelpingyouavoidtheoverheadofmappingsomanymemorypages,largememorypageson

Linuxcannotbeswappedtodisk.Thisisamajoradvantagebecausehavingyourheapspaceswaptodiskcanwreckhavocontheperformanceofyourapplication.

Largepagesupportbeginsat2MBandcanrunashighas256MBonsomehardwarearchitectures.These

numberswillvary,andyouwillneedtondoutthevaluesforyourspecicserver.AllthemajorJVM

systemssupportlargememorypagesonLinux.Becauseitcanbetrickytosetup,weprovidesomespecic

instructions in the appendix.

Whenyouuselarge-pagememory,keepinmindthatthememoryisnotavailabletoapplicationsingeneral.

Tootherapplications,yoursystemwilllookasthoughithashadmemoryremovedfromit,becausethis

memorywillbededicatedtoyourspecicapplication.Refertotheappendixformorespecicinformation

aboutconguringyourapplicationtouselargememorypages.

lInux-specIfIc tunIng: tunIng the vIrtual memory manager

InLinux,thevirtualmemorymanageristunable.Insomecases,youcanachieveperformancebenetsby

changingthesettings.Youwontbeusingthelesystemcachemuch,andyoudontwantthesystemto

favorthelesystemcacheoveryourapplications,whichcansometimesoccurwiththedefaultsettings.

In/etc/sysctl.conf you can set vm.swappinessto1topreventapplicationsfrombeingswappedto

disk when there is memory pressure.

dataBase tunIng update

Moderndatabases,especially64-bit,areextremelyefcientatcachingdata.Intheearlydaysof64-bit

databases,largebuffersizeswouldslowperformanceduetoelongatedsearchtimes,butthisisnolonger

true.RedHat'stesting,hasresultedinOLTPapplicationswithverylargebuffercachesthatshowvery

good results.

Consideryourapplicationscharacteristicsespeciallyitsread/writeratioswhendecidingwhetherandhowtousedatabasecaching.Themajorityofapplicationsweseetodayareread-intensive,asmostapplications

drivetheirbusinesslogicbyreadingdatafromthedatabase.Themoreread-intensivetheapplicationis,the

morecachingcanhelp.Iftheapplicationiswrite-heavy,orthedatasetisverylarge(fromadatawarehouse,

forexample),thenalargecachewonthelpandmayslowdowntheapplication.

YoushoulduseDirectI/Oifyourdatabasesupportsit.Especiallywithalargecacheorawrite-intensive

workload,youshouldavoiddoublebufferingwiththelesystembuffercache.NotethatMySQL5documen-

tationindicatesthatqueriesmayslowdownbyuptoafactorofthreewhenusingDIRECTI/O,butRedHat

hasnotexperiencedthisincaseswhereaproperlysizedbufferpoolisused.IfyouareusingDIRECTI/O,you

shouldbesuretousethevirtualmemorysettingmentionedearlier.ConsiderusingasynchronousI/Oifyour

database supports it.

case study

TodemonstratethedramaticresultsthatcanbeachievedwithJBossEAP5performancetuning,

an experiment was performed using a sample application.

ItbeganwithJBossEAPsdefaultconguration,alongwithallLinuxparametersattheirdefaults;RedHat

EnterpriseLinuxwasusedforalltests.UsingGrinder,anopensourceJavaLoadTestingFramework,the

highestthroughputthatcouldbeachievedwithallofthesesettingswasmeasured.Usingthesameappli-

cation,manyoftheoptimizationscoveredinthisguidewereapplied.Allotherhardware,database,and

networkcongurationsremainedthesamethroughouteachtestscenario.



13/20

www.jb. 13

Virtual users

1 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90

80

70

60

50

40

30

20

10

0

Transactionsp

ersecond

JBoss EAP 5 GA ResultsMean, no optimizations

figure 1a

ThesampleapplicationwasanEJB3applicationwithtwoservletsfortheUI,statelessandstatefulsession

beansformostofthebusinesslogic,amessagedrivenbeanforsomeasynchronousprocessing,andentitiesforthepersistence.Fortheinitial,non-optimizedtests,defaultJBossEAPcongurationswereused(thread

poolof64,defaultheapof1.3GB).

Fortheoptimizedtestruns,congurationsweretunedspecicallywithscalinginmind;heapsizewas

adjustedto3.7GBwithlargepagememory,thethreadpoolwasincreasedupto150,andthedatabase

connectionpoolwasalsoat150.Additionally,thefollowingJVMoptimizationswereaddedtothecongura-

tion:-XX:+AggressiveOptsand+XX:DoEscapeAnalysis.

Performanceresultsforbothscenariosareshownbelowinthefollowingsetofgraphs.Therstsetof

results measures scalability using mean transactions per second (TPS). The initial test run was conducted

withdefault,non-optimizedcongurations(Figure1A).Thesecondtestrunusedthesameapplication,but

leveragedtheperformanceoptimizationshighlightedpreviously(Figure1B).



14/20

14 www.jb.

Virtual users

JBoss EAP 5 GA ResultsMean, optimized

95 100 105 110 115 120 125 130 135 140 145 150 155 160 165 170

140

120

100

80

60

40

20

0

Transactions

persecond

figure 1B

Notethedramaticimprovementsbetweeneachscenario.Therst,non-optimizedtestscaledupto90

virtualusers,achievingmorethan75,000peakTPS.ByoptimizingtheJBossEAPcongurations,we

wereabletoalmostdoubletheamountofvirtualusersaddedto170andmorethandoubledthepeak

amountoftransactionsatoverthan140,000TPS.

Thenextsetofgraphsfocusonapplicationresponsivenessunderload.Withoutoptimizations(gure2A),

theapplicationpeakedat90virtualuserswitha300millisecondaveragerequestresponse.



15/20

www.jb. 15

1 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90

300

250

200

150

100

50

0

Millisecond

responsetime

Virtual users


figure 2a



16/20

16 www.jb.

Virtual users


95 100 105 110 115 120 125 130 135 140 145 150 155 160 165 170

300

250

200

150

100

50

0

Millisecondresponsetime

figure 2B

Withoptimizations(gure2B),whenscalingoutto170virtualusers,meanapplicationresponsiveness

waslessthan300millisecondsperrequest,anoticeableimprovementoverthenon-optimizedscenario.



17/20

www.jb. 17

application performance tuning:a continuous process

Tuninganapplicationforoptimalperformancecanensureapositiveuserexperience,promotebusiness

productivity,andhelptooptimizeuseofhardwareandsoftwareresources.Performancetuningisnota

one-timetask,butanongoingprocessthatensuresawell-performingapplicationasbusinessconditionsand

systemtechnologychangeovertime.Applicationdevelopersandarchitectsshouldalwaysbepreparedto

tunetheirapplicationsforperformancebothbeforeandaftertheyareputintoproduction.Asalways,the

morebusiness-criticalanapplicationisandthehigherthevolumeoftransactionsitmustsupport,themore

important performance tuning will be to your business.

Whentunedinaccordancewiththecharacteristicsofyourapplication,JBossEAPcanprovidesuperior

applicationperformance.Thispaperhasgivenyouanoverviewofbasicperformanceprinciplesaswellas

an introduction to performance tuning best practices for JBoss EAP. Keep in mind that these techniquesapplytoanyoftheRedHatplatformsthatJBossEAPsupports:JBossEnterprisePortalPlatform,JBoss

EnterpriseSOAPlatform,andJBossEnterpriseBRMS.

ForadditionalinformationonJBossEAPperformancetuning,pleasereferto:

Managingapplicationperformance:JBossOperationsNetwork

JBossEAP5AdministrationandCongurationGuide

JBossEAPOptimizationandTuningAssistanceServices

JBossEAP5ProductDocumentation



18/20

18 www.jb.

appendix: using large-page memory(linux-specific instructions)

Thisappendixcontainsaprocedureandexampledesignedtohelpyoutakeadvantageoflarge-pagememory

foryourapplications.SeethesectionLinux-specictuning:Largememorypagesonpage11.

1. tell the Jvm to use large-page memory.

SunJVMandOpenJDKrequirethefollowingoption,passedonthecommandline,touselargepages:

-XX:+UseLargePages

TheSuninstructionsleaveitatthat,butifyoudonothingelseyouwillmostlikelygetthefollowingerror:

Failed to reserve shared memory (error-no=12)

The following sections describe additional steps you should complete.

2. set kernel parameters.

Set three kernel parameters in /etc/sysctl.confasfollows:

kernel.shmmax = n

Where n is equal to the number of bytes of the maximum shared memory segment allowed on the system.

YoushouldsetittoatleastthesizeofthelargestheapsizeyouwanttousefortheJVM.Alternatively,

youcansetittothetotalamountofmemoryinthesystem,andyouwillneverhave

torevisitit.

vm.nr_hugepages = n

Wherenisequaltothenumberoflargepages.Youwillneedtolookupthelargepagesizein

/proc/meminfo.

vm.huge_tlb_shm_group = gid

WheregidistheIDofasharedgroupIDfortheusersyouwanttohaveaccesstothelargepages.

This setting enables you to limit access to the large memory segment.

3. set lImIts.conf parameters.

Next,setthesememlocklimitsin/etc/security/limits.conf:

soft memlock n

hard memlock n

Where istheruntimeuseroftheJVM,andnisthenumberofpagesfromvm.nr_hugepages

multiplied by the page size in KB from /proc/meminfo.

4. persIst your settIngs.

Enterthecommand:

sysctl p

Thisensuresthatthesettingsyoucreatedinstep3willsurviveareboot.



19/20

www.jb. 19

5. reBoot.

WhentheOSallocatesthesepages,itmustndcontiguousmemoryforthemortheoperationwillfail.

Arebootwillpreventthisproblem.

6. confIrm page allocatIon.

Whenlargepagesareallocated,/proc/meminfo will display a non-zero number for HugePages_Total.

Ifyoudonotseeanon-zeronumber,thenyouarenotusingthelargepages,andsomethingiscongured

incorrectly.

WehavesometimesseenaproblemwithMySQLwheretheSELinuxpolicywaspreventingitfromaccessing

the large pages. Check /var/log/messages for avc_denied messages (error-no=13 permission

denied) in the mysqld.log.

example: settIng large memory pages

Consideraserverwith8GBofmemory.Wewillallocate6GBtobesharedbytheJBossEAPJVMand

aMySQLdatabase.

Thepagesizeis2MB,asshownin/proc/meminfo (Hugepagesize: 2048KB)

Conguration:/etc/sysctl.conf

Changemaximumsharedmemorysegmentsizeto8GB.

kernel.shmmax = 8589934592

Add the gid to the hugetlb_shm_grouptogiveaccesstotheusers.

vm.hugetlb_shm_group = 501

Add6GBin2MBpagestobesharedbetweentheJVMandMySQL.

vm.nr_hugepages = 3072

Calculations:

1024*1024*1024*8=8589934592

(1024*1024*1024*6)/(1024*1024*2)or6GB/2MB=3072pages

Conguration:/etc/security/limits.conf

AddthelimitsformemlocktoallowtheJVMandMySQLtoaccessthelarge-pagememory.

jboss soft memlock 6291456

jboss hard memlock 6291456

mysql soft memlock 6291456

mysql hard memlock 6291456

root soft memlock 6291456

root hard memlock 6291456

Calculations:

3072largepages*2048KBpagesize3072*2048=6291456

Conguration:/etc/group

AddJBossandMySQLuserstothe501( hugetlb) group in /etc/grouptogiveuserspermission

to attach to the shared memory segment.



20/20

Copyright2010RedHat,Inc.RedHat,RedHatEnterpriseLinux,theShadowmanlogo,JBoss,MetaMatrix,andRHCEaretrademarksofRedHat,Inc.,registeredintheU.S.andothercountries.Linux is the registered

JBoss sales and InquIrIes

north amerIca

1888REDHAT1

www.jboss.com

europe, mIddle east

and afrIca

0080073342835

www.europe.redhat.com

[email protected]

asIa pacIfIc

+6564904200

www.apac.redhat.com

[email protected]

latIn amerIca

+541143416200

www.latam.redhat.com

[email protected]

www.jb.

aBout red hat

RedHatwasfoundedin1993andisheadquarteredinRaleigh,NC.Today,withmorethan60ofcesaround

theworld,RedHatisthelargestpubliclytradedtechnologycompanyfullycommittedtoopensource.That

commitmenthaspaidoffovertime,forusandourcustomers,provingthevalueofopensourcesoftware

andestablishingaviablebusinessmodelbuiltaroundtheopensourceway.RedHatprovideshigh-quality,

affordabletechnologytotheenterprise.Oursolutionsaredeliveredviasubscriptionandrangefromoper-

atingsystemsandplatformslikeRedHatEnterpriseLinuxandJBossEnterpriseMiddleware,toapplication

andmanagementtools,aswellasconsulting,training,andsupport.

jb jeap5 performance tuning wp web

Documents