for Web based System2006.02.01http://www.javaservice.comWonYoung LeePerformance Analysis Mathematical Approach on

Question: What is the definition of Performance?A.jspB.jsphttp://192.168.0.2/a.jsp http://192.168.0.2/b.jspUnit Response time0.5 secondsUnit Response time4.0 seconds

Application A: Response Time Grahp510152025510Concurrent UsersResponse Time(sec)1

510152025510Concurrent UsersResponse Time(sec)Application B: Response Time Graph

Comparison of A and B 510152025510Concurrent UsersResponse Time(sec)AB

1. Objective of Performance1.How many Clients2.Reasonable response time3.How to test or measure

1.1 Visit time & Think time1. DefinitionClick !Click !Response TimeThink TimeRequest IntervalRequest Interval(sec) = Response Time(sec) + Think Time(sec)Visit TimeNew User VisitedClick !Click !Click !Click !Click !Click !Leave

1.2 Think timeClick !Click !Response TimeThink TimeRequest IntervalThinkTime look like a constant value for the specific business domain1. Definition

1.3 Concurrent User Concurrent Users : 6TimeRequest Interval( Think Time) Different IP AddressesVisit TimeDefinitionConcurrent User = ActiveUser + Inactive UserNote: HTTP is connection-less1. DefinitionMax ThinktimeNote: How many active users?

Request IntervalResponse Time Think Time Concurrent User 1.4 Active UserTime ClickClickActive User Active Service Worker-ThreadSystem 1. Definition

1.5 ThroughputTimeUnit : tph, tpm, tps, (pps, rps, ops, hit/sec)3,600 tph = 60 tpm = 1 tpsMeasure TimeDefinition1. DefinitionNOTE: Arrival Rate, Service Rate

2. Request/Response System Model1.Mathematical Approach2.Queuing Theory3.Quantitative Analysis4.Measuring

2.1 Request and Response2. Request/Response System ModelAverage Response TimeThroughputAssumption: No ThinkTimeQuestion:What is the ability of her ?Right Answer:Maximum throughput# of Test Users# of Test Users# per minutesAve. Resp. Time

2.2 MeasuringThroughput2. Request/Response System ModelResponse TimeVirtual User (Assumption: ThinkTime=0)IncreasingVirtual User(ThinkTime=0)Throughput(tps)ResponseTime(sec)

2.3 Littles Law2. Request/Response System ModelResponse Time(sec)ActiveUser(ThinkTime=0)Number of ActiveUser = Throughput(tps) x Average Response Time(sec)Number of ActiveUserAverage Response Time(sec)Throughput(tps) = ActiveUserThroughput(tps)ResponseTime(sec)NRTN = T x R

Active Service Request IntervalResponse Time Think Time Concurrent User Worker-Thread2.4 ActiveUsers LawSystem ClickAve. ResponseTime(sec)Ave. ResponseTime(sec) + ThinkTime(sec) ActiveUser = ConcurrentUser x 2. Request/Response System ModelActive User ResponseTimes Law

2.4.1 Proff of ActiveUsers Law2. Request/Response System ModelActiveUserResp.Time(sec)Throughput(tps) = ConcurrentUserRequest Interval(=Resp.Time+ThinkTime)Throughput(tps) = Littles Law Resp.Time(sec)Request Interval(=Resp.Time+ThinkTime)ActiveUser = ConcurrentUser x ActiveUserResp.Time(sec)ConcurrentUserRequest Interval(=Resp.Time+ThinkTime)=

2.4.2 Meaning of Active User -1-Request RateService RateActive UserAve. ResponseTime(sec)Ave. ResponseTime(sec) + ThinkTime(sec) ActiveUser = ConcurrentUser x 2. Request/Response System Model

2.4.3 Meaning of Active User -2-2. Request/Response System Model

2.5 Concurrent User Equation2. Request/Response System ModelDefinitionEquationsResponseTime Resp.Time + ThinkTime= ActiveUser x ( 1 + ConcurrentUser = ActiveUser xResp.Time(sec)Request Interval(=Resp.Time+ThinkTime)ActiveUser = ConcurrentUser x = Throughtput x ( ResponseTime + ThinkTime)ActiveUserResp.Time(sec)Throughput(tps) = Littles Law )= ActiveUser + ( Throughput x ThinkTime)(Note: Throughput(tps) : ArrivalRate or ServiceRate)ResponseTimeThinkTime

Permanent CookieThinktimeArrival RateResponsetimeActive User(Service)- Concurrent User2.6 Example of Concurrent User Monitoring2. Request/Response System Model

2.7 SLA & ThinkTimeSLA(Service Level Agreement) - Concurrent User : 5,000 - Average Response Time: less than 3 secSLA(Service Level Agreement) - Concurrent User : 5,000 - Think Time = 30 (Example) - Average Response Time: less than 3 secThinkTime Agreement- Tele-market : 10-15sec- MIS Intranet : 15-20 sec- Internet Banking : 25-35 sec- Online Shopping Mall : 30-40 sec- Community : more longerReal ThinkTime Data+2. Request/Response System Model

2.8 Performance Test ThinkTime = 30Virtual User(ThinkTime=30 ConcurrentUser)Throughput(tps)5,000Response Time1.5235.38,0004.0Throughput GraphAverageResponseTime Graph158.7ConcurrentUser = Throughtput x { ResponseTime + ThinkTime(30) }

5000 = 158.7 x ( 1.5 + 30 ) 8000 = 235.3 x ( 4.0 + 30 )2. Request/Response System Model

2.9 Performance Test ThinkTime=0Virtual User(ThinkTime=0 ActiveUser)Throughput(tps)238Response Time158.7235.3941Throughput GraphAverage Response Time Graph1.54.0ConcurrentUser 5,000 8,000ConcurrentUser = ActiveUser + Throughput(tps) x ThinkTime(sec)ConcurrentUser = ActiveUser x { 1 + ThinkTime(sec)Resp.Time(sec)}ConcurrentUser = Throughput(tps) x { Resp.Time(sec)+ThinkTime(sec)}2. Request/Response System Model

2.10 Queuing Theory - G/G/1 Response TimeUtilizationActiveUserThroughput (

Concurrent Userreq/sec??Concurrent UserConcurrent UserArrival Rate()Active User(N)Response Time(R)Response TimeActiveUserArrival rateArrival rate2.11 Maximum Concurrent User2. Request/Response System Model

2.12 Saturation Point, Buckle Zone2. Request/Response System Model

Tuning ? Whats mean?2.13 Understanding of Throughput Graph2. Request/Response System Model

2.14 Throughput and Active UserDemo2. Request/Response System Model

3. Multiple Applications- Different Hit Ratio- Different Performance Homogeneous/heterogeneous Bottleneck Condition2. Request/Response System Model

3.1 2-Application model3. Multiple Applications40TPSmax204020AB402010403020ABAB 5401015ABActive Threads40TPSSaturation PointA application4020TPSSaturation PointB applicationActive Threads40Active Threads40TPS40202010ABActive Threads40TPS401030 5AB30Active Threads40TPS401510BA1030 Graph 1Graph 2Graph 31 : 13 : 11 : 3TPSmaxTPSmaxTPSmax

3.1.1 2-Applicatoin Model40A20102030B10graph 1graph 3graph 2ODifferent Arrival RateLimited by same resource bottleneck Homogeneous bottleneck condition3. Multiple Applications

xr1r23.1.2 2-Applicatoin ModelT2

yT1xxT1yT2+= 1=QPO(r1,r2)

r2r1A: r1 (req/sec)B: r2 (req/sec)R2

R1

y3. Multiple Applications

3.2 3-Application ModelQOT2T3yzP(0, T2, 0)(0, 0, T3)(0, 0)r1r2xT1r3xT1yT2+= 1zT3+xr1yr2=zr3=(T1, 0, 0)(r1, r2, r3)3. Multiple Applications

3.3 n-Applicatoin ModelNOTE: Under Homogeneous Bottleneck Condition3. Multiple Applications( Critical Inequality Performance Equation )

3.4 Performance UtilizationT2T3yzxT1xT1yT2+= 1zT3+0.12 + 0.13 + 0.45 = 0.70 { ri / Ti } x 100 = 70 %QP { ri / Ti } 1.03. Multiple ApplicationsUtilization of Critical Performance

App.TiTPSmaxrireq/secri / Tix40048.20.12y91.20.13z5022.70.45

3.5 Revision of Utilization of Critical Performancea.jsp : 5.4 req/secb.jsp : 1.9 req/secc.jsp : 10.7 req/secd.jsp : 4.5 req/sec..Arrival Rate { ri / Ti } * 1.0 ( is total hit ratio of ai) { ri / Ti } / = { ri / Ti } / (22.5/ 25) = 88%3. Multiple Applications

AppTiTPSmaxrireq/secri / Tia1425.40.129a2 91.90.210a37010.70.153a4154.50.300Subri =22.5 { ri / Ti } = 0.79TOTAL25(req/s)

QOzP(0, 0)r2xr3xr1yr2=zr3=(r1, r2, r3)Arrival rate linebottleneck saturation flatBottleneck surface { ri / Ti } 1.0 +(>0)3.6 Multiple Bottleneck TheoryHeterogeneous bottleneck condition3. Multiple ApplicationsCritical Inequality Performance Equation (final)

3.7 Performance Matrix i/i (1.0+) (0)( is total hit ratio of ai)Critical Performance Utilization: i/i(1.0+)(When ConcurrentUser = N)( 0)1.0NOTE: 8:2 Rule3. Multiple Applications

App.Arrival Rate(tps) Arrival RatioMaxThroughputi /ia111 /=111/1a2 22 /=222/2a333 /=333/3annn /=nnn/n(Others)others1-(Ex.20%)N/AN/ASUM i + others = i + 0.2 = 1.0N/A i/i

3.8 Objective of Performance EstimationArrival RateConcurrent UsersNkNkCurrent Performance Utilization : i/i(1.0+)1.0 (100%)x kCurrentCurrentFutureFuture3. Multiple ApplicationsHow many concurrent users can be accepted?How many times than now?

3.9 Performance Test for Multiple ScenarioThinkTime = 28.5 sec(Example)Response Time GraphVirtual User(ThinkTime = 28.5 sec) 942Acceptable Response Time3. Multiple Applications

App.Ratioia10.5a2 0.2a30.1(Others)0.2(20%)SUM0.8(80%)

Maximum adaptable concurrent user: 219 Expected maximum TPS : 6.515(tps) Maximum adaptable concurrent user : 202 Expected maximum TPS : 6.719 (tps) Result of Test using Multiple Scenario at the same timeResult of Test using each application test.3.10 Example of Performance Test1 + 1 = 2.5 ?3. Multiple Applications

Sheet1

Hit CountArrival RateR(Ratio)T(TPS)Rmax(Req/sec)Arrival Rate/TR/T

1.(listItemInfo)2970.0830.1416.250.14Rmax0.0050.0086

2.(listitem)2880.0800.131.50.13Rmax0.0530.0867JAVA EXCEPTION ERROR

3.(viewitm_real)1800.0500.0818.50.08Rmax0.0030.0043

4.(index)1160.0320.052.750.05Rmax0.0120.0182CPU 100% FULL

5.(searchItemByDetailInfo)1090.0300.0580.05Rmax0.0040.00638 DB 95% FULL

6.(listImpendingBidItem)1610.0450.07280.07Rmax0.0020.0025 CPU 70%

7.(searchZipAddress_01)550.0150.03600.03Rmax0.0000.0005 CPU 100%

8.viewItem_bond530.0150.0219.50.02Rmax0.0010.0010 CPU 90%

9.(listKamcoAucNotice)480.0130.022.50.02Rmax0.0050.00808 DB CPU 100%

10.newAucNotice450.0130.0240.02Rmax0.0030.00506 DB CPU 100%

11.FAQViewFAQ440.0120.02240.02Rmax0.0010.0008WEB CPU 80%

12.LISTFAQListFAQ430.0120.0219.50.02Rmax0.0010.0010DB , WEB CPU 90%

13.-viewAucNotice430.0120.02240.02Rmax0.0000.0008 - FULL

14.IDcheckDuplicateUserId330.0090.02670.02Rmax0.0000.0003 CPU - 94%

15.listItemInfoBySched600.0170.0122.50.01Rmax0.0010.0004 CPU - 90%

16.listAucNotice260.0070.0190.01Rmax0.0010.0011DB CPU 100%

17.welcomeRegisterForm240.0070.0122.50.01Rmax0.0000.0004 CPU 90%

18.createUserForm220.0060.0123.50.01Rmax0.0000.0004 CPU 80%

19.listDocForm210.0060.01320.01Rmax0.0000.0003 CPU 90%

20.searchItem210.0060.01190.01Rmax0.0000.0005 CPU 90%

21.listNewItem190.0050.0140.01Rmax0.0010.0025DB CPU 100%

22.loginForm190.0050.0125.50.01Rmax0.0000.0004 CPU 90%

23.contractPersonalUser170.0050.01240.01Rmax0.0000.0004 CPU - 70%

24.login160.0040.013.50.01Rmax0.0010.0029DB CPU 100%

Sum0.4890.790.79Rmax0.0950.1535

0.1535Rmax

3.11 Termination about Performance TestLoad TestStress TestAvailability TestPerformance Test

Relative Performance Problem SQL Query bottleneck (DB index, Full Scan, Heavy Query) Bottleneck on back-end transactions (CICS,TUXEDO, TCP/IP Socket) Relative bad performance on specific application (synchronized, CPU time) Relative bad performance on most of application caused by too small H/W Other side issue, for example, Network bottleneck.Two Types of Performance Problem Conditional Performance Problem JDBC Connection resource leakage Memory leakage (need too large memory, Memory Leak, Native Memory Leak) Unbalanced WAS tuning (Pool Size, Number of Thread, Heap Size) Caused by a bug on JVM/WAS/JDBC (Sybase JDBC, JVM Bug, WAS Bug) Thread Lock/Dead Lock (Application issue, Firewall issue)

Relative Performance ProblemRequest RateService RateActive User

Relative Performance Problem: Ramp up Test

Relative Performance Problem: Ramp up Test

JDBC Connection resource leakage Memory leakage (need too large memory, Memory Leak, Native Memory Leak) Unbalanced WAS tuning (Pool Size, Thread , Heap Size) Caused by a bug on JVM/WAS/JDBC (Sybase JDBC, JVM Bug, WAS Bug) Thread Lock/Dead Lock (Application/Framework issue, Firewall issue)

Database lock caused by uncommited nor unrollbacked database issue (buffer full, unexpected Batch Job, ..) upload or download a large file unexpected infinite loop on an application: CPU 100% Disk/memory Full bad performance on specific application or specific usresConditional Performance Problem

S80(12-way,8GB, 78,126 tpmC)S80(6-way,6GB, 41,140 tpmC)

Visitors a day: 4,506 usersPeak Concurrent users: 275 usersPeak Arrival Rate : 18.3 tpsRequest Interval : 18 secVistors vs Concurrent users: 6.1%

CPU : ?(unkown, no issue)119,266tpmC/18.3tps = 6,517 tpmC/tps3.1 Issue on Capacity Planning

3.1.1 K-university KMS System

3.1.2 B company M80 (4-way, 3GB, 34,588 tpmC) x 2S80 (12-way, 8GB, 78,126 tpmC) x 26F1 (4-way, 4GB, 44,500 tpmC) x 1H80 (2-way, 1GB) x 8

[Seoul:M80(4-way)x2: 69,176 tpmC ]Visitors a day: 1,621(total 3,435)Peak Concurrent Users: 600 usersHits a day : 466,639 hitsPeak Arrival Rate : 20.0 tpsRequest Interval : 30 secAverage Visit time: 1:25:43 Visitors vs Concurrent Users: 31%CPU Utilization: 70-100%69,176tpmC/20tps= 3,459 tpmC/tps

6F1(4-way,4GB, 44,500 tpmC ) Added.CPU Utilization: 60%113,676tpmC/20tps= 5,684 tpmC/tps

3.1.3 N-bank Internet bankingM80(4-way, 4GB, 34,588 tpmC) x 3

Visitors a day: 96,753 usersPeak Concurrent Users: 1,500-2,000 usersHits a day : 1,795,867 hitsPeak Arrival Rate : 58.3 tpsRequest Interval : 25.7 secVisit time: 6min 25secAverage hits per visit time: 18.6 clicksVisitors vs Concurrent Users: 1.6-2.0%

CPU Utilization: 70%103,764 tpmC/58.3tps= 1,780tpmC/tps

3.1.4 J-bank CRM SystemS85(12-way, 32GB, 124,818tpmC) X 2+ HOST DB

Visitors a day: 5,028 usersPeak Concurrent Users: 250 usersHits a day : 264,060 hitsPeak Arrival Rate : 10 tpsRequest Interval : 25.3 secVisit time: 4min 10secAverage hits per visit time: 9.9 clicksVisitors vs Concurrent Users: 4.97 %

CPU Utilization: 30-40% 249,636tpmC / 10 tps = 24,964 tpmC/tps

3.1.5 D-insurance e-Hanaro SystemWAS 6H1(4-Way 4 GB, 40,763 tpmC)DB H70(4-Way 4GB, 17,134 tpmC)+HOST CICS

Visitors a day: 2,800 user(resistered 3,300)Peak Concurrent Users: 350 userHits a day : 301,190 hitsPeak Arrival Rate : 33 tpsRequest Interval : 9 secVisit time: 16 minAverage Hits per visit time: 120.5 clicksVisitors vs Concurrent Users: 12.5 %

CPU Utilization: 50%40,763 tpmC / 33 tps= 1,235 tpmC/tps

3.1.6 D companyWAS H80(2-way, 2GB, 14,756(?) tpmC)DB S80(12-way, 8GB, 67,908tpmC)

Visitors: 258 usersPeak Concurrent Users: 45 userHits a day : 65,192 hitsPeak Arrival Rate : 4.2 tpsRequest Interval : 11-18 secVisitors vs Concurrent Users: 17.4 %

CPU Utilization: 50-60% (DB CPU: 35%)14,756 tpmC / 4.2 tps= 3,513 tpmC/tps

3.1.7 K-bank eCRM SystemWAS H80 (2-way, 4GB, 14,756(?) tpmC) x 2DB M80(2way-4GB, 18,647(?) tpmC)

Visitors a day: 37,951 usersPeak Concurrent Users: 230 usersHits a day : 235,527 hitsPeak Arrival Rate : 5.83 tpsRequest Interval : 24.7 secVisit time: 2 min 33 secAverage hits per visit time: 6.2 clicksVisitors vs Concurrent Users: 0.6 %

CPU Utilization: ?(unknown, no issued)29,512 tpmC / 5.83 tps= 5,062 tpmC/tps

3.1.8 K-CardM80 (4-way, 4GB, 34,588 tpmC) x 9WSBCC , Servlet/JSP, CTG

Visitors a day: 5,323 usersVisitors an hour: 4,500 usersPeak Concurrent Users: 2,800 usersHits a day : 6,198,133 hitsPeak Arrival Rate : 217 tpsRequest Interval : 23 secVisitors vs Concurrent Users: 53 %

CPU Utilization: (see left bottom side grahp)311,292tpmC / 217 tps= 1,435 tpmC/tps

[2nd machine]Peak Arrival Rate : 31.7 tpsPeak Concurrent Users: 600 users34,588 tpmC / 31.7 tps= 1,091 tpmC/tps

[4th machine]Peak Arrival Rate : 50 tps34,588 tpmC / 50 tps= 692 tpmC/tpsCPU Utilization

3.1.9 Result of my statisticstpmC / 1 tps(based on 70% cpu)

Analysis Methodology3.2 Bad Process for Perf. Mgmt.ProposalPilot BMTAnalysis/DevelopingFunction estTesting withSmall real usersSystemOpenMonitoringUnit App.Perf.TestPerf.PredictionPerf. DataLoggingAdditional App. DevelopApplyingApp.WorkloadAnalsysExperienceddataCapacityEstimationBMT methodologyAppTuningPerf AnalysisUnit App.Perf.TestPerf.PredictionPerf Analysis