j2ee: black box in the oracle world - new york oracle user...
TRANSCRIPT
![Page 1: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/1.jpg)
J2EE: Black Box in the Oracle World
December 2005
![Page 2: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/2.jpg)
DBA Hot Seat: Have these things everhappened to you?
DBA Application Owner
IT Manager
Software Vendor
![Page 3: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/3.jpg)
Hotseat comes from Unclear Ownership
Need evidence to show where the problem exists,avoid “Finger Pointing” to the database
It’s an Application issue! It’s a Database issue!
App OwnerDBA
![Page 4: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/4.jpg)
Unclear where to add capacityinvestment?
Need to idendtify what improvement will occur beforeexpenditures or tuning projects…
Where do we put the hardwareinvestment – Database or App
Server?
DBA
Management
I need it…
J2EE Owner
I need it…
![Page 5: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/5.jpg)
Problems must occur multiple times
Why did we have poorperformance this morning?
Management DBA
Not sure, we needto wait for it tohappen again
Need to watch productiontransactions, not justsynthetic/test transactions
![Page 6: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/6.jpg)
Typical Multi-Tier System
J2EE layer
App Layer Oracle eBusiness
PeoplesoftSiebelSAP
Custom apps
OS Independent OS Independent
Oracle database
OS Independent
Web Server
Web page request
Service Levels Measured Here…
…Blame assigned Here
![Page 7: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/7.jpg)
Typical Java tools
Focus on system measurement, not WaitTime/Service TimeJava details, no database visibility – but mostJava delays are due to databaseNo real time/constant monitoring
![Page 8: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/8.jpg)
Typical Oracle Database Tools
Focus on database in isolation – noconnection to the J2EE applicationLook system wide, not at individual SQLsCount executions, not Wait TimeTraces for special sessions, no continuousmonitoring
![Page 9: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/9.jpg)
System Centric Monitoring: How Can YouManage Performance this Way?
Focus on system operationNo correlation to transactionperformance
Idle threads
Number ofrequests
WaitingRequests
MemoryUsage
![Page 10: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/10.jpg)
Wait-Time Based Performance Analysis
Emerging best-practice for database tuning• “You can’t tell how long something took by counting how
many times it happened. … If you’re only measuring eventcounts, then you’re not measuring what the users careabout.”
—Optimizing Oracle Performance, O’Reilly Press
Oracle is starting to build wait-based tuning tools intothe database particularly in 10g
Tune by determining where processing time is spent
![Page 11: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/11.jpg)
Track Wait Time,Not System Counters
SQL 1
SQL 2
SQL 3
Resources I/O Network RedoLocks
• Watching Counters leads to wrong conclusions: Time is more relevant
• Total System Counters hide information: Need breakdown toindividual steps in your transaction
5 R
25 R
50 Reads
TotalSystemCounter
80K Reads
30 Minutes
15M
5M
6 M
10 M
100 Minutes
35 A
50 A
50 A
125 Attempts
4 M
200 Minutes
5M
4 M
200 Minutes
5M
5K Packets 216K Writes
![Page 12: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/12.jpg)
1st Step: Course Grained Measure of TimeBetween Systems
![Page 13: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/13.jpg)
Many Tools Never See Inside YourMulti-Tier Application
Black Box
Black Box
Black Box
Black Box
Black Box
Web – J2EE - Oracle
Standard approach: Treat each system as a Black BoxNo detail inside the systemWhere are the bottlenecks, and who is responsible?
![Page 14: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/14.jpg)
3 Requirements for J2EE to Oracle Visibility
Every TransactionEvery Step inside each layerMeasure Time – That’s what users care about
![Page 15: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/15.jpg)
Performance Monitoring where it MattersMost
J2EE layer
App Layer Oracle eBusiness
PeoplesoftSiebelSAP
Custom apps
OS Independent OS Independent
Oracle database
OS Independent
Web Server
Web page request
This is where thewait timeaccumulates
![Page 16: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/16.jpg)
Desired Correlation Between Java andOracle
Key Questions toAnswer
Where are thetransactionbottlenecks?
Why is Javawaiting on the
database?
What in theApp server isdriving myOracle load
![Page 17: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/17.jpg)
Essential Step: Correlate Application WaitTime to SQL Waits
Question: Which SQL statements are causing Wait Time formy application?
![Page 18: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/18.jpg)
Correlate URL Requests to Specific SQLs
Question: Which User Requests are responsible for drivingSQL Wait Times?
![Page 19: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/19.jpg)
Result: Identify Where the BottlenecksOccur
App Server Oracle Database
![Page 20: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/20.jpg)
Get down to Details
Recall 3 requirements: 1. Individualrequest
2. Everystep
3. Waittime
![Page 21: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/21.jpg)
Typical J2EE Performance Test Setup –Trial and Error to Find Bottleneck
• Functional Test/ Memory Leaks
• Build Test Environment
4. Single session Profile – time measure
3. Simulate Load
5. Measure system parameters
6. Try some changes
Typical Cycle Desired Cycle• Functional Test/ Memory Leaks
• Validate in Test Environment
3. Measure Real Transactions
4. Identify bottleneck steps
5. Monitor Service Time Compliance
![Page 22: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/22.jpg)
Java Measurement Techniques
JMX – Heavy LoadProfiler – Test transactions onlyByte code instrumentation – ProductionMonitoring• Insert monitor byte into selected Java methods• Jar file pre-processor = No runtime intrusion• Test/validate instrumented jars on integration
system• Download to production environment
Method execution correlates with JDBC/SQL
![Page 23: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/23.jpg)
Identify Execution Paths
• SQL – Method Call – URL Request
• Allocate Wait Time to each
![Page 24: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/24.jpg)
Conclusions
DBA does not have to take the blameSolve database problems by understandingthe other systemsLook for Wait Time – that’s where the actionisConnect SQL Waits with J2EE application timeWatch production data, not synthetictransactions
![Page 25: J2EE: Black Box in the Oracle World - New York Oracle User ...nyoug.org/Presentations/2005/200512_j2ee.pdf · Siebel SAP Custom apps ... Essential Step: Correlate Application Wait](https://reader038.vdocuments.site/reader038/viewer/2022110222/5aac2fd77f8b9a2b4c8cd5f3/html5/thumbnails/25.jpg)
About the Author
Don Bergal, Confio SoftwareResponsible for Product Management, amongother things…17 Years software product managementacross enterprise, telco, wireless, [email protected]