1

J2EE: Black Box in the Oracle World

Don Bergal, Confio Software

August 17, 2006

Objective: Answer a Question

“Why does a DBA need visibility upstream in the application in order to manage the database?”

An alternate title for today:

“How to be a better DBA by watching something other than the Oracle database”

Agenda

J2EE Black Box in an Oracle World

a) The Black Box Application Problem

b) The DBA Hot Seat

c) System Statistics versus Wait Time Diagnostics

d) A Better J2EE End-To-End Performance Solution

e) Best Practices Response Time Diagnostics Methodology

f) Q & A

The Black Box Application Problem

J2EE: Black Box in Oracle World

How can you manage here?

If you cannot see here!

Web customer

Internal user

Batch job

Billing

Financial System

Web store

Black Box:

Applications and End Users

See the applications that impact database performance

Web customer

Internal user

Batch job

Billing

Financial System

Web store

Database

App Servers Web Users

Result: Disconnect Between IT Groups?

DBA’s

Databases• Database Management

• Database Development

• SQL Development

• SQL Tuning & Optimization Tools

• Ratio Based Monitoring & Tuning

J2EE Web & App Tiers

Java Development

• Client Web/Applet/Application

• JSP/Servlet Development

• EJB Container Development

• JDBC Development

• Uses Desktop J2EE Profilers

No Visibility into the Database Tier

No Visibility into the J2EE Tier

Effect: Black Boxes leave DBA defenseless

‘Silo-based, Single System’ tools only shows database system perspective (even though they are blamed for problem in any system). Can’t tell when to hand-off problem to another group (Dev/SysAdmin/etc…)

Blamed for all layers, but can only see the database

Black Box

Black Box

The DBA Hot Seat

Finger Pointing Happens When You Can’t See the Problem

What’s wrong with your *&!@$# database?

It’s a database issue!

Java/Web DeveloperManagement DBA

Guilty until proven innocent

It’s a configuration

issue!

Software Vendor

It’s a database issue!

Network Engineers DBA Hot Seat

Do Customers See the Problem Before You?

Java Developer

DBA

Not my code…

What performance problems ?!?

SLA ViolationU

ser

Wait

Tim

eWarning!

Not my Network,

Servers, Load Balancer, Firewall…

System Admin

Hot Seat comes from UNCLEAR Ownership

Need evidence to show where the problem exists, avoid “Finger Pointing” to the database

It’s an Application issue!

It’s a Database issue!

Java App OwnerDBA

14

DBA

NetworkingDeveloper

SystemAdmin

Performance Suspects Line Up…

Lack of Visibility is Source of Problem

J2EE layer

App Layer Oracle eBusiness

Oracle 9iAS Peoplesoft

SiebelJD Edwards

SAPCustom apps

Oracle databaseWeb Server

Web page requestApache

WebLogicWebSphere

IIS

Service Levels Measured Here…

…Blame assigned Here

Typical Multi-Tiered J2EE System Leaves the DBA Blind

Why Does the DBA Need J2EE Visibility?

You can’t control what you can’t seeYou can’t show/demonstrate it’s not a DB IssueYou can’t manage your database if you are blind to the

source of the requests

Traditional Performance Tools versus Wait-Time Diagnostics

Current Tools – Never Cross the Silo Boundaries

Tune inside each “Silo” to improve end-end performance

“System Health” or ratio-based approach to performance…

Statistics…% Utilization…Event Counts… No direct relationship to End-User experience or SLA Weak linkage between tiers limits collaboration

CPU

Memory

Sessions

DatabaseJ2EE ServerWeb Server

CPU

Heap Size

Threads

CPU

Reads

Processes

Standard J2EE App Performance Monitoring: How Does This Help Identify User Response?

Idle threads

Number of requests

Throughput

Waiting Requests

Memory (Heap) Usage

J2EE Profilers: High Load, Test Data Only

OS Level System Monitoring (rstatd/perfmon)

Network System Monitoring

Traditional Tools vs. Wait-Time Methodology

Ratio-Base traditional tools show “Oracle system health” but no link to end user service

Use Wait Time to correlate user requests to SQLs and performance impact

A Better J2EE End-To-End Performance Solution

Requirements for Oracle & J2EE Monitoring

Use Oracle Wait-Time Based Monitoring Visibility across Java and Oracle layers Lightweight JVM Method Tracker Agnostic JVM server support Fast installation and data collection

techniques Cost Effective EASE OF USE – MAKE IT OBVIOUS!

Wait-Time Based Performance Analysis

Tune by determining where client request time is spent

Determine where Oracle resources are spent during the execution of SQL execution

Emerging best-practice for database tuning• “You can’t tell how long something took by counting how many

times it happened. … If you’re only measuring event counts, then you’re not measuring what the users care about.”

—Optimizing Oracle Performance, O’Reilly Press

Oracle is building up Wait-based tuning capabilities with each database release…10g has over 800 wait-time based performance counters

Oracle Wait Interface Book

Free book at www.confio.com

A Wait-Time J2EE Performance Methodology: 3 Key Requirements

1. Follow every transaction individually2. Identify every “step” along the path3. Measure the “wait-time” at each step

Result: Minimize time to resolution and division of work by clearly identifying which step is performing poorly.

Alternative Monitoring Architectures

Best Practices Wait Time Diagnostics Methodology(Day in the life on the performance team)

4 Steps for J2EE to Oracle Visibility

1. Capture Every User Request (end-to-end transaction)2. Capture Every Step Inside Each Tier3. Measure Response Time of Each Step 4. Identify Possible Causes Based on Wait-Times

Since every step is visible, you can always see the problem

4 Steps for J2EE to Oracle Visibility

1. Capture Every User Request (end-to-end transaction)2. Capture Every Step Inside Each Tier3. Measure Response Time of Each Step 4. Identify Possible Causes Based on Wait-Times

Since every step is visible, you can always see the problem

1. Capture Every User Request

Identify the slowest performing end-to-end Requests

Slowest performing can be:

• Longest average request-response time

• Largest accumulated Wait-Time in application (across all sessions)

1. Capture Every User Request

PlaceOrder.jsp Request has most serious delays

2. Capture Every Step Inside Each Layer

Identify the slowest steps within the slowest Request

Key Requirement: Correlate Application Response to SQL Response Times

Question: Which SQL statements are causing Wait Time for my application?

2. Capture Every Step, Identify Slowest

UpdateOrder Total (SQL) is slowest in PlaceOrder.jsp(Most Wait-Time accumulated)

3. Measure Response Time of Each Step

Focus on steps that consume the most time

Detailed Response Time for the Critical Step

Update Order Total• ‘Undo Segment Extension’ Wait-Event is bottleneck

4. Possible Causes Based on Wait Times

Diagnose the step with the greatest Wait Time

4. Possible Causes Based on Wait Times

Execution path and time spent on each ‘Hot Spot”

Identify Chain of Calls

• URL Request -> Method Call –> SQL

• Allocate Wait Time to each Method or SQL Call

• Profiler data Without the System Impact

Recall 3 Key Requirements to Provide the Analysis

2. Identify Every step – Java method

or Wait-Event

3. Measure Wait time for Every Step

1. Identify individual Request

Customer Example

Retail transaction system Online media fulfillment Customer requests running slowly,

unknown reason Ignite for Oracle and Ignite for Java used

to examine situation

Wait Time for “Authorize Transaction” Identifies Problem Steps

Associates Request Wait-Time with Single Procedure

Same query multiple times in single procedure Same row retrieved 7 times Change to select once, cut Wait-Time to less

than half

Summary: Look For…

Continuous monitoring 24/7 – not session/period specific

Production data – not synthetic “test” transactions

Correlate Application & Oracle – where most of the time is spent

Individual transactions, not system-wide statistics

Visibility for the DBA

DBA has a need to see the J2EE Application Solve database problems by understanding the

other systems Look for Wait Time – that’s where the action is Achieve collaboration between Developers,

Architects and DBA’s

DBA can be the leader, not the one to take the blame!

About Confio Software

Wait-Time based performance tools for continuous, low impact monitoring

Ignite for Oracle – Deep, detailed visibility into SQLs and Wait-Events

Ignite for Java – End-to-End Application through Oracle visibility, correlates Java and SQL Wait-Time

Ignite for SQL Server – Coming soon, customer testing

Free Trial at www.confio.com

About the Author

Don Bergal, Confio Software Responsible for product management,

among other things… 18 years software/telecom industry

experience donbergal@confio.com

Q & A