using statspack to track down bad code

8/9/2019 Using Statspack to Track Down Bad Code

1/11

USING STATSPACKTO TRACK DOWN BAD CODEMichael R. Ault Burleson Consulting

IntroductionMany times a developer may be given the task of helping the DBA find and resolve an applications poorly performing code.Of course first one must define what constitutes poor performance. Is poor performance a certain number of logical orphysical IOs? Is it a certain number of consistent reads? I believe it needs to be defined in the context of the specificapplication. For example, in an order entry situation inserts should be expected to be sub-second. Another example would bethat in a customer service application the customer information screen should be expected to be populated within less thanseven seconds. Yet another would be that a decision support system needs to return results within 15 minutes. Each of theseapplications has different expectations for performance and each must be tuned using that expectation set in mind.

Another key concept when tuning is the concept of enough is enough. This concept means to set specific tuning goals andwhen you reach them, go on to the next problem. Tuning Oracle has been likened to a video game with infinite levels, there isalways a way to get a few more milliseconds or microseconds of performance from Oracle, you have to know when to quit!

In this presentation we will look at using the statspack, and by extension the AWR, tool for finding and correcting bad SQLin an application.

Setting Up StatspackBefore you use statspack it must be installed. This is fairly simple to do. To install statspack you must be on release 8.1.7 ofOracle or later (you can go back as far as 8.1.5 but results can be interesting with these older releases) also, be very carefulabout what you have running, we have seen issues with early versions of ColdFusion and Statspack colliding. Anyway, thesteps to install are:

1. Make sure dbms_jobs and dbms_shared_pool are installed in the system.

2. Review the spcreate.sql series of called scripts and eliminate the calls to install the packages in step 1.

3. Create a perfstat tablespace

4. Run the spcreate.sql script, usually in the $ORACLE_HOME/rdbms/admin directory

5. Use the statspack.snap procedure to test the install

6. Start automated statistics runs with spauto.sql

The spcreate.sql script runs a series of other scripts that creates the appropriate user, gives them the proper grants and buildsthe proper tables and packages for the statspack process. By default it create the perfstat user with the password perfstat. Notein step 2 we suggest editing the appropriate script to remove the build/rebuild of the dbms_jobs and dbsm_shared_pool. Thisis to prevent locking issues should these packages already be installed. It is strongly suggested that these be prebuilt so youcan ensure there are no issues.

Once statspack is installed, you can begin testing.

AWR UsageIn 10g we now have AWR. The Automatic Workload Repository (AWR) defaults to a collection interval every 30 minutesand collects data that is the foundation for all of the other self-tuning features. AWR is very much like STATSPACK,especially the level-5 STATSPACK collection mechanism where top SQL is collected every hour, based on your rollingthresholds for high-use SQL. In addition to the SQL, AWR collects detailed run-time statistics on the top SQL (disk reads,executions, consistent gets) and uses this information to adjust the rolling collection threshold. This technique ensures thatAWR always collects the most resource intensive SQL. The AWR system provides reports that can be run against the AWRtables that provide the same type of data a statspack report used to, and, much more. AWR takes advantage of the ADDM(Adam) to gather its statistics using non-SQL based collection techniques which is much more efficient than the old SQL

www.odtug.com ODTUG 2005
http://www.odtug.com/http://www.odtug.com/


2/11

Using Statspack to Find Bad SQL Ault

based statspack and less intrusive on the database. The awrrpt.sql script is used to generate the reports that take the place ofthe Statspack reports in 10g.

Testing Using Statspack or AWRStatspack and AWR are simply statistics capture tools that also provide a report that shows you performance related statistics.Part of the statistics are several listings of SQL statements sliced and diced by number of consistent gets, number of physicalreads, number of parses, number of executions and number of versions. Of course for a statement to appear there, it musthave been run!

The prime thing to remember when testing with statspack is that the statspack process will only capture what is currentlygoing on in the database. It takes snapshots of the current state and allows you to choose two snapshots to compare. IN thecase of the SQL statements, if they are still in the shared pool between the two statspack runs, then your changes may bemasked by old code runs. Therefore for testing purposes you need to flush the shared pool, execute a snapshot, then test yourchanges and execute another snapshot:

1. Use the ALTER SYSTEM FLUSH SHARED_POOL; command to flush old SQL form the pool.

2. Execute the command execute statspack.snap; to begin a statspack capture window.

3. Run your test code

4. Execute the command execute statspack.snap; to end the statspack capture window.

5. Run the spreport.sql script to generate a report based on the time interval between steps 2 and 4.

If on the other hand, you dont already know what SQL is causing issues, which may be the case in a large system with thegeneral user complaint Everything is Slow! use the automated statspack gathering (spauto.sql) to capture a profile ofstatspack runs across the time periods when the users have the problems. Running statspack off hours probably wont tell youa lot, you need to run it when the problem is occurring!

Evaluating Statspack or AWR for Code IssuesThe statspack and AWR reports have a number of sections devoted entirely to code. There are also sections devoted to waitsand one to latches. Using the cross-mix between waits, latches and the reported SQL the developer can isolate and thencorrect the problem SQL. Generally you will start with the major hitters, SQL that shows up in more than one of the major

areas relegated to SQL. For example, if a SQL shows up in the top ten in both the consistent gets and disk reads area then it isprobably a good candidate for optimization.

For this paper we will be using example outputs from Statspack reports.

However, if you are looking at code in the statspack report that appears to be a problem but it is not used all the time or is anad hoc query, then it may not be an issue unless it has the potential to become a major use SQL. Lets look at some actualstatspack outputs and see what we can determine from them about the code they contain. Look at Figure 1.

Snap Id Snap Time Sessions Curs/Sess Comment------- ------------------ -------- --------- -------------------

Begin Snap: 1240 26-May-05 15:00:02 34 5.0End Snap: 1241 26-May-05 16:00:02 33 4.9Elapsed: 60.00 (mins)

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % Total

Event Waits Time (s) Ela Time-------------------------------------------- ------------ ----------- --------db file sequential read 320,448 4,588 32.25direct path read 58,652 2,988 21.00CPU time 2,182 15.34PX Deq: Execute Reply 1,428 1,257 8.83db file scattered read 11,406 1,020 7.17

-------------------------------------------------------------

SQL ordered by Gets for DB: TEST10 Instance: TEST10 Snaps: 1240 -1241-> End Buffer Gets Threshold: 10000-> Note that resources reported for PL/SQL includes the resources used by



3/11


all SQL statements called within the PL/SQL code. As individual SQLstatements are also reported, it is possible and valid for the summedtotal % to exceed 100

CPU ElapsdBuffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value

--------------- ------------ -------------- ------ -------- --------- ----------1,340,065 1 1,340,065.0 3.3 27.85 27.75 2183036294

Module: SQLNav5.exeSELECT /*+ FIRST_ROWS */ KEY||','||WRTN_PREM||','||WRTN_EXPSR||','||EARNED FROM ( SELECT /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */ pcoopt.pol_id||','||pcoopt.pol_tran_id||','||pcoopt.veh_unit_nbr||','|| pcoopt.covg_mp_cd||','||pcoopt.covg_cd KEY, SUM(pcoopt.veh_covg_wrtn_prem_

1,278,490 1 1,278,490.0 3.2 11.15 12.26 656581566Module: SQLNav5.exeSelect /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||','||EARNED From ( Select /*+ FIRST_ROWS */ --/*+ INDEX(pcoo

pt POL_COVG_ON_OFF_PREM_TRAN_IDX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */ pcoopt.pol_id||','||

pcoopt.pol_tran_id||','||pcoopt.veh_unit_nbr||','||

972,208 1 972,208.0 2.4 10.10 13.08 3656618493Module: SQLNav5.exe

Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||','||EARNED From ( Select /*+ INDEX(pcoopt POL_COVG_ON_OFF_PRE M_TRAN_IDX1) */ --/*+ INDEX(pcoopt POL_COVG_ON_OFF_PREM_TRAN_IDX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */

pcoopt.pol_id||','||pcoopt.pol_tran_id||','||pcoop

796,391 20,322 39.2 2.0 90.85 3506.03 2624188903Module: SQL*PlusSELECT /*+ INDEX(p1 iu_pol_isc) */ t1.pol_tran_eff_dt

FROM pol p1, pol_tran t1 WHERE p1.pol_nbr = :b2 AND p1.ign_sys_cd =:b1 AND t1.pol_id = p1.pol_id A

ND t1.pol_tran_typ_cd = 'CN' AND

707,029 1 707,029.0 1.7 7.01 6.84 324622422Module: SQLNav5.exeSelect /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||

','||EARNED From ( Select /*+ INDEX(pcoopt POL_COVG_ON_OFF_PRE M_TRAN_IDX1) */ --/*+ INDEX(pcoopt POL_COVG_ON_OFF_PREM_TRAN_IDX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */


113,134 19,300 5.9 0.3 2.96 3.06 1584996842Module: SQL*PlusSELECT /*+ INDEX(p3 iu_pol_isc) */ MAX(p3.pol_expr_dt)

FROM pol p3 WHERE p3.pol_n br = :b2 AND p3.ign_sys_cd = :b1

55,151 6 9,191.8 0.1 3.62 85.42 2433730562Module: jre@testit (TNS V1-V3)/* OracleOEM */ SELECT d.status "Status", d.tablespace_name "Name", d.contents "Type", d.extent_management "Extent Management",TO_CHAR(NVL(a.bytes / 1024 / 1024, 0),'99,999,990.900') "Size (M)", TO_CHAR(NVL(a.bytes - NVL(f.bytes, 0), 0)/1024/1024,'9999999

9.999') ||'/'||TO_CHAR(NVL(a.bytes/1024/1024, 0), '99999999.999'

47,895 1 47,895.0 0.1 27.18 577.44 1017018803Module: jre@testit (TNS V1-V3)/* OracleOEM */ SELECT DL.SEGMENT_FILEID, DL.SEGMENT_BLOCK, F.FILE# FNO, DL.BLOCK BNO, DL.LENGTH, DL.EXTENT_ID FROM SYS.DBA_LMT_USED_EXTENTS DL, SYS.FILE$ F WHERE DL.TABLESPACE_ID = :1 AND DL.TABLESPACE_ID = F.TS# AND DL.FILEID = F.RELFILE# UNION ALL SELECT -1, -1, F.FILE# FNO, DL.BLOCK_ID BNO, DL.BLOCKS, -1 FROM SYS.D

38,004 4 9,501.0 0.1 0.45 0.49 4287766396Module: SQLNav5.exe



4/11


select a.column_name,a.column_length,a.column_position,b.column_expression,decode(a.descend,'ASC','N','Y') descend,a.index_ownerobject_owner,a.index_name object_name from sys.dba_ind_columns

a,sys.dba_ind_expressions b where a.index_owner=:schema and a.index_name=:object_name and b.index_owner(+)=a.index_owner and b.i

-------------------------------------------------------------

SQL ordered by Reads for DB: TEST10 Instance: TEST10 Snaps: 1240 -1241-> End Disk Reads Threshold: 1000

CPU ElapsdPhysical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value

--------------- ------------ -------------- ------ -------- --------- ----------90,051 20,322 4.4 1.9 90.85 3506.03 2624188903

Module: SQL*PlusSELECT /*+ INDEX(p1 iu_pol_isc) */ t1.pol_tran_eff_dt



9,752 1 9,752.0 0.2 27.18 577.44 1017018803Module: jre@testit (TNS V1-V3)/* OracleOEM */ SELECT DL.SEGMENT_FILEID, DL.SEGMENT_BLOCK, F.FI

LE# FNO, DL.BLOCK BNO, DL.LENGTH, DL.EXTENT_ID FROM SYS.DBA_LMT_USED_EXTENTS DL, SYS.FILE$ F WHERE DL.TABLESPACE_ID = :1 AND DL.TABLESPACE_ID = F.TS# AND DL.FILEID = F.RELFILE# UNION ALL SELECT -1, -1, F.FILE# FNO, DL.BLOCK_ID BNO, DL.BLOCKS, -1 FROM SYS.D

1,619 6 269.8 0.0 3.62 85.42 2433730562Module: jre@testit (TNS V1-V3)/* OracleOEM */ SELECT d.status "Status", d.tablespace_name "Name", d.contents "Type", d.extent_management "Extent Management",TO_CHAR(NVL(a.bytes / 1024 / 1024, 0),'99,999,990.900') "Size (M)", TO_CHAR(NVL(a.bytes - NVL(f.bytes, 0), 0)/1024/1024,'99999999.999') ||'/'||TO_CHAR(NVL(a.bytes/1024/1024, 0), '99999999.999'

1,466 1 1,466.0 0.0 10.10 13.08 3656618493Module: SQLNav5.exeSelect /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||','||EARNED From ( Select /*+ INDEX(pcoopt POL_COVG_ON_OFF_PRE

M_TRAN_IDX1) */ --/*+ INDEX(pcoopt POL_COVG_ON_OFF_PREM_TRAN_IDX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */pcoopt.pol_id||','||pcoopt.pol_tran_id||','||pcoop

852 1 852.0 0.0 11.15 12.26 656581566Module: SQLNav5.exeSelect /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||','||EARNED From ( Select /*+ FIRST_ROWS */ --/*+ INDEX(pcoo



443 1 443.0 0.0 1.65 4.51 2892151655Module: SQL*PlusBEGIN statspack.snap(i_snap_level=>6); END;

75 4 18.8 0.0 0.30 1.47 2243218285

Module: SQLNav5.exeselect :schema owner,a.object_name,a.object_id,a.created,a.last_ddl_time,decode(a.status,'VALID',0,'INVALID',1,2) status,decode(

b.partitioned,'YES','Y','NO','N') partitioned,'N' object_table,' N' external_table,decode(b.nested,'YES','Y','N') nested,decode(b.IOT_Type,'IOT',1,'IOT_OVERFLOW',2,0) IOT_Type,b.IOT_Name,b.temp

62 326 0.2 0.0 0.81 1.46 1604489463Module: jre@testit (TNS V1-V3)select file#, block#, type#, nvl(spare1,0), hwmincr, cachehintfrom seg$ where ts# = :1



5/11


54 1 54.0 0.0 0.27 0.65 4119687667Module: jre@testit (TNS V1-V3)/* OracleOEM */ SELECT -1, RELATIVE_FNO, HEADER_BLOCK, SEGMENT_TYPE, OWNER, SEGMENT_NAME, PARTITION_NAME, EXTENTS, BLOCKS, MIN_EXTENTS, MAX_EXTENTS, (INITIAL_EXTENT/1024), (NEXT_EXTENT/1024),PCT_INCREASE FROM SYS.DBA_SEGMENTS WHERE TABLESPACE_NAME = :1 ORDER BY 2, 3

21 19 1.1 0.0 0.04 0.60 4080861370select owner#,name,namespace,remoteowner,linkname,p_timestamp,p_obj#, d_owner#, nvl(property,0),subname from dependency$,obj$ where d_obj#=:1 and p_obj#=obj#(+) order by order#

Figure 1: Basic Sections from Statspack

From the data in Figure 1, we first need to determine if there are problems. We can see that the duration is good (one hour)and we can see that our majority waits are also IO related (db file sequential and scattered reads, direct reads) these type ofwaits show we are seeing large numbers of single and multiblock IOs as well as probable sorts (the direct IO events.) Thisclues us to look at the physical reads to see what SQL is occurring there. Look at Figure 2.

Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value--------------- ------------ -------------- ------ -------- --------- ----------

90,051 20,322 4.4 1.9 90.85 3506.03 2624188903Module: SQL*Plus

SELECT /*+ INDEX(p1 iu_pol_isc) */ t1.pol_tran_eff_dtFROM pol p1, pol_tran t1 WHERE p1.p

ol_nbr = :b2 AND p1.ign_sys_cd =:b1 AND t1.pol_id = p1.pol_id A



M_TRAN_IDX1) */ --/*+ INDEX(pcoopt POL_COVG_ON_OFF_PREM_TRAN_IDX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */


852 1 852.0 0.0 11.15 12.26 656581566Module: SQLNav5.exeSelect /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||

','||EARNED From ( Select /*+ FIRST_ROWS */ --/*+ INDEX(pcoo pt POL_COVG_ON_OFF_PREM_TRAN_IDX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */ pcoopt.pol_id||','||


Figure 2: SQLs with Large IOs

The three SQL statements in Figure 2 are application SQL statements, the other of the top 10 are all from monitoring tools.Of the three, two are limited to only a single execution (whether it is from our period or fell outside of it, is hard to tell)however we have one SQL that was executed a whopping 20,322 times. Even though it only does 4.4 reads per execution, itdoes a great number of these. By tuning this SQL we can reduce the physical IO requirements of the application. That we aregetting lots of small reads (4.4 IOs per execution) indicates that this may be the source of many of our sequential read waitsreported in this period. Generally sequential read waits can be mitigated by more cache memory, we may not need to tunethis query.

The scattered read waits are probably due to the monitoring SQL being generated by the monitoring reads. The problem heremay also be memory related as the memory may not be large enough to hold the data needed by the application.

Lets look further into our SQL examples here by looking at the application code that is doing the most consistent gets next.Look at Figure 3.

Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value--------------- ------------ -------------- ------ -------- --------- ----------

1,340,065 1 1,340,065.0 3.3 27.85 27.75 2183036294Module: SQLNav5.exeSELECT /*+ FIRST_ROWS */ KEY||','||WRTN_PREM||','||WRTN_EXPSR||','||EARNED FROM ( SELECT /*+ INDEX(pcoopt PK_POL_COVG_ON_OF



6/11


F_PREM_TRAN) */ pcoopt.pol_id||','||pcoopt.pol_tran_id||','||pcoopt.veh_unit_nbr||','|| pcoopt.covg_mp_cd||','||pcoopt.covg_cd KEY, SUM(pcoopt.veh_covg_wrtn_prem_


pt POL_COVG_ON_OFF_PREM_TRAN_IDX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */ pcoopt.pol_id||','||pcoopt.pol_tran_id||','||pcoopt.veh_unit_nbr||','||





FROM pol p1, pol_tran t1 WHERE p1.pol_nbr = :b2 AND p1.ign_sys_cd =

:b1 AND t1.pol_id = p1.pol_id A ND t1.pol_tran_typ_cd = 'CN' AND




113,134 19,300 5.9 0.3 2.96 3.06 1584996842Module: SQL*PlusSELECT /*+ INDEX(p3 iu_pol_isc) */ MAX(p3.pol_expr_dt)

FROM pol p3 WHERE p3.pol_n br = :b2 AND p3.ign_sys_cd = :b1

Figure 3: Extracted Consistent Gets SQLFrom this SQL in Figure 3, we see one SQL statement that is also repeated in our physical gets top SQL codes:







We know it is the same code, even though we cant see all of it by the hash value: 2624188903 being identical. This then isprobably again, the code we should look at optimizing because of this. We can obtain the entire SQL statement by extractingit from the V$SQLTEXT view using the hash value provided. Using this hash code, we can also (if we are in 9i or greater)extract the explain plan for the code from the V$SQL_PLAN table as well.

Another statement that appears in both sets is this one:

1,466 1 1,466.0 0.0 10.10 13.08 3656618493



7/11


Module: SQLNav5.exeSelect /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||','||EARNED From ( Select /*+ INDEX(pcoopt POL_COVG_ON_OFF_PRE






And the next one as well:

852 1 852.0 0.0 11.15 12.26 656581566Module: SQLNav5.exeSelect /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||','||EARNED From ( Select /*+ FIRST_ROWS */ --/*+ INDEX(pcoo






This pretty much tells us that these statements are the ones we should concentrate our tuning efforts on them first, then moveon to the other application based statements.

But how about clear indicators of troubled SQL? One issue I see over and over again, is recursion. Recursion occurs becausea statement doesnt use bind variables. For example, look at Figure 4.

Snap IdSnap Time Sessions Curs/Sess Comment------- ------------------ -------- --------- -------------------

Begin Snap: 12 07-Jun-04 17:53:55 117 8.2End Snap: 13 07-Jun-04 18:03:18 107 7.4

Elapsed: 9.38(mins)

Instance Efficiency Percentages (Target 100%)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Buffer Nowait %: 100.00 Redo NoWait %: 99.98Buffer Hit %: 98.55 In-memory Sort %: 100.00Library Hit %: 99.51 Soft Parse %: 98.80

Execute to Parse %: 63.14 Latch Hit %: 99.90Parse CPU to Parse Elapsd %: 68.58 % Non-Parse CPU: 99.45

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % Total

Event Waits Time (s) Ela Time------------------------------------ ------------ --------- --------CPU time 1,570 75.13latch free 13,348 193 9.21SQL*Net more data to client 327,015 147 7.03log file sync 3,263 91 4.34db file scattered read 191,897 44 2.13

-------------------------------------------------------------

CPU ElapsdBuffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value

--------------- ------------ -------------- ------ -------- --------- ----------



8/11


19,607,674 245 80,031.3 12.9 150.15 206.17 1654560632 Module: ? @wsrv1.linearlive.com (TNS V1-V3)SELECT COUNT(recruit_id) count, MAX(created) max FROM recruit WHERE internet_app = 1 AND stage = 'application' AND status = 'unreached' AND has_phone = 1 AND office_id = :office_id

19,448,207 244 79,705.8 12.8 149.57 203.95 219061387 Module: ? @wsrv1.linearlive.com (TNS V1-V3)

SELECT COUNT(recruit_id) count FROM recruit WHERE internet_app =1 AND stage = 'application' AND status = 'unreached' AND has_phone = 1 AND office_id = :office_id AND TO_CHAR(created, 'DD-MON-YY') = :office_today

3,507,015 44 79,704.9 2.3 26.22 33.46 922577606 Module: ? @wsrv1.linearlive.com (TNS V1-V3)SELECT MAX(created) max FROM recruit WHERE internet_app = 1 ANDstage = 'application' AND status = 'unreached' AND has_phone = 1AND office_id = :office_id

3,139,718 264 11,892.9 2.1 29.10 32.12 2158914250 Module: ? @wsrv1.linearlive.com (TNS V1-V3)SELECT COUNT(r.recruit_id) count FROM recruit r INNER JOIN inter

view_roster ir ON (r.recruit_id = ir.recruit_id) INNER JOIN interview i ON (i.interview_id = ir.interview_id) WHERE i.office_id= :office_id AND r.status IN ('scheduled', 'confirmed') AND r.ha

s_phone = 1 AND ir.last = 1 AND (i.start_time BETWEEN SYSDATE AN

2,381,778 54 44,107.0 1.6 143.82 198.66 992807648 Module: ? @wsrv1.linearlive.com (TNS V1-V3)SELECT mailer_id, first_name, last_name, zip, tracking_number, class, school_name, year FROM mailer WHERE tracking_number = :tracking_number ORDER BY year DESC

1,392,943 108 12,897.6 0.9 34.14 43.47 1151498535 Module: ? @wsrv1.linearlive.com (TNS V1-V3)SELECT recruit_id FROM recruit WHERE mailer_id = :mailer_id

1,275,420 16 79,713.8 0.8 10.27 14.01 1265818810 Module: ? @wsrv1.linearlive.com (TNS V1-V3)SELECT COUNT(r.recruit_id) count FROM recruit r INNER JOIN office o ON (o.office_id = r.office_id) WHERE r.internet_app = 1 ANDr.stage = 'application' AND r.status = 'unreached' AND r.has_pho

ne = 1 AND (r.nhlm_date < '07-JUN-04' OR r.nhlm_status != r.status) AND r.office_id = '96' AND TO_CHAR(r.created, 'J') End Disk Reads Threshold: 1000



9/11


CPU ElapsdPhysical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value

--------------- ------------ -------------- ------ -------- --------- ----------2,081,753 54 38,551.0 89.7 143.82 198.66 992807648

Module: ? @wsrv1.linearlive.com (TNS V1-V3)SELECT mailer_id, first_name, last_name, zip, tracking_number, class, school_name, year FROM mailer WHERE tracking_number = :tracking_number ORDER BY year DESC

63,977 176 363.5 2.8 9.98 12.74 2919827978 Module: ? @wsrv1.linearlive.com (TNS V1-V3)SELECT city, state FROM zip WHERE zip = :zip

6,542 4 1,635.5 0.3 1.94 3.58 1792306764 Module: ? @wsrv1.linearlive.com (TNS V1-V3)SELECT DISTINCT(r.recruit_id), CONCAT(r.first_name, CONCAT(' ',r.last_name)) name, r.source_main, r.stage, r.status, r.city, r.has_notes, o.name office_name, r.created FROM recruit r INNER JOIN recruit_office ro ON (r.recruit_id = ro.recruit_id) INNER JOI

N office o ON (o.office_id = ro.office_id) WHERE r.status != 'ac

6,396 12 533.0 0.3 8.47 1244.88 3633465809 Module: ? @wsrv1.linearlive.com (TNS V1-V3)SELECT * FROM (SELECT r.recruit_id, CONCAT(r.first_name, CONCAT(' ', r.last_name)) name, r.source_main, r.has_notes, r.stage, r.

status, o.office_id, o.name office_name, i.start_time, ir.created, ir.reminded, ir.prior_qual, ir.user_id, ((TO_CHAR(TO_DATE('07-JUN-04', 'DD-MON-YY'), 'J') - TO_CHAR(i.start_time, 'J')) + 1)



3,398 2 1,699.0 0.1 0.58 0.57 1480883621 Module: ? @wsrv1.linearlive.com (TNS V1-V3)SELECT DISTINCT(r.recruit_id), CONCAT(r.first_name, CONCAT(' ',r.last_name)) name, r.source_main, r.stage, r.status, r.city, r.has_notes, o.name office_name, r.created FROM recruit r INNER JO

IN recruit_office ro ON (r.recruit_id = ro.recruit_id) INNER JOIN office o ON (o.office_id = ro.office_id) WHERE r.status != 'ac

3,328 7 475.4 0.1 4.98 549.37 4234417328 Module: ? @wsrv1.linearlive.com (TNS V1-V3)SELECT r.recruit_id, CONCAT(r.first_name, CONCAT(' ', r.last_name)) name, r.source_main, r.has_notes, r.stage, r.status, o.office_id, o.name office_name, i.start_time, ir.created, ir.reminded,ir.prior_qual, ir.user_id, ((TO_CHAR(TO_DATE('07-JUN-04', 'DD-M

ON-YY'), 'J') - TO_CHAR(i.start_time, 'J')) + 1) day_lapse FROM





2,986 2 1,493.0 0.1 0.92 0.93 3796993827 Module: ? @wsrv1.linearlive.com (TNS V1-V3)



10/11


SELECT DISTINCT(r.recruit_id), CONCAT(r.first_name, CONCAT(' ',r.last_name)) name, r.source_main, r.stage, r.status, r.city, r.has_notes, o.name office_name, r.created FROM recruit r INNER JOIN recruit_office ro ON (r.recruit_id = ro.recruit_id) INNER JOI


% TotalParse Calls Executions Parses Hash Value

------------ ------------ -------- ----------4,025 8,043 8.72 3300435981 Module: ? @wsrv1.linearlive.com (TNS V1-V3)SELECT val FROM storage WHERE sid = :sid AND name = :name

1,694 3,387 3.67 602811079 Module: ? @wsrv2.linearlive.com (TNS V1-V3)SELECT CONCAT(stage, CONCAT('|', status)) return FROM recruit WHERE recruit_id = :bind_item_id

1,385 3,318 3.00 1013713935 Module: ? @wsrv2.linearlive.com (TNS V1-V3)SELECT phone, extension, name, rank FROM recruit_phone WHERE recruit_id = :recruit_id ORDER BY rank

1,324 2,648 2.87 1553378658 Module: ? @wsrv2.linearlive.com (TNS V1-V3)

SELECT id, start_time FROM history_user_collated WHERE user_id =:user_id AND end_time >= (SYSDATE - 15/1440)

1,321 1,322 2.86 1553758124 Module: ? @wsrv1.linearlive.com (TNS V1-V3)INSERT /*+ IDX(0) */ INTO "LINEAR"."MLOG$_HISTORY_USER_COLLATE"(dmltype$$,old_new$$,snaptime$$,change_linear$$,"ID") VALUES (:d,:o,to_date('4000-01-01:00:00:00','YYYY-MM-DD:HH24:MI:SS'),:c,:1)

1,299 1,299 2.81 188214805 Module: ? @wsrv2.linearlive.com (TNS V1-V3)UPDATE history_user_collated SET end_time = :end_time, seconds =:second_count WHERE id = :huc_id

1,254 2,508 2.72 706506348 Module: ? @wsrv2.linearlive.com (TNS V1-V3)

SELECT COUNT(*) count FROM interview_roster WHERE interview_id =:interview_id AND cancelled = 0 AND hide = 0

1,254 5,013 2.72 3547551472 Module: ? @wsrv2.linearlive.com (TNS V1-V3)SELECT COUNT(*) count FROM receptionist_nh_lm WHERE recruit_id =:recruit_id AND type = :type

1,252 2,505 2.71 1604412695 Module: ? @wsrv2.linearlive.com (TNS V1-V3)SELECT * FROM (SELECT created, type FROM receptionist_nh_lm WHERE recruit_id = :recruit_id ORDER BY created DESC) WHERE rownum =1

1,252 5,007 2.71 3600969427 Module: ? @wsrv2.linearlive.com (TNS V1-V3)SELECT COUNT(*) count FROM receptionist_nh_lm WHERE recruit_id =

:recruit_id AND type = :type AND TO_CHAR(created, 'DD-MON-YY')= :date_today

-------------------------------------------------------------

Figure 4: Problem SQL Indications

In looking at the waits, events and SQL in the above excerpts, we see a lot of CPU time being used, lots of buffer gets andlots of physical reads. We also see that our key waits are for SQL area (latch free) events. This usually indicates issues withrecursion. We can see that the various parse related ratios are very much less than 100% which is the ultimate tuning goal.This would indicate no re-parsing (at least hard parsing) was occurring. Reparsing is generally caused by lack of bind



11/11


variables. As we look through the SQL in Figure 4 we see a number of SQL statements that are not using bind variables andsome that use a mix of literal and bind variables.

In this situation we can apply the comparisons we used for the buffer reads and the physical reads as well as non-use of bindvariables to find and repair the problem SQL statements. If you are unable to correct the non-use of bind variables, using theCURSOR_SHARING initialization parameter will help with the parse situation.

Identify SQL using CommentsIt can be very difficult to pull your PL/SQL procedure code out of the background code in an instance shared pool, largetkprof, trace output or from v$sql_plan. I suggest placing a comment in each SQL statement that identifies the SQL. Anexample of this is:

CURSOR get_latch ISSELECT /* DBA_UTIL.get_latch */a.name,100.*b.sleeps/b.getsFROM

v$latchname a, v$latch bWHERE

a.latch# = b.latch# and b.sleeps > 0;

Now to find all SQL code in the shared pool from the DBA_UTILITIES package I can simply query the V$SQLAREA orV$SQLTEXT, or the V$SQL_PLAN VPT to find code entries with '%DBA_UTIL%' in the SQL_TEXT column. In addition,in any Statspack output, the code identifies itself.

SummaryIn this paper I have tried to convey the importance of using statspack to help find and isolate SQL statements that needtuning. We have covered the installation and use of Statspack and have discussed AWR and its use along with statspack.Developers should utilize statspack or AWR for point monitoring of development environments and for continuedmonitoring of production environments.


using statspack to track down bad code

Documents