GC Buffer Busy Waits Analysis

• Buffer busy waits, or gc buffer busy waits, are one of the most challenging contentions when work on Oracle database performance.

• With this presentation, we use a real world production issue to show the process to identify the root cause of contentions.

• The following is the real time active session waits from v$session. There are 6 sessions fighting for file 3831, block 2, with block class as 13. The message line shows 13 as “file header block”, which can be queried from v$waitstat as its rownum.

• In this presentation, we will mine AWR to find out the root cause.

• A common starting point to use AWR to analyze Oracle performance is dba_hist_sys_time_model.

• This database just had table reorganizations for almost all tables to reduce data dictionary contentions by removing subpartitions. I was trying to check the effectiveness of this effort.

• The following is time model after table re-organization. The time model before table re-organization is the next slide.

• It can be easily spotted that 16th hour which had significant high DB time.

• This is the time model before table re-organization.• While visually we can see the overall improvement, the peak time improvement is very limited. For example,

16th hour of 2013-06-17 used 126,495 seconds on ETL nodes (node 1 and 2), while 17th hour of 2013-06-11 used 151,878 seconds, only 17% improvement.

• The DB CPU time was 16,341 seconds before re-organization, and was 21,313 seconds after. It is possible that Oracle did more tasks within an hour after table re-organization. So comparing time model for one single AWR could be misleading. Only the application side response time or throughput could be right measurement.

• Nevertheless, it is worth to analyze where the time was spent. For the one peak (16th) hour after table re-organization, DB time was 126,495 seconds, and CPU time was 21,313 seconds. So where was the remaining 105,182 seconds, 83% of DB time?

• Wait events are always the best place to find out where the time was spent: the Oracle sessions should be either on CPU to do something, or should wait for something.

• In this slide, we limit our data to a single snap. Well, we need two snapshots since the data were recorded in accumulated values. Note the BEGIN_TIME is misleading here because it was the beginning time for the snapshot used for subtraction.

• Anyway, two “gc buffer busy …” events dominated the remaining time, 51,585 seconds, 40% of DB time. Next 4 User I/O events used 38,336 seconds, 30% of DB time. The average wait time for “direct path read” and “db file sequential read” indicate some IO capacity issue with this database, which can also contribute to “gc buffer busy …” waits.

• Here is the picture before table re-organization. • The top 6 wait events, except the 5th “direct path read”, were all shared pool and row cache

related. The total time was 115,072 seconds, 76% of total DB time. • While “gc buffer busy acquire” was at top 10, but total time was 2,955 seconds, only 2% of

total DB time.• While I am happy to see the table re-organization did remove the shared pool and row cache

related contentions, it is worth the effort to check if we can improve the new wait pattern.

• There are several places we can use to research (gc) buffer busy waits. • The first one we will check is AWR segment statistics, related to buffer busy, row locks and ITL locks. Note AWR

segment statistics record data in delta, so we only need look at the concerned time range.• From the following table, we see SYS.SEG$ and SYS.I_FILE#_BLOCK# were the major objects with buffer busy

waits. Also there were relatively large ITL waits on SYS.SEG$ and ROW_LOCK waits on I_FILE#_BLOCK#.• Since the main tasks during the period on both nodes were ETL tasks, so it is highly possible there are a lot of

activities on SYS.SEG$ to add new segments or to update segments.• Another interesting thing is the high logic IO on SYS.SEG$, at 420M.

• Since the dominant events were “gc buffer busy …”, we should really look at GC portion of AWR segment statistics.

• The following table shows vey large and dominant GC_BUFFER_BUSY values for both SYS.SEG$ and SYS.I_FILE#_BLOCK#.

• So we can be pretty much sure that the heavy “gc buffer busy …” were from segment management.

• In previous slide, we assume segment management was the reason for gc buffer busy contention. We need find more evidence to support our assumption.

• dba_hist_waitstat is another good tool to analyze buffer busy waits. From the following table, we see that waits on “file header block” dominated all buffer busy wait classes. This further strengthens our assumption that segment management is to be blamed.

• Note: I have no idea what is the unit for the time measurement, which is not documented.

• Earlier slides indicate there were high LIO on SYS.SEG$. So it is interesting to find out what SQL statement contributed that high LIO.

• The high LIO is a very simple recursive sys query:– select file#, block#, blocks from seg$ where type# = 3 and ts# = :1

• Here is the plan for the recursive SQL. It touches exact two objects we see high (gc) buffer busy waits: SEG$ and I_FILE#_BLOCK#.

• Since this is frequently running recursive SQL, I grabbed its plan from library cache so that I can get predicates. You can also get the plan from AWR, without predicate.

• This query basically requests all the seg$ data related to a single tablespace and then filter it by type#.

• You can find Oracle internal table definitions from Oracle installation, for example, /home/oracle/product/11.2.0.3/rdbms/admin. seg$ is defined inside dcore.bsq.

• From here, we can see type# 3 is TEMPORARY segment.

• The next question is, what is the possible value for the bind variable :1. While not accurate to reflect all cases, we can use dba_hist_sqlbind to get some idea.

• Here the ts# is 167, which is tablespace UDDFACTD_36M26_20130601_D.

SQL> select name,position, value_string from dba_hist_sqlbindwhere dbid=1340737874 and snap_id=11889 and instance_number=2and sql_id='at1ygf4rd7cvj'

NAME POSITION VALUE_STRI------------------------------ ---------- ----------:1 1 167

SQL> select * from v$tablespace where ts#=167;

TS# NAME INC BIG FLA ENC---------- ------------------------------ --- --- --- --- 167 UDDFACTD_36M26_20130601_D YES NO YES

• So what happened on that tablespace? Fortunately, we can aggregate metrics from dba_hist_filestatxs at tablespace level, then sort by total wait time.

• From the following table, we can see UDDFACTD_36M26_20130601_D has dominant wait_time, and has top physical blocks read (PHYBLKRD) and physical blocks write (PHYBLKWRT). Its average single block read time (AVG_SINGLEBLKRD_MS) was the worst.

• The majority of physical IO (read and write) went to two tablespaces: UDD_LARGE_DATA and UDDFACTD_36M26_20130601_D.

• AWR ASH can further help to identify the source of gc buffer busy waits. Since we suspect UDDFACTD_36M26_20130601_D was to be blamed, we will use AWR ASH to check the percentage related to this tablespace.

• For the snap we concerned, there were 4,682 samples for gc buffer busy waits with P3 as 13 (file header block), and 3,728 samples were related to tablespace UDDFACTD_36M26_20130601_D, at 80%.

• Why need we look at gc buffer busy wait with P3 as 13? P3 is the block class. The meaning can be find from v$waitstat. Silly enough, the class id is the rownum when you query the view. (from Kyle Hailey’s blog)

• The following table shows 13 as file header block, which dominates the wait count and time.

• To match tablespace, P1 of gc busy wait event is the file#.

• An interesting question is, why was there a query asking seg$ for type#=3 which was temporary segment.

• Note the ETL tasks of this database use direct load. So the segments allocated before commit is considered as temporary segments. There is another recursive sys query which will update seg$ with right information.

• There was another recursive query which creates entry inside seg$. Note this query also has relatively high cluster waits.

• To find out the type# when an entry is created, we need use sql trace (10046 trace file) with bind enabled.

So, What Are All Those About?• Some database systems have tablespace management convention

by time, for example, one tablespace is created for one month or one day, etc. All tables will use the same tablespace for partitioned data storage. It is very convenient for space management.

• But in this case, there are 105 tables to be loaded concurrently into the same tablespace. – The number of entries inside seg$ for the same tablespace is not small,

that is why we see high LIO on seg$.– The concurrency to fight new segments to load 105 tables into the same

tablespace creates hot blocks.• One idea is to use multiple tablespace to reduce the contentions on

hot blocks, but more tests are required to see if it actually works.